Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

doc: add kb for troubleshooting webhook not accessible issue #1040

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
+50 −0
Open

doc: add kb for troubleshooting webhook not accessible issue #1040

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 50 additions & 0 deletions ...troubleshooting-manager-stuck-in-crash-loop-due-to-webhook-is-not-accessible.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
---
title: "Troubleshooting: Longhorn Manager Stuck in CrashLoopBackOff Due to Webhook Not Accessible"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
title: "Troubleshooting: Longhorn Manager Stuck in CrashLoopBackOff Due to Webhook Not Accessible"
title: "Troubleshooting: Longhorn Manager Stuck in CrashLoopBackOff State Due to Inaccessible Webhook"

authors:
- "Jack Lin"
draft: false
date: 2025-01-17
versions:
- ">= v1.5.0"
categories:
- "longhorn manager"
---

## Applicable versions

Longhorn >= v1.5.0.

## Symptoms

Starting from v1.5.0, the webhook services were merged into the Longhorn Manager. During startup, the manager first initializes the admission and conversion webhook services, ensuring they are accessible by curling the webhook service's URL before starting the manager service.

In some cases, the Longhorn Manager pod may enter a CrashLoopBackOff state due to the webhook service being inaccessible. Its failure can lead to the manager pod being repeatedly restarted. Below, we outline the three most common root causes for this issue and provide solutions to resolve it.
Comment on lines +19 to +21
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Starting from v1.5.0, the webhook services were merged into the Longhorn Manager. During startup, the manager first initializes the admission and conversion webhook services, ensuring they are accessible by curling the webhook service's URL before starting the manager service.
In some cases, the Longhorn Manager pod may enter a CrashLoopBackOff state due to the webhook service being inaccessible. Its failure can lead to the manager pod being repeatedly restarted. Below, we outline the three most common root causes for this issue and provide solutions to resolve it.
The webhook services were merged into Longhorn Manager in v1.5.0. Because of the merge, Longhorn Manager now initializes the admission and conversion webhook services first during startup. To ensure that these services are accessible, Longhorn sends a request to the webhook service URL before starting the Longhorn Manager service.
In certain situations, the webhook service may become inaccessible and cause the Longhorn Manager pod to enter a CrashLoopBackOff state. This failure can lead to repeated attempts to restart the pod.
The following sections outline the most common root causes for this issue and their corresponding solutions.


### Root Cause 1: Firewall Is Not Set Correctly

The firewall configuration may be preventing communication between the pods on different nodes in your Kubernetes cluster. This can block the Longhorn Manager from accessing the webhook service, resulting in the CrashLoopBackOff state.

Please ensure that the pods on all nodes are able to communicate with each other. This can be verified by checking the firewall rules and ensuring that inter-pod communication is not blocked.
Comment on lines +23 to +27
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
### Root Cause 1: Firewall Is Not Set Correctly
The firewall configuration may be preventing communication between the pods on different nodes in your Kubernetes cluster. This can block the Longhorn Manager from accessing the webhook service, resulting in the CrashLoopBackOff state.
Please ensure that the pods on all nodes are able to communicate with each other. This can be verified by checking the firewall rules and ensuring that inter-pod communication is not blocked.
### Root Cause 1: Misconfigured Firewall
Incorrect firewall configuration may block communication between pods on different nodes in your Kubernetes cluster. Longhorn Manager is unable to access the webhook service, resulting in the CrashLoopBackOff state.
Check your firewall rules and ensure that inter-pod communication is not blocked.


### Root Cause 2: DNS Doesn't Work Correctly

If DNS resolution is not functioning as expected, the Longhorn Manager may be unable to reach the webhook service by its DNS name. This is particularly important when accessing services through their internal Kubernetes DNS names.

Please ensure that the dns works by executing into a pod and test if DNS resolution works correctly by attempting to curl the webhook service:
Comment on lines +29 to +33
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
### Root Cause 2: DNS Doesn't Work Correctly
If DNS resolution is not functioning as expected, the Longhorn Manager may be unable to reach the webhook service by its DNS name. This is particularly important when accessing services through their internal Kubernetes DNS names.
Please ensure that the dns works by executing into a pod and test if DNS resolution works correctly by attempting to curl the webhook service:
### Root Cause 2: DNS Resolution Issues
DNS resolution is crucial for accessing services via their internal Kubernetes DNS names. When DNS resolution is not functioning as expected, Longhorn Manager may be unable to reach the webhook service via its DNS name.
Execute the webhook service in a pod, and then check if DNS resolution is functioning correctly by running the following commands:


```bash
kubectl exec -it <pod-name> -- /bin/bash
curl https://longhorn-conversion-webhook.longhorn-system.svc:9501/v1/healthz
```

Or Please verify if the CoreDNS or kube-dns service is running correctly. For more details on how to check this, refer to the official Kubernetes documentation on [Debugging DNS Resolution](https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/) for more information.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Or Please verify if the CoreDNS or kube-dns service is running correctly. For more details on how to check this, refer to the official Kubernetes documentation on [Debugging DNS Resolution](https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/) for more information.
You can also check if either CoreDNS or Kube-DNS is running correctly. For more information, see [Debugging DNS Resolution](https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/) in the Kubernetes documentation.


### Root Cause 3: Hairpin Is Not Set Correctly

Hairpinning allows a pod to access itself using its Service IP. This is a common issue in single-node clusters but can also happen in a muti-node cluster, where a pod may fail to access services via the service's internal DNS name.

Please verify that the hairpin setting is enabled. The hairpin setting ensures that a pod can access itself via its Service IP. You can refer to the official Kubernetes documentation [Edge case: A Pod fails to reach itself via the Service IP](https://kubernetes.io/docs/tasks/debug/debug-application/debug-service/#a-pod-fails-to-reach-itself-via-the-service-ip) for more information on hairpinning in the cluster.
Comment on lines +42 to +46
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
### Root Cause 3: Hairpin Is Not Set Correctly
Hairpinning allows a pod to access itself using its Service IP. This is a common issue in single-node clusters but can also happen in a muti-node cluster, where a pod may fail to access services via the service's internal DNS name.
Please verify that the hairpin setting is enabled. The hairpin setting ensures that a pod can access itself via its Service IP. You can refer to the official Kubernetes documentation [Edge case: A Pod fails to reach itself via the Service IP](https://kubernetes.io/docs/tasks/debug/debug-application/debug-service/#a-pod-fails-to-reach-itself-via-the-service-ip) for more information on hairpinning in the cluster.
### Root Cause 3: Hairpinning Not Implemented Correctly
Hairpinning allows a pod to access itself via its service IP. In some cases, however, a pod may fail to access a service via the service's internal DNS name. This issue is common in single-node clusters and may also occur in some multi-node clusters.
Verify that the `hairpin-mode` flag, which ensures that a pod can access itself via its service IP, is set correctly. For more information, see [Edge case: A Pod fails to reach itself via the Service IP](https://kubernetes.io/docs/tasks/debug/debug-application/debug-service/#a-pod-fails-to-reach-itself-via-the-service-ip) in the Kubernetes documentation.


## Related Information

* Longhorn issue: [#8293](https://github.com/longhorn/longhorn/issues/8293)
Loading