doc: add kb for troubleshooting webhook not accessible issue #1040
+50
−0
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for longhornio ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
PhanLe1010
reviewed
Jan 17, 2025
content/kb/troubleshooting-manager-stuck-in-crash-loop-due-to-webhook-is-not-accessible.md
Outdated
Show resolved
Hide resolved
ref: longhorn/longhorn 8293 Signed-off-by: Jack Lin <[email protected]>
ChanYiLin
force-pushed
the
LH8293_add_kb_for_webhook_service_not_accessible
branch
from
74264bd to
9beb3e1
Compare
Comment on lines
+19
to
+21
| Starting from v1.5.0, the webhook services were merged into the Longhorn Manager. During startup, the manager first initializes the admission and conversion webhook services, ensuring they are accessible by curling the webhook service's URL before starting the manager service. | ||
|
|
||
| In some cases, the Longhorn Manager pod may enter a CrashLoopBackOff state due to the webhook service being inaccessible. Its failure can lead to the manager pod being repeatedly restarted. Below, we outline the three most common root causes for this issue and provide solutions to resolve it. |
There was a problem hiding this comment.
Suggested change
| Starting from v1.5.0, the webhook services were merged into the Longhorn Manager. During startup, the manager first initializes the admission and conversion webhook services, ensuring they are accessible by curling the webhook service's URL before starting the manager service. | |
| In some cases, the Longhorn Manager pod may enter a CrashLoopBackOff state due to the webhook service being inaccessible. Its failure can lead to the manager pod being repeatedly restarted. Below, we outline the three most common root causes for this issue and provide solutions to resolve it. | |
| The webhook services were merged into Longhorn Manager in v1.5.0. Because of the merge, Longhorn Manager now initializes the admission and conversion webhook services first during startup. To ensure that these services are accessible, Longhorn sends a request to the webhook service URL before starting the Longhorn Manager service. | |
| In certain situations, the webhook service may become inaccessible and cause the Longhorn Manager pod to enter a CrashLoopBackOff state. This failure can lead to repeated attempts to restart the pod. | |
| The following sections outline the most common root causes for this issue and their corresponding solutions. |
Comment on lines
+23
to
+27
| ### Root Cause 1: Firewall Is Not Set Correctly | ||
|
|
||
| The firewall configuration may be preventing communication between the pods on different nodes in your Kubernetes cluster. This can block the Longhorn Manager from accessing the webhook service, resulting in the CrashLoopBackOff state. | ||
|
|
||
| Please ensure that the pods on all nodes are able to communicate with each other. This can be verified by checking the firewall rules and ensuring that inter-pod communication is not blocked. |
There was a problem hiding this comment.
Suggested change
| ### Root Cause 1: Firewall Is Not Set Correctly | |
| The firewall configuration may be preventing communication between the pods on different nodes in your Kubernetes cluster. This can block the Longhorn Manager from accessing the webhook service, resulting in the CrashLoopBackOff state. | |
| Please ensure that the pods on all nodes are able to communicate with each other. This can be verified by checking the firewall rules and ensuring that inter-pod communication is not blocked. | |
| ### Root Cause 1: Misconfigured Firewall | |
| Incorrect firewall configuration may block communication between pods on different nodes in your Kubernetes cluster. Longhorn Manager is unable to access the webhook service, resulting in the CrashLoopBackOff state. | |
| Check your firewall rules and ensure that inter-pod communication is not blocked. |
Comment on lines
+29
to
+33
| ### Root Cause 2: DNS Doesn't Work Correctly | ||
|
|
||
| If DNS resolution is not functioning as expected, the Longhorn Manager may be unable to reach the webhook service by its DNS name. This is particularly important when accessing services through their internal Kubernetes DNS names. | ||
|
|
||
| Please ensure that the dns works by executing into a pod and test if DNS resolution works correctly by attempting to curl the webhook service: |
There was a problem hiding this comment.
Suggested change
| ### Root Cause 2: DNS Doesn't Work Correctly | |
| If DNS resolution is not functioning as expected, the Longhorn Manager may be unable to reach the webhook service by its DNS name. This is particularly important when accessing services through their internal Kubernetes DNS names. | |
| Please ensure that the dns works by executing into a pod and test if DNS resolution works correctly by attempting to curl the webhook service: | |
| ### Root Cause 2: DNS Resolution Issues | |
| DNS resolution is crucial for accessing services via their internal Kubernetes DNS names. When DNS resolution is not functioning as expected, Longhorn Manager may be unable to reach the webhook service via its DNS name. | |
| Execute the webhook service in a pod, and then check if DNS resolution is functioning correctly by running the following commands: |
| curl https://longhorn-conversion-webhook.longhorn-system.svc:9501/v1/healthz | ||
| ``` | ||
|
|
||
| Or Please verify if the CoreDNS or kube-dns service is running correctly. For more details on how to check this, refer to the official Kubernetes documentation on [Debugging DNS Resolution](https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/) for more information. |
There was a problem hiding this comment.
Suggested change
| Or Please verify if the CoreDNS or kube-dns service is running correctly. For more details on how to check this, refer to the official Kubernetes documentation on [Debugging DNS Resolution](https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/) for more information. | |
| You can also check if either CoreDNS or Kube-DNS is running correctly. For more information, see [Debugging DNS Resolution](https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/) in the Kubernetes documentation. |
Comment on lines
+42
to
+46
| ### Root Cause 3: Hairpin Is Not Set Correctly | ||
|
|
||
| Hairpinning allows a pod to access itself using its Service IP. This is a common issue in single-node clusters but can also happen in a muti-node cluster, where a pod may fail to access services via the service's internal DNS name. | ||
|
|
||
| Please verify that the hairpin setting is enabled. The hairpin setting ensures that a pod can access itself via its Service IP. You can refer to the official Kubernetes documentation [Edge case: A Pod fails to reach itself via the Service IP](https://kubernetes.io/docs/tasks/debug/debug-application/debug-service/#a-pod-fails-to-reach-itself-via-the-service-ip) for more information on hairpinning in the cluster. |
There was a problem hiding this comment.
Suggested change
| ### Root Cause 3: Hairpin Is Not Set Correctly | |
| Hairpinning allows a pod to access itself using its Service IP. This is a common issue in single-node clusters but can also happen in a muti-node cluster, where a pod may fail to access services via the service's internal DNS name. | |
| Please verify that the hairpin setting is enabled. The hairpin setting ensures that a pod can access itself via its Service IP. You can refer to the official Kubernetes documentation [Edge case: A Pod fails to reach itself via the Service IP](https://kubernetes.io/docs/tasks/debug/debug-application/debug-service/#a-pod-fails-to-reach-itself-via-the-service-ip) for more information on hairpinning in the cluster. | |
| ### Root Cause 3: Hairpinning Not Implemented Correctly | |
| Hairpinning allows a pod to access itself via its service IP. In some cases, however, a pod may fail to access a service via the service's internal DNS name. This issue is common in single-node clusters and may also occur in some multi-node clusters. | |
| Verify that the `hairpin-mode` flag, which ensures that a pod can access itself via its service IP, is set correctly. For more information, see [Edge case: A Pod fails to reach itself via the Service IP](https://kubernetes.io/docs/tasks/debug/debug-application/debug-service/#a-pod-fails-to-reach-itself-via-the-service-ip) in the Kubernetes documentation. |
| @@ -0,0 +1,50 @@ | |||
| --- | |||
| title: "Troubleshooting: Longhorn Manager Stuck in CrashLoopBackOff Due to Webhook Not Accessible" | |||
There was a problem hiding this comment.
Suggested change
| title: "Troubleshooting: Longhorn Manager Stuck in CrashLoopBackOff Due to Webhook Not Accessible" | |
| title: "Troubleshooting: Longhorn Manager Stuck in CrashLoopBackOff State Due to Inaccessible Webhook" |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
ref: longhorn/longhorn#8293
add a kb for troubleshooting webhook not accessible issue