Warning from source: Check took too long.

[physics-flower]

Self-hosted on Kubernetes, Airbyte v1.1.0 (please let me know if further details are needed)

Most of our Airbyte syncs failed this morning (and the ones that haven't failed are still running after several hours).

I have asked my engineering team to look into any potential issues with our Airbyte instance, but would appreciate some guidance on what might be going wrong.

There are no logs for most syncs, just the following warnings:
Attempt 1: "Warning from source: Refresh Schema took too long."
Subsequent attempts: "Warning from source: Check took too long."

We did however get some logs from the Sentry sync:
Attempt 1: Expected sync logs, followed by

2025-01-03 00:42:58 replication-orchestrator > readFromSource: source exception
io.airbyte.workers.internal.exception.SourceException: Source process exited with non-zero exit code 143
        at io.airbyte.workers.general.BufferedReplicationWorker.readFromSource(BufferedReplicationWorker.java:364) ~[io.airbyte-airbyte-commons-worker-1.1.0.jar:?]
        at io.airbyte.workers.general.BufferedReplicationWorker.lambda$runAsyncWithHeartbeatCheck$3(BufferedReplicationWorker.java:222) ~[io.airbyte-airbyte-commons-worker-1.1.0.jar:?]
        at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
2025-01-03 00:42:58 replication-orchestrator > readFromSource: done. (source.isFinished:true, fromSource.isClosed:false)
2025-01-03 00:42:59 replication-orchestrator > processMessage: done. (fromSource.isDone:true, forDest.isClosed:false)
2025-01-03 00:43:06 INFO i.a.a.SegmentAnalyticsClient(close):232 - Closing Segment analytics client...

Attempts 2-5: "Warning from source: Check took too long."
Attempt 6 (final):

2025-01-03 01:30:09 INFO i.a.w.l.c.WorkloadApiClient(claim):75 - Claimed: true for cdaf146a-9b75-49fd-9dd2-9d64a0bb4781_571_5_check via API for local
2025-01-03 01:30:09 INFO i.a.w.l.p.s.m.Stage(apply):39 - APPLY Stage: CHECK_STATUS — (workloadId = cdaf146a-9b75-49fd-9dd2-9d64a0bb4781_571_5_check) — (dataplaneId = local)
2025-01-03 01:30:09 INFO i.a.w.l.p.s.CheckStatusStage(applyStage):59 - No pod found running for workload cdaf146a-9b75-49fd-9dd2-9d64a0bb4781_571_5_check
2025-01-03 01:30:09 INFO i.a.w.l.p.s.m.Stage(apply):39 - APPLY Stage: BUILD — (workloadId = cdaf146a-9b75-49fd-9dd2-9d64a0bb4781_571_5_check) — (dataplaneId = local)
2025-01-03 01:30:09 INFO i.a.w.l.p.s.m.Stage(apply):39 - APPLY Stage: MUTEX — (workloadId = cdaf146a-9b75-49fd-9dd2-9d64a0bb4781_571_5_check) — (dataplaneId = local)
2025-01-03 01:30:09 INFO i.a.w.l.p.s.EnforceMutexStage(applyStage):50 - No mutex key specified for workload: cdaf146a-9b75-49fd-9dd2-9d64a0bb4781_571_5_check. Continuing...
2025-01-03 01:30:09 INFO i.a.w.l.p.s.m.Stage(apply):39 - APPLY Stage: LAUNCH — (workloadId = cdaf146a-9b75-49fd-9dd2-9d64a0bb4781_571_5_check) — (dataplaneId = local)
2025-01-03 01:30:13 INFO i.a.w.l.c.WorkloadApiClient(updateStatusToLaunched):60 - Attempting to update workload: cdaf146a-9b75-49fd-9dd2-9d64a0bb4781_571_5_check to LAUNCHED.
2025-01-03 01:30:13 INFO i.a.w.l.p.h.SuccessHandler(accept):60 - Pipeline completed for workload: cdaf146a-9b75-49fd-9dd2-9d64a0bb4781_571_5_check.

When I try to test a source, I get "Airbyte is temporarily unavailable. Please try again. (HTTP 502)"

[DahanShay]

Same here.. I'm using v1.1.0 self-hosted, all my connections have failed since Thursday.
When I'm trying to create a new source I get 'An unknown error occurred. (HTTP 504)'

[physics-flower]

We found it was caused by this common issue https://discuss.airbyte.io/t/error-with-s3-storage-backend-and-need-for-log-cleaning-sweeper-setup/7145/4

It was solved by increasing minio's storage within the Kubernetes cluster

[DahanShay]

@physics-flower
Thank you.
Can you please share how you implemented the solution within your help chart yaml file?

here is what I have right now, but not sure how fix it:

storage:
    # -- The storage backend type. Supports s3, gcs, minio (default)
    type: minio # default storage used
    # -- Secret name where storage provider credentials are stored
    #storageSecretName: "airbyte-config-secrets"

minio:
  image:
    # -- Minio image used by Minio helm chart
    repository: minio/minio
    # -- Minio tag image
    tag: RELEASE.2023-11-20T22-40-07Z

  storage:
    volumeClaimValue: 2Gi

  # -- Node labels for pod assignment, see https://kubernetes.io/docs/user-guide/node-selection/
  ##
  nodeSelector: {}

  # -- Tolerations for minio pod assignment, see https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
  ##
  tolerations: []

  # -- Affinity and anti-affinity for minio pod assignment, see
  # https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity
  affinity: {}