Java heap space error in MSSQL source connector #48348
Replies: 6 comments 7 replies
-
|
@niveditabaliga-payroc can you share what resources the cluster/server Airbyte is running has (total memory/cpu/disk)? |
Beta Was this translation helpful? Give feedback.
-
|
32 gb ram cpu DISK |
Beta Was this translation helpful? Give feedback.
-
|
@rodireich could you help taking a look into this issue? |
Beta Was this translation helpful? Give feedback.
-
|
Ideally when the java heap space error happens it should state allocated vs used stats plus which pod(s) name this occurred in so we would know what configuration knob to turn. In our experience we see multiple log messages indicating that the actual memory and disk resource usage is very low then suddenly we get the dreaded javaheapspace error 2024-11-05 01:19:11 destination > 2024-11-05 01:19:11 INFO i.a.c.i.d.FlushWorkers(printWorkerInfo):146 - [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0 |
Beta Was this translation helpful? Give feedback.
-
@rodireich, @marcosmarxm any thoughts here. @niveditabaliga-payroc and I are at a bit of a dead end with airbyte on this use case. |
Beta Was this translation helpful? Give feedback.
-
|
Curious: What is the reason to extend initial waiting time in the config? Are we expecting large database schema? |
Beta Was this translation helpful? Give feedback.
-
|
@xiaohansong yes. The source DB/schema has thousands of tables of which we pull in a handful for now. |
Beta Was this translation helpful? Give feedback.
-
|
Got it, thanks for the context. You can try to adjust the resource of the source pod. I understand your machine has 32 GB RAM in total, but the airbyte runs connector in a resource constraint container and they are not fully utilizing all the memory by default configuration. Follow this doc: https://docs.airbyte.com/operator-guides/configuring-connector-resources#configuring-connection-specific-requirements Please update |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for the input @xiaohansong |
Beta Was this translation helpful? Give feedback.
-
|
@xiaohansong , @rodireich - Thoughts? what other factors can be adjusted to "chunk" the processing? does the cdc sysnc process need to load the entire sql server _ct table into memory before processing? The log indicated that hardly any ram was being utilized as indicated in the included log of the initial post from @niveditabaliga-payroc |
Beta Was this translation helpful? Give feedback.
-
|
Anything on this @marcosmarxm? |
Beta Was this translation helpful? Give feedback.
-
|
@marcosmarxm , @rodireich - Airbyte successfully can pull 755GB worth of data from a source table in an initialization of the CDC based sql server connection but consistently falls over if there is any significant change to the source table. How can we adjust airbyte to handle this? lowering the batch size and setting large source connection max memory limit has not helped? ### result each time is java heap space error. also note we are using latest abctl version Any thoughts? |
Beta Was this translation helpful? Give feedback.
-
|
I'm currently having a similar issue with DynamoDB as a source. I run into this java heap space problem a lot. |
Beta Was this translation helpful? Give feedback.
-
Hello,
My team migrated to Airbyte at the start of this year and mid this year, we started using the CDC capability of MSSQL. After a lot of hiccups, posts on the Slack channel, reading through various posts and documents online, we were able to get CDC working and in a stable place. However, one of the jobs has started failing again on a java heap space error. The CDC _CT table had approx 91M rows at the time of investigation and the log file retention period on the DB is 3 days.
default_workspace_job_17454_attempt_1_txt.txt
`airbyte-db-0:/$ psql airbyte -d db-airbyte -t -A -c "UPDATE actor_definition SET resource_requirements='{"jobSpecific": [{"jobType": "sync", "resourceRequirements": {"cpu_limit": "2", "cpu_request": "2", "memory_limit": "29Gi", "memory_request": "2Gi"}}]}' WHERE id = 'b5ea17b1-f170-46dc-bc31-cc744ca984c1';"
airbyte-db-0:/$ psql -U airbyte -d db-airbyte -t -A -c "UPDATE connection SET resource_requirements = '{"cpu_limit": "0.5", "cpu_request": "0.5", "memory_limit": "28Gi", "memory_request": "5Gi"}' WHERE id = '4f763461-2f5a-4535-a3fe-33d99b718a05';"`
5. Attached is a screenshot of the UI parameters
We are looking for insights into how to best debug and resolve this issue.
Beta Was this translation helpful? Give feedback.
All reactions