[Bug]: Building wheels failing on self hosted actions, working on github's runners

[damccorm]

What happened?

Starting Dec 10, building Python wheels started failing with a bunch of seg faults. It seems likely due to something with the underlying hardware or image. I tried updating to using GitHub hosted runners and it seems like this works. We can do this as a workaround, but we should understand the problem and switch back to self-hosted to avoid being blocked on GitHub quota.

Example failure - https://github.com/apache/beam/actions/runs/12625457564

This also impacts some other workflows which I will switch over to GitHub hosted, but we should similarly switch back

Workflows where we've seen this issue:

• .github/workflows/build_wheels.yml

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[damccorm]

@claudevdm did some good investigation here - Claude would you mind adding in any investigation you've done/things you've tried

[claudevdm]

Sure, here is what I found

• The workflow started failing consistently on Dec 10
• The only difference I could find since they started failing is a runner/host Kernel Version: 6.1.100+ to Kernel Version: 6.1.112+ change
• The kernel date change lines up with a release on Dec 10
• The arm workflows have to do some cross-compilation thing using qemu
• cibuildwheels pulls quay.io/pypa/manylinux2014_aarch64 which uses gcc 10 to compile the cyton code
• It seems there are some incompatible changes with the new kernel release and cross compiling with gcc 10