REPL integration test failures with macOS julia nightly

[lgoettgens]

First observed in https://github.com/oscar-system/GAP.jl/actions/runs/12625852387/job/35178150930.
With julia 1.12.0-DEV.1829 it succeeded, but with 1.12.0-DEV.1836 it fails. This leaves JuliaLang/julia@638dacc...89afe20 as the diff.

spawn julia --startup-file=no --history-file=no --banner=no --project=@.
�[?2004h
julia> 

us
julia> 
us
ingi
julia> 
usi
ngn
julia> 
usin
gg
julia> 
using
 GA
julia> 
using GA
PP
julia> 
using GAP

julia> 
using GAP

2004l1049h7h1049l
1l25l12l25h


julia> 

2004h
julia> 

 TIMEOUT 
Error: Process completed with exit code 2.

[benlorenz]

I have seen this happen earlier and disappear again, e.g. https://github.com/oscar-system/GAP.jl/actions/runs/12467506675/job/34796923548?pr=1067 with 1.12.0-DEV.1792.

[fingolfin]

@lgoettgens how systematic was this range obtained? Is this from CI logs? I wonder how reliable "it did not crash" is, i.e. maybe this error only appears with a certain probability, and one should re-run the test a couple times with a specific Julia version?

[lgoettgens]

@lgoettgens how systematic was this range obtained? Is this from CI logs? I wonder how reliable "it did not crash" is, i.e. maybe this error only appears with a certain probability, and one should re-run the test a couple times with a specific Julia version?

Just from ci logs from the scheduled jobs on master. I can try to bisect this in the next days

[fingolfin]

Would indeed be good to bisect where this comes from.

[lgoettgens]

Would indeed be good to bisect where this comes from.

Thanks for reminding me. I am on it and report back once finished

[lgoettgens]

After some researching and debugging, this seems to not be a regression, but instead has always been a random failure. In https://github.com/oscar-system/GAP.jl/actions/runs/12832003066, I ran the relevant part from the CI job with julia JuliaLang/julia@ff97facbc94, which is exactly the version that the macos-nightly job in #1595 right before merging succeeded with. And even in this version, 3 out of 10 similar jobs failed due to a timeout in the expect testsuite.

Under these circumstances, I am not sure how sensible a full bisection is. Or rather, how far I have to go back in time to find a good commit.

[fingolfin]

instead has always been a random failure

With "always" you mean "since we added the expect based tests" ? AFAIK the failures are however always with nightly, not with 1.10 or 1.11, right?

Perhaps the expect tests simply are too fragile and we need to make them more resilient, somehow, but in order to do this it would be really helpful to know what really triggers this... And to test a potential fix we need to reliably reproduce the issue. Ideally locally for quicker turnaround.

Of course if we an repro locally then we could also bisect Julia locally. sigh

[lgoettgens]

I can try to bisect nightly against the 1.11 branch off or something like that. (again with many similar CI jobs, and considering it a fail if one of them has a timeout)