[Bug]: Opensearch build workflow flaky, creating multiple issues on Plugin repos

[monusingh-1]

Describe the bug

When OS builds are triggered the workflow fails intermittently and cuts issue on plugin repos, the issues are then resolved in next run. When the next run happens a new issue is created.

129 issues have been cut so far https://github.com/opensearch-project/cross-cluster-replication/issues?q=is%3Aissue++%5BAUTOCUT%5D+Distribution+Build+Failed+for+cross-cluster-replication-3.0.0+

https://build.ci.opensearch.org/job/distribution-build-opensearch/8674/consoleFull
On CCR

* What went wrong:
Could not open settings generic class cache for settings file '/tmp/tmpf5abke27/cross-cluster-replication/settings.gradle' (/usr/share/opensearch/.gradle/caches/7.6.1/scripts/86cxd8y0mmz3g16wn40v6nbpn).
> BUG! exception in phase 'semantic analysis' in source unit '_BuildScript_' Unsupported class file major version 64

on K-nn

* What went wrong:
Could not open settings generic class cache for settings file '/tmp/tmp4l9pm7bs/k-NN/settings.gradle' (/usr/share/opensearch/.gradle/caches/7.6.1/scripts/1b6isqorp8hfaws3ypmmtcmnx).
> BUG! exception in phase 'semantic analysis' in source unit '_BuildScript_' Unsupported class file major version 64

Could you please confirm if the issues are actionable by plugin maintainers ?
How can we address the falky nature of this build ?

To reproduce

Expected behavior

No response

Screenshots

If applicable, add screenshots to help explain your problem.

Host / Environment

No response

Additional context

No response

Relevant log output

No response

[gaiksaya]

Hi @monusingh-1

The builds are not flaky but the way GH create issue is written is buggy. As per my analysis this is the culprit: https://github.com/opensearch-project/opensearch-build/blob/main/manifests/3.0.0/opensearch-3.0.0.yml#L9
https://github.com/opensearch-project/opensearch-build/blob/main/jenkins/opensearch/distribution-build.jenkinsfile#L645

Looks like our infra is building on different java versions for linux and windows.
Since plugins did not upgrade to jdk 20 everything fails on linux which is legit.
However, (I think and pretty sure) on windows it finds compatible java version and builds itself successfully closing all the GH issues for all below components that are and should be broken today:

++ scoop list jdk
++ grep 20
++ head -1
++ cut -d ' ' -f1
+ scoop reset openjdk14
Resetting openjdk14 (14.0.2-12).

2023-10-17 13:11:57 ERROR    Failed plugins are ['opensearch-reports', 'performance-analyzer', 'opensearch-observability', 'k-NN', 'neural-search', 'geospatial', 'cross-cluster-replication', 'sql', 'asynchronous-search']

@peterzhuamazon created a PR to fix these recently. However, the failures are right. The error mentioned by you is java incompatible error (have seen that before).
Thanks!

[gaiksaya]

I think the above PR fixes the false positive. 3.0.0 is building right now https://build.ci.opensearch.org/blue/organizations/jenkins/distribution-build-opensearch/detail/distribution-build-opensearch/8677/pipeline/75 and I can see geospatial plugin failing on windows similar to linux (it was not happening before).
Windows is also fixed now:

2023-10-19 01:11:09 ERROR    Failed plugins are ['opensearch-reports', 'opensearch-observability', 'k-NN', 'neural-search', 'geospatial', 'cross-cluster-replication', 'sql', 'asynchronous-search']

Will create an issue to fix the GH creation logic.
Thanks!

[gaiksaya]

Created issue to add more details to GH issue #4155
Closing this issue.