Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Onboarding Jenkins prod docker images to github ci check #1538

Merged
merged 2 commits into from
Oct 30, 2023
Merged

Onboarding Jenkins prod docker images to github ci check #1538

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
bdceddb
Onboarding Jenkins prod docker images to github ci check
peterzhuamazon Oct 23, 2023
d4a71ee
Add suggested changes in the comments
peterzhuamazon Oct 27, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
113 changes: 82 additions & 31 deletions .github/workflows/CI-workflow.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,15 +14,29 @@ permissions:
contents: read

jobs:
Build-ml:
Get-CI-Image-Tag:
uses: opensearch-project/opensearch-build/.github/workflows/get-ci-image-tag.yml@main
with:
product: opensearch

Build-ml-linux:
needs: Get-CI-Image-Tag
strategy:
matrix:
java: [11, 17, 20]

name: Build and Test MLCommons Plugin
name: Build and Test MLCommons Plugin on linux
if: github.repository == 'opensearch-project/ml-commons'
environment: ml-commons-cicd-env
outputs:
build-test-linux: ${{ steps.step-build-test-linux.outputs.build-test-linux }}
runs-on: ubuntu-latest
container:
# using the same image which is used by opensearch-build team to build the OpenSearch Distribution
# this image tag is subject to change as more dependencies and updates will arrive over time
image: ${{ needs.Get-CI-Image-Tag.outputs.ci-image-version-linux }}
# need to switch to root so that github actions can install runner binary on container without permission issues.
options: --user root

steps:
- name: Setup Java ${{ matrix.java }}
Expand All @@ -41,43 +55,86 @@ jobs:
ref: ${{ github.event.pull_request.head.sha }}

- name: Build and Run Tests
id: step-build-test-linux
run: |
export OPENAI_KEY=$(aws secretsmanager get-secret-value --secret-id github_openai_key --query SecretString --output text)
export COHERE_KEY=$(aws secretsmanager get-secret-value --secret-id github_cohere_key --query SecretString --output text)
echo "::add-mask::$OPENAI_KEY"
echo "::add-mask::$COHERE_KEY"
./gradlew build
- name: Publish to Maven Local
run: |
./gradlew publishToMavenLocal
- name: Multi Nodes Integration Testing
run: |
export OPENAI_KEY=$(aws secretsmanager get-secret-value --secret-id github_openai_key --query SecretString --output text)
export COHERE_KEY=$(aws secretsmanager get-secret-value --secret-id github_cohere_key --query SecretString --output text)
echo "::add-mask::$OPENAI_KEY"
echo "::add-mask::$COHERE_KEY"
./gradlew integTest -PnumNodes=3
chown -R 1000:1000 `pwd`
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just from curiosity: is user 1000 and group id is also 1000? And where this user is defined?

su `id -un 1000` -c 'whoami && java -version &&
export OPENAI_KEY=`aws secretsmanager get-secret-value --secret-id github_openai_key --query SecretString --output text` &&
export COHERE_KEY=`aws secretsmanager get-secret-value --secret-id github_cohere_key --query SecretString --output text` &&
echo "::add-mask::$OPENAI_KEY" &&
echo "::add-mask::$COHERE_KEY" &&
./gradlew build &&
./gradlew publishToMavenLoca &&
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

publishToMavenLoca --> publishToMavenLocal

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch!

./gradlew integTest -PnumNodes=3'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add comment? Previously we had step names like: Multi Nodes Integration Testing which helps to understand. May be we can add these step names as a comment for better clarification?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure.

plugin=`basename $(ls plugin/build/distributions/*.zip)`
echo $plugin
mv plugin/build/distributions/$plugin ./
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Curious to know the output of plugin/build/distributions/$plugin

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will change it to mv -v so you will see the entire output.

echo "build-test-linux=$plugin" >> $GITHUB_OUTPUT

- name: Upload Coverage Report
uses: codecov/codecov-action@v1
with:
flags: ml-commons
token: ${{ secrets.CODECOV_TOKEN }}

- uses: actions/upload-artifact@v3
with:
name: ml-plugin-linux-${{ matrix.java }}
path: ${{ steps.step-build-test-linux.outputs.build-test-linux }}
if-no-files-found: error


Test-ml-linux-docker:
needs: Build-ml-linux
strategy:
matrix:
java: [11, 17, 20]

name: Test MLCommons Plugin on linux docker
if: github.repository == 'opensearch-project/ml-commons'
environment: ml-commons-cicd-env
runs-on: ubuntu-latest

steps:
- uses: aws-actions/configure-aws-credentials@v2
with:
role-to-assume: ${{ secrets.ML_ROLE }}
aws-region: us-west-2

- name: Checkout MLCommons
uses: actions/checkout@v3
with:
ref: ${{ github.event.pull_request.head.sha }}

- uses: actions/download-artifact@v3
with:
name: ml-plugin-linux-${{ matrix.java }}

- name: Pull and Run Docker
run: |
plugin=`basename $(ls plugin/build/distributions/*.zip)`
plugin=${{ needs.Build-ml-linux.outputs.build-test-linux }}
version=`echo $plugin|awk -F- '{print $3}'| cut -d. -f 1-3`
plugin_version=`echo $plugin|awk -F- '{print $3}'| cut -d. -f 1-4`
qualifier=`echo $plugin|awk -F- '{print $4}'| cut -d. -f 1-1`
candidate_version=`echo $plugin|awk -F- '{print $5}'| cut -d. -f 1-1`
docker_version=$version-$qualifier

[[ -z $candidate_version ]] && candidate_version=$qualifier && qualifier=""
if [ -n "$qualifier" ] && [ "$qualifier" != "SNAPSHOT" ]; then
qualifier=-${qualifier}
else
qualifier=""
fi

echo plugin version plugin_version qualifier candidate_version docker_version
echo "($plugin) ($version) ($plugin_version) ($qualifier) ($candidate_version) ($docker_version)"
docker_version=$version$qualifier

echo plugin version plugin_version qualifier docker_version
echo "($plugin) ($version) ($plugin_version) ($qualifier) ($docker_version)"

pwd && ls -l ./$plugin

cd ..
if docker pull opensearchstaging/opensearch:$docker_version
then
echo "FROM opensearchstaging/opensearch:$docker_version" >> Dockerfile
echo "RUN if [ -d /usr/share/opensearch/plugins/opensearch-ml ]; then /usr/share/opensearch/bin/opensearch-plugin remove opensearch-ml; fi" >> Dockerfile
echo "ADD ml-commons/plugin/build/distributions/$plugin /tmp/" >> Dockerfile
echo "COPY $plugin /tmp/" >> Dockerfile
echo "RUN /usr/share/opensearch/bin/opensearch-plugin install --batch file:/tmp/$plugin" >> Dockerfile
docker build -t opensearch-ml:test .
echo "imagePresent=true" >> $GITHUB_ENV
Expand Down Expand Up @@ -107,12 +164,6 @@ jobs:
./gradlew integTest -Dtests.rest.cluster=localhost:9200 -Dtests.cluster=localhost:9200 -Dtests.clustername="docker-cluster"
fi

- name: Upload Coverage Report
uses: codecov/codecov-action@v1
with:
flags: ml-commons
token: ${{ secrets.CODECOV_TOKEN }}

Build-ml-windows:
strategy:
matrix:
Expand Down
Loading