ci: speed up workflows and reduce costs #1545

strophy · 2023-10-31T04:48:18Z

Issue being fixed or feature implemented

Builds were running in parallel, often building the same code to run different tests.
The Rust build cache is a complex solution that has multiple downsides. It cause compilation failures due to pollution from different branches. The cache size is huge (5 - 120 GB), making it inefficient.
Costs for self-hosted runners are pretty high.
Release workflow takes about 1 hour
Test workflow takes 30 mins without dashmate tests and 1 hour with dashmate tests

What was done?

Use sccache instead of target dir cache for Rust builds
Cache only with s3
Changed hardcoded Rust toolchain to "stable"
Updated change filter paths to trigger package tests
Build JS artifacts only once and then share them with subsequent jobs
Build test docker images only once and then share them with test suite and dashmate workflows
Use free runners where it's possible
Removed Rust compilation error job
Cache local network setup data to speed up test suite and dashmate workflows
Run dashmate tests in parallel
Run Clippy only for specific package (without deps)
Make sure that Rust build a stick to the lock file
Changed test workflow structure to control the flow and increase visibility
Support S3 storage in sccache in Dockerfile
Introduced DASHMATE_E2E_TESTS_SKIP_IMAGE_BUILD in to disable image builds in dashmate tests
Introduced DASHMATE_E2E_TESTS_LOCAL_HOMEDIR to set specific home dir in dashmate tests
Build arm and amd images for release in parallel without emulation
Switch to cheaper arm runners
Run test suite in 2 parallel jobs for Node.JS
Introduce BROWSER_TEST_BATCH_INDEX and BROWSER_TEST_BATCH_TOTAL to run only batch of test suite tests in browsers

How Has This Been Tested?

Running workflows

Breaking Changes

None

Checklist:

I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have added or updated relevant unit/integration/functional/e2e tests
I have added "!" to the title and described breaking changes in the corresponding section if my code contains any
I have made corresponding changes to the documentation if needed

For repository code-owners and collaborators only

I have assigned this pull request to a milestone

github-advanced-security · 2023-11-01T04:16:05Z

This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation.

strophy

Didn't review changes under packages/ much but the CI stuff looks good to me! Couple of small changes to make

.github/actions/docker/action.yaml

.github/actions/nodejs/action.yaml

lklimek · 2023-11-13T12:31:16Z

.github/actions/docker/action.yaml

-              all = true
-              keepBytes = 30000000000 # 30 GB
-              keepDuration = 864000 # 10 days
+            gc = false


it means the cache size will grow indefinitely

We drop it every time. Only cargo registry mount is stored on S3 where we have own retention policy for files (3 days)

lklimek · 2023-11-13T12:32:55Z

.github/actions/local-network/action.yaml

+      shell: bash
+      run: echo "sha=$(git log -1 --format="%h" -- packages/dashmate)" >> $GITHUB_OUTPUT
+
+    # TODO: Use upload artifacts action instead


check this todo

I wasn't going to do it this iteration

lklimek · 2023-11-13T12:39:07Z

.github/actions/local-network/action.yaml

+        key: local-network-volumes/${{ steps.dashmate-fingerprint.outputs.sha }}
+      if: steps.local-network-data.outputs.cache-hit != 'true'
+
+    - name: Configure pre-built docker images


Where do we get these images from? Looks like we assume they are already pushed (= we should have some dependency on another job declared above). Please add dependency/comment explaining that.

It's not a job, it's an action that pulls images to do the work. Workflows what use this action defines dependencies to build-images job.

lklimek · 2023-11-13T12:42:04Z

.github/actions/nodejs/action.yaml

    - name: Cache NPM build artifacts (S3 bucket cache)
-      uses: everpcpc/actions-cache@v1
-      if: contains(runner.name, 'ubuntu-platform')
+      uses: strophy/actions-cache@opendal-update


We should prefer @master, + we should migrate that repo to dashpay org

We expect some updates on upstream soon. When they don't we will change it.

lklimek · 2023-11-13T12:43:29Z

.github/actions/rust/action.yaml

    - name: Install protoc
      id: deps-protoc
      shell: bash
      run: |
        curl -Lo /tmp/protoc.zip \
-          https://github.com/protocolbuffers/protobuf/releases/download/v22.0/protoc-22.0-linux-x86_64.zip
+          https://github.com/protocolbuffers/protobuf/releases/download/v22.0/protoc-22.0-linux-aarch_64.zip


Will it work on different architectures? maybe we should add some if depending on runner arch?

It works this way according to the tests. We want to move this step to AMI. When infra guys have time they will do it.

lklimek · 2023-11-13T12:53:07Z

.github/workflows/tests-build-image.yml

+          target: ${{ inputs.target }}
+          platform: linux/arm64
+          push_tags: true
+          dockerhub_username: ${{ secrets.DOCKERHUB_USERNAME }}


why we use dockerhub_username if we use ECR? If this is correct (eg. just var name), please add comment

We login to our pairs docker hub user to extend pull limits

shumkov

Great job guys!

strophy force-pushed the ci/build-cache branch 2 times, most recently from 8124b92 to d279ced Compare October 31, 2023 06:21

ci: try multi-line expression

01260d0

strophy force-pushed the ci/build-cache branch from d279ced to 01260d0 Compare October 31, 2023 06:22

strophy and others added 3 commits October 31, 2023 16:31

ci: add steps to build layer cache

6a39ac3

ci: refactor conditions and runs

421c950

refactor: move things back and forth

1b6b0b9

shumkov changed the title ~~ci: pre-cache builds~~ [WIP] ci: pre-cache builds Oct 31, 2023

shumkov marked this pull request as ready for review October 31, 2023 11:53

shumkov requested review from shumkov, antouhou and QuantumExplorer as code owners October 31, 2023 11:53

chore: empty commit to fix CI

4ed50f2

shumkov changed the title ~~[WIP] ci: pre-cache builds~~ ci: pre-cache builds Oct 31, 2023

refactor: some renaming

bb69884

shumkov changed the title ~~ci: pre-cache builds~~ [WIP] ci: pre-cache builds Oct 31, 2023

shumkov added 3 commits November 1, 2023 10:34

refactor: continue to move things

8ad9d7b

refactor: continue to move things

2021e69

refactor: continue to move things

4ecfb59

shumkov and others added 10 commits November 1, 2023 11:46

refactor: continue to move things

71ea0e8

refactor: continue to move things

0523b78

ci: Save s3 cache failed: Error: getaddrinfo ENOTFOUND s3..amazonaws.com

5773e7a

ci: try to remove configure-aws-credentials step

07ce3b6

ci: try upstream actions-cache

fbca54f

ci: try to use merged actions-cache

96290df

ci: try with updated opendal

f1f3989

ci: run action from branch

8a42490

chore: trigger ci

5c5c852

chore: trigger ci

4f27dac

shumkov added 15 commits November 12, 2023 12:04

ci: fix tests4x

a4c5512

ci: run linting on 2x

523e2be

ci: refactor calls

47b2798

ci: fix test-runner input

dcedc48

ci: linter on free runner

9120679

test: run tests suite in parallel

b605297

ci: test conditions

6df7414

ci: test conditions

533b407

ci: test conditions

376e0ee

ci: test conditions

d3b98b7

ci: test conditions

9f42000

ci: test conditions

295d12e

chore: fix linter

34dd2e9

fix: multiple fixes

4d2257b

fix: multiple fixes

c973f32

shumkov changed the title ~~[WIP] ci: speed up workflows and reduce costs~~ ci: speed up workflows and reduce costs Nov 12, 2023

shumkov added 5 commits November 13, 2023 09:40

ci: use brave browser

47b1754

ci: move functional tests to a separate job

9074b5f

tests: fix browser tests suite parallel loader

3188982

ci: try multiple options for browser tests

cda2028

ci: fix local network action

1019b04

strophy commented Nov 13, 2023

View reviewed changes

.github/actions/docker/action.yaml Show resolved Hide resolved

.github/actions/nodejs/action.yaml Show resolved Hide resolved

shumkov added 4 commits November 13, 2023 16:11

test: fix batching test suite in browsers

ea4abe3

ci: remove testing jobs

bd795d2

ci: remove invalid input

196343b

test: remove invalid assert in test suite

007dddf

lklimek reviewed Nov 13, 2023

View reviewed changes

shumkov approved these changes Nov 13, 2023

View reviewed changes

shumkov merged commit f14e3e6 into master Nov 13, 2023
78 checks passed

shumkov deleted the ci/build-cache branch November 13, 2023 15:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: speed up workflows and reduce costs #1545

ci: speed up workflows and reduce costs #1545

strophy commented Oct 31, 2023 •

edited by shumkov

Loading

github-advanced-security bot commented Nov 1, 2023

strophy left a comment

lklimek Nov 13, 2023

shumkov Nov 13, 2023

lklimek Nov 13, 2023

shumkov Nov 13, 2023

lklimek Nov 13, 2023

shumkov Nov 13, 2023

lklimek Nov 13, 2023

shumkov Nov 13, 2023

lklimek Nov 13, 2023

shumkov Nov 13, 2023

lklimek Nov 13, 2023

shumkov Nov 13, 2023

shumkov left a comment

ci: speed up workflows and reduce costs #1545

ci: speed up workflows and reduce costs #1545

Conversation

strophy commented Oct 31, 2023 • edited by shumkov Loading

Issue being fixed or feature implemented

What was done?

How Has This Been Tested?

Breaking Changes

Checklist:

github-advanced-security bot commented Nov 1, 2023

strophy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shumkov left a comment

Choose a reason for hiding this comment

strophy commented Oct 31, 2023 •

edited by shumkov

Loading