From 0e0082d6a3f1d1c242f79ab2e4ae32fbbe645934 Mon Sep 17 00:00:00 2001 From: "Aldo \"xoen\" Giambelluca" Date: Wed, 9 Dec 2020 16:45:40 +0000 Subject: [PATCH 1/3] GHA: Fix for markdown lint branch warning This should make this error go away ``` Warning: Github Action: actionshub/markdownlint has migrated to the main branch as default, the master branch will be removed ``` --- .github/workflows/jupyter-lab-test-and-build.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/jupyter-lab-test-and-build.yml b/.github/workflows/jupyter-lab-test-and-build.yml index e96a105..178e393 100644 --- a/.github/workflows/jupyter-lab-test-and-build.yml +++ b/.github/workflows/jupyter-lab-test-and-build.yml @@ -22,7 +22,7 @@ jobs: - name: Check out code uses: actions/checkout@main - name: Run Markdown Lint - uses: actionshub/markdownlint@master + uses: actionshub/markdownlint@main docker: runs-on: [self-hosted, ministryofjustice, ecr] From 2c3c194089dd325ba4387786badc6eb317b5ee88 Mon Sep 17 00:00:00 2001 From: "Aldo \"xoen\" Giambelluca" Date: Wed, 9 Dec 2020 16:09:47 +0000 Subject: [PATCH 2/3] README: Describe image flavours and high level release steps --- README.md | 39 ++++++++++++++++++++++++++++++++++++--- 1 file changed, 36 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index e2c577a..2efaab0 100644 --- a/README.md +++ b/README.md @@ -10,9 +10,14 @@ Contents: CI/CD: -- CI: Github Actions is configured to build each of these docker images +- CI: Github Actions [is configured](./.github/workflows/jupyter-lab-test-and-build.yml) to + 1. build these docker images + 2. run tests using [`inspec`](https://community.chef.io/tools/chef-inspec/) + 3. push image to AWS ECR + - Helm chart: -- Deployment: Done by control panel when a user requests for themselves a Jupyter pod. Deploys using helm. +- Deployment: Done by Control Panel when a user deploys JupyterLab for + themselves. This releases the helm chart above. ## About Jupyter Notebook @@ -22,7 +27,19 @@ From [Jupter](http://jupyter.org): > visualizations and explanatory text. Uses include: data cleaning and transformation, numerical simulation, statistical > modeling, machine learning and much more." -## Docker image +## Docker images + +We currently have 3 flavours of JupyterLab: + +- `datascience-notebook` is the standard image currently used by most of + our users +- `allspark-notebook` is similar to the `datascience-notebook` one but it + includes Spark. This is currently used mainly by the Data Engineer team + and it's [deployed manually](https://github.com/ministryofjustice/analytics-platform/wiki/Deploying-jupyterlab#spark-version). + Long term we may use this image by default instead of the `datascience-notebook` one. +- `oracle-datascience-notebook` is a temporary image which contains drivers to + connect to Oracle databases as part of some niche and temporary work. + This image is hopefully going to disappear very soon. These images are derived from [jupyter/docker-stacks](https://github.com/jupyter/docker-stacks/blob/master/README.md). @@ -36,6 +53,22 @@ From the sub-directory for the image you want to build make build ``` +### Releasing a new image version + +Once your changes are approved and merged into the `main` branch, create a +new release tag from the GitHub interface. + +This will trigger a new run of the GitHub Actions worflow which will build +the images and push to [our private AWS ECR registry](https://eu-west-1.console.aws.amazon.com/ecr/repositories?region=eu-west-1). + +Once the image is correctly pushed to this registry you can update the relevant +[JupyterLab helm chart values](https://github.com/ministryofjustice/analytics-platform-helm-charts/blob/e2bc45e798ab97ee70a2f5a3cf52440648f23f81/charts/jupyter-lab/values.yaml#L31-L33) and/or update a specific user's +kubernetes `Deployment`. + +If you're releasing a new version of the JupyterLab helm chart, talk with a +Control Panel admin to be sure this new version is added to the Tools catalogue +and users can deploy/upgrade JupyterLab and use it. + ## Disabling authentication In order to disable the authentication, we append `--NotebookApp.token=''` as an argument From ea9e5c3a8c112faabf6deee9c8c0aae071524d7d Mon Sep 17 00:00:00 2001 From: "Aldo \"xoen\" Giambelluca" Date: Wed, 9 Dec 2020 16:50:57 +0000 Subject: [PATCH 3/3] Fix markdown linter's compaints --- README.md | 40 +++++++++++++++++----------------------- 1 file changed, 17 insertions(+), 23 deletions(-) diff --git a/README.md b/README.md index 2efaab0..e0b24d6 100644 --- a/README.md +++ b/README.md @@ -1,23 +1,17 @@ # analytics-platform-jupyter-notebook -Jupyter Notebook Docker image for Analytics Platform - -Contents: - -- [Datascience-Notebook] -- [Oracle-Datascience-Notebook] -- [Allspark-Notebook] +JupyterLab Docker images for Analytical Platform. CI/CD: -- CI: Github Actions [is configured](./.github/workflows/jupyter-lab-test-and-build.yml) to +* CI: Github Actions [is configured](./.github/workflows/jupyter-lab-test-and-build.yml) to 1. build these docker images - 2. run tests using [`inspec`](https://community.chef.io/tools/chef-inspec/) - 3. push image to AWS ECR + 1. run tests using [`inspec`](https://community.chef.io/tools/chef-inspec/) + 1. push image to AWS ECR +* Helm chart: +* Deployment: Done by Control Panel when a user deploys JupyterLab for -- Helm chart: -- Deployment: Done by Control Panel when a user deploys JupyterLab for - themselves. This releases the helm chart above. +themselves. This releases the helm chart above. ## About Jupyter Notebook @@ -31,13 +25,12 @@ From [Jupter](http://jupyter.org): We currently have 3 flavours of JupyterLab: -- `datascience-notebook` is the standard image currently used by most of - our users -- `allspark-notebook` is similar to the `datascience-notebook` one but it +* `datascience-notebook` is the standard image currently used by most of our users +* `allspark-notebook` is similar to the `datascience-notebook` one but it includes Spark. This is currently used mainly by the Data Engineer team and it's [deployed manually](https://github.com/ministryofjustice/analytics-platform/wiki/Deploying-jupyterlab#spark-version). Long term we may use this image by default instead of the `datascience-notebook` one. -- `oracle-datascience-notebook` is a temporary image which contains drivers to +* `oracle-datascience-notebook` is a temporary image which contains drivers to connect to Oracle databases as part of some niche and temporary work. This image is hopefully going to disappear very soon. @@ -49,7 +42,7 @@ These images are derived from [jupyter/docker-stacks](https://github.com/jupyter From the sub-directory for the image you want to build -```bash +``` bash make build ``` @@ -63,7 +56,7 @@ the images and push to [our private AWS ECR registry](https://eu-west-1.console. Once the image is correctly pushed to this registry you can update the relevant [JupyterLab helm chart values](https://github.com/ministryofjustice/analytics-platform-helm-charts/blob/e2bc45e798ab97ee70a2f5a3cf52440648f23f81/charts/jupyter-lab/values.yaml#L31-L33) and/or update a specific user's -kubernetes `Deployment`. +kubernetes `Deployment` . If you're releasing a new version of the JupyterLab helm chart, talk with a Control Panel admin to be sure this new version is added to the Tools catalogue @@ -73,13 +66,13 @@ and users can deploy/upgrade JupyterLab and use it. In order to disable the authentication, we append `--NotebookApp.token=''` as an argument -```bash +``` bash docker container run -d --rm -p 8888:8888 jupyter/datascience-notebook start.sh jupyter lab --NotebookApp.token='' ``` ## Grant `sudo` with disabled authentication -```bash +``` bash docker container run -d --rm -p 8888:8888 -e GRANT_SUDO=yes jupyter/datascience-notebook start.sh jupyter lab --NotebookApp.token='' ``` @@ -87,5 +80,6 @@ docker container run -d --rm -p 8888:8888 -e GRANT_SUDO=yes jupyter/datascience- These images work, but with questions to resolve: -- The image creates a 'jovyan' user with UID 1000, we'll need to figure it out how this will work with the NFS home (rename/change user UID?) - - anaconda/jupyter dashboard are installed as this user, this may need to change +* The image creates a 'jovyan' user with UID 1000, we'll need to figure it + out how this will work with the NFS home (rename/change user UID?) +* anaconda/jupyter dashboard are installed as this user, this may need to change