diff --git a/lectures/containers/containers.Rmd b/lectures/containers/containers.Rmd deleted file mode 100644 index 574f19f0..00000000 --- a/lectures/containers/containers.Rmd +++ /dev/null @@ -1,286 +0,0 @@ ---- -title: "Containerization with Docker and Singularity" -subtitle: "Tools for Reproducible Research NBIS course" -output: - xaringan::moon_reader: - self-contained: true - seal: false - css: ["default", "../template.css"] - nature: - slideNumberFormat: "" ---- - -layout: true - - - ---- - -class: center, middle - -*Making self-contained, distributable projects with* - -.pull-left[ - -
- -] - -.pull-right[ - -
-Singularity -] - ---- -# What is Docker? - -* Standardized packaging for software and dependencies -* Docker lets you create and run applications securely isolated in a **container**, packaged with all its dependencies and libraries. - ---- - -# Docker in lifescience - -Docker containers can be used to allow others to reproduce a complete analysis: - -> To further support reproducibility, we bundled all tools and dependencies into one Docker container available on DockerHub [19]. *docker run* executes the aforementioned Makefile inside the container. - -.small[(click [here](https://github.com/metagenomics/2015-biogas-cebitec/blob/master/Dockerfile) to see the Dockerfile)] - -```bash -docker run metagenomics/2015-biogas-cebitec -``` - -.small[Bremges *et al* GigaScience (2015) 4:33, [doi:10.1186/s13742-015-0073-6](https://doi.org/10.1186/s13742-015-0073-6)] - ---- - -# Docker in lifescience - -Docker can also be used to define software environments and settings for benchmarking studies - -[CAMI Challenge](https://data.cami-challenge.org/): an independent evaluation of several tools in the field of metagenomics - ->... we defined standards for submitting the software itself, along with parameter settings and required databases and implemented them in Docker container templates... - -.small[ -Sczyrba *et al* Nature Methods (2017) 14:1063–1071, [doi.org/10.1038/nmeth.4458](https://doi.org/10.1038/nmeth.4458) -] - ---- - -# Docker in lifescience - -Docker can also be used to define software environments and settings for benchmarking studies - -[2nd CAMI Challenge](https://www.microbiome-cosi.org/cami/cami/cami2) - -> For reproducibility, participants could submit a Docker container containing the complete workflow, a bioconda script or a software repository with detailed installation instructions... - -.small[ -Meyer *et al* Nature Methods (2022) 19:429-440, -[https://doi.org/10.1038/s41592-022-01431-4](https://doi.org/10.1038/s41592-022-01431-4) -] - ---- - -# Docker nomenclature - --- - -* A Docker .green[file] is a recipe used to build a Docker .green[image] - --- - -* A Docker .green[image] is a standalone executable package of software - --- - -* A Docker .green[container] is a standard unit of software run on the Docker Engine. - --- - -* .green[Docker Hub] is an online service for sharing docker images - ---- - -# Images and containers - -Example: Use a different OS - -.pull-right[] - --- - -.pull-left[ - -.small[Check OS on local machine] - -```bash -$ uname -a -Darwin johnsmbp.local 19.6.0 \ -Darwin Kernel Version 19.6.0: \ -Tue Jan 12 22:13:05 PST 2021; \ -root:xnu-6153.141.16~1/RELEASE_X86_64 x86_64 -``` - -.small[Pull Ubuntu (Linux) image] - -```bash -$ docker pull ubuntu:16.04 -16.04: Pulling from library/ubuntu -22dc81ace0ea: Pull complete -... -Digest: sha256:e348fbbea0e0a0e73ab0370de151e7800684445c509d46195aef73e090a49bd6 -Status: Downloaded newer image for ubuntu:16.04 -``` - -.small[Run a container and check OS version] - -```bash -$ docker run -it ubuntu:16.04 -root@407b0fd13fe5:/# uname -a -Linux 407b0fd13fe5 4.9.60-linuxkit-aufs #1 SMP Mon Nov 6 16:00:12 UTC -2017 x86_64 x86_64 x86_64 GNU/Linux -``` -] - ---- - -# Mounting volumes - - -`$ docker run -it -v $PWD/data:/home/data ubuntu:16.04` - -.pull-left[ - -.tiny[Local project directory:] - -``` -$ ls project/ - -|- doc/ -| -|- data/ -| |- raw_external/ -| |- raw_internal/ -| |- meta/ -| -|- code/ -|- notebooks/ -| -|- intermediate/ -|- scratch/ -|- logs/ -| -|- results/ -| |- figures/ -| |- tables/ -| |- reports/ -| -|- Snakefile -|- config.yml -|- environment.yml -|- Dockerfile -``` -] -.pull-right[ -.tiny[Docker image file system] -``` -$ ls / - -|- bin/ -|- boot/ -|- dev/ -|- etc/ -|- home/ -| |- data/ # data folder mounted under home/ -| |- raw_external/ -| |- raw_internal/ -| |- meta/ -|- lib/ -|- lib64/ -|- media/ -|- opt/ -|- proc/ -|- root/ -|- run/ -|- sys/ -|- tmp/ -|- usr/ -|- var/ -``` -] - ---- - -# What can I use Docker for? - --- - -* As an advanced .green[environment] manager - -```bash -docker run -it -v $PWD:/home my_image /bin/bash -``` - --- - -* To package your .green[code] with the .green[environment] it needs - -```bash -docker run \ --v $PWD/data:/home/data \ --v $PWD/results:/home/results \ -my_image snakemake report.pdf -``` - --- - -* To package a whole workflow with .green[environment], .green[code] and .green[data] (*e.g.* to accompany a manuscript). - -```bash -docker run \ --v $PWD/results:/home/results \ -my_image snakemake report.pdf -``` - -* and much more... - ---- -# What is Singularity? - - - -an open source container platform suitable for HPC clusters - -[https://sylabs.io/guides/latest/user-guide/](https://sylabs.io/guides/latest/user-guide/) - ->Singularity was created to run complex applications on HPC clusters in a simple, portable, and reproducible way. - ---- -# What is Apptainer? - - - -In 2021 Singularity joined the Linux Foundation and became Apptainer. - -The company Sylabs still maintains **Singularity** - ---- - -# Docker vs. Singularity - -| | | -| ------ | ----------- | -| runs as a daemon process with superuser privileges | runs as regular user | -| images stored centrally | image files that you can move around. No layers! | -| isolates the host and container file system by default | containers have access to host file system | -| well supported on Mac/Linux/Windows | limited support on Mac/Windows | - ---- - -class: center, middle - -# Questions? diff --git a/lectures/containers/containers.html b/lectures/containers/containers.html new file mode 100644 index 00000000..ff9d1f7e --- /dev/null +++ b/lectures/containers/containers.html @@ -0,0 +1,2623 @@ + + + + + + + + + + + + + Working with containers + + + + + + + + + + + + + + + +
+
+ +
+

Working with containers

+ +
+
+ +
+
+

What is Docker?

+
+
    +
  • Standardized packaging for software, dependencies as well as the operating system
  • +
  • Docker lets you create and run applications securely isolated in containers
  • +
+
+
+
+

Docker in life science

+

Docker containers can be used to allow others to reproduce a complete analysis:

+
+

To further support reproducibility, we bundled all tools and dependencies into one Docker container available on DockerHub. docker run executes the aforementioned Makefile inside the container. 1

+
+


+
+

Reproducing the paper can now be done with one command:

+
docker run metagenomics/2015-biogas-cebitec
+
+
+
+

Docker in life science

+

Docker can also be used to define software environments and settings for benchmarking studies:

+


+

CAMI Challenge: an independent evaluation of several tools in the field of metagenomics.

+
+

[…] We defined standards for submitting the software itself, along with parameter settings and required databases and implemented them in Docker container templates … 1

+
+
+
+

Docker in life science

+

Docker can also be used to define software environments and settings for benchmarking studies:

+


+

2nd CAMI Challenge

+
+

For reproducibility, participants could submit a Docker container containing the complete workflow, a Bioconda script or a software repository with detailed installation instructions […] 1

+
+
+
+

Docker nomenclature

+
+
    +
  • A Docker image is a stand-alone executable package of software
  • +
  • A Docker container is an instance of a Docker image
  • +
  • A Dockerfile is a recipe used to build a Docker image
  • +
  • Docker Hub is an online service for hosting Docker images
  • +
+
+
+
+
+

+
+
+
+
+
+

An example: using a different OS

+
+

Check OS on local machine

+
$ uname -a
+Darwin johnsmbp.local 19.6.0 Darwin Kernel Version 19.6.0: [...] x86_64
+
+
+

Pull Ubuntu (Linux) image

+
$ docker pull ubuntu:16.04
+16.04: Pulling from library/ubuntu
+22dc81ace0ea: Pull complete
+...
+Digest: sha256:e348fbbea0e0a0e73ab0370de151e7800684445c509d46195aef73e090a49bd6
+Status: Downloaded image for ubuntu:16.04
+
+
+

Run the container interactively and check OS version

+
$ docker run -it ubuntu:16.04
+root@407b0fd13fe5:/## uname -a
+Linux 407b0fd13fe5 4.9.60-linuxkit-aufs [...] x86_64 GNU/Linux
+
+
+
+

Mounting volumes

+
+
+
+

Local project/ directory:

+
$ ls
+|- doc/
+|- data/
+|  |- raw_external/
+|  |- raw_internal/
+|  |- meta/
+|- code/
+|- notebooks/
+|- intermediate/
+|- scratch/
+|- logs/
+|- results/
+|- Snakefile
+|- config.yml
+|- environment.yml
+|- Dockerfile
+
+
+
+

Docker container file system:

+
$ docker run \
+    -v $PWD/data:/home/data \
+    ubuntu:16.04 ls
+|- bin/
+|- dev/
+|- etc/
+|- home/
+|  |- data/
+|     |- raw_external/
+|     |- raw_internal/
+|     |- meta/
+|- lib/
+|- opt/
+|- run/
+|- sys/
+|- usr/
+
+
+
+
+
+

What can I use Docker for?

+
+
    +
  • As an advanced environment manager
  • +
+
docker run -it -v $PWD:/home my_image /bin/bash
+
+
+
    +
  • To package your code with the environment it needs
  • +
+
docker run \
+    -v $PWD/data:/home/data \
+    -v $PWD/results:/home/results \
+    my_image snakemake report.pdf
+
+
+
    +
  • To package a whole project with environment, code and data (e.g. to accompany a manuscript).
  • +
+
docker run \
+    -v $PWD/results:/home/results \
+    my_image snakemake report.pdf
+
+
+
+

What is Singularity?

+
+
+

+
+
+
+
    +
  • Another software for working with containers, similar to Docker
  • +
  • A Singularity image is contained in a single file, facilitating portability
  • +
  • Does not require root access, making it suitable for work at HPCs
  • +
+
+
+
+

What is Apptainer?

+ +
+
    +
  • In 2021 Singularity joined the Linux Foundation and became Apptainer.
  • +
  • The company Sylabs still maintains Singularity as a commercial piece of software.
  • +
  • SingularityCE is an Open Source project supported by Sylabs.
  • +
  • The three versions are similar today, but will diverge with time.
  • +
  • Apptainer is the most commonly adopted version of the scientific community.
  • +
+
+
+
+

Docker vs. Apptainer

+
+
+
+
+

+
+
+
+
+
+

+
+
+
+
+ ++++ + + + + + + + + + + + + + + + + + + +
Runs as a daemon process with superuser privilegesRuns as a regular user
Images stored centrallyImage files that you can move around.
Isolates the host and container file system by defaultContainers have access to host file system
Well-supported on Mac, Linux and WindowsLimited support on Mac and Windows
+
+
+

Questions?

+ + +
+ + +
+
+ + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/lectures/containers/containers.pdf b/lectures/containers/containers.pdf deleted file mode 100644 index 8a10bdc1..00000000 Binary files a/lectures/containers/containers.pdf and /dev/null differ diff --git a/lectures/containers/containers.qmd b/lectures/containers/containers.qmd new file mode 100644 index 00000000..db819bfd --- /dev/null +++ b/lectures/containers/containers.qmd @@ -0,0 +1,238 @@ +--- +title: 'Working with containers + + + ' +format: nbis-revealjs +--- + +## What is [Docker]{.green}? + +::: {.incremental} +- Standardized packaging for software, dependencies as well as the operating + system +- Docker lets you create and run applications securely isolated in + [containers]{.green} +::: + +## Docker in life science + +Docker containers can be used to allow others to reproduce a complete analysis: + +> To further support reproducibility, we bundled all tools and dependencies into +> one Docker container available on DockerHub. `docker run` executes the +> aforementioned Makefile inside the container. ^[_Bremges et al GigaScience +> (2015) 4:33, [doi:10.1186/s13742-015-0073-6](https://doi.org/10.1186/s13742-015-0073-6)_] + +
+ +::: {.fragment} +Reproducing the paper can now be done with one command: +```bash +docker run metagenomics/2015-biogas-cebitec +``` +::: + +## Docker in life science + +Docker can also be used to define software environments and settings for +benchmarking studies: + +
+ +[CAMI Challenge](https://data.cami-challenge.org/): an independent evaluation of +several tools in the field of metagenomics. + +> [...] We defined standards for submitting the software itself, along with +> parameter settings and required databases and implemented them in Docker +> container templates ... ^[_Sczyrba et al Nature Methods (2017) 14:1063–1071, +> [doi.org/10.1038/nmeth.4458](https://doi.org/10.1038/nmeth.4458)_] + +## Docker in life science + +Docker can also be used to define software environments and settings for +benchmarking studies: + +
+ +[2nd CAMI Challenge](https://www.microbiome-cosi.org/cami/cami/cami2) + +> For reproducibility, participants could submit a Docker container containing +> the complete workflow, a Bioconda script or a software repository with +> detailed installation instructions [...] ^[_Meyer et al Nature Methods (2022) +> 19:429-440, [https://doi.org/10.1038/s41592-022-01431-4](https://doi.org/10.1038/s41592-022-01431-4)_] + +## Docker nomenclature + +::: {.incremental} +- A Docker [image]{.green} is a stand-alone executable package of software +- A Docker [container]{.green} is an instance of a Docker image +- A [Dockerfile]{.green} is a recipe used to build a Docker image +- [Docker Hub]{.green} is an online service for hosting Docker images +::: + +::: {.fragment} +![](images/images_and_containers.png){height=350 fig-align="center"} +::: + +## An example: using a different OS + +::: {.fragment} +Check OS on local machine +```{.bash code-line-numbers="|1|2"} +$ uname -a +Darwin johnsmbp.local 19.6.0 Darwin Kernel Version 19.6.0: [...] x86_64 +``` +::: + +::: {.fragment} +Pull Ubuntu (Linux) image +```{.bash code-line-numbers="|1|2-6"} +$ docker pull ubuntu:16.04 +16.04: Pulling from library/ubuntu +22dc81ace0ea: Pull complete +... +Digest: sha256:e348fbbea0e0a0e73ab0370de151e7800684445c509d46195aef73e090a49bd6 +Status: Downloaded image for ubuntu:16.04 +``` +::: + +::: {.fragment} +Run the container interactively and check OS version +```{.bash code-line-numbers="|1|2|3"} +$ docker run -it ubuntu:16.04 +root@407b0fd13fe5:/## uname -a +Linux 407b0fd13fe5 4.9.60-linuxkit-aufs [...] x86_64 GNU/Linux +``` +::: + +## Mounting volumes + +:::: {.columns height=90%} + +::: {.column} +::: {.fragment} +Local `project/` directory: +```{.bash code-line-numbers="1|2-16|3-6"} +$ ls +|- doc/ +|- data/ +| |- raw_external/ +| |- raw_internal/ +| |- meta/ +|- code/ +|- notebooks/ +|- intermediate/ +|- scratch/ +|- logs/ +|- results/ +|- Snakefile +|- config.yml +|- environment.yml +|- Dockerfile +``` +::: +::: + +::: {.column} +::: {.fragment} +Docker container file system: +```{.bash code-line-numbers="1-3|4-16|7-11"} +$ docker run \ + -v $PWD/data:/home/data \ + ubuntu:16.04 ls +|- bin/ +|- dev/ +|- etc/ +|- home/ +| |- data/ +| |- raw_external/ +| |- raw_internal/ +| |- meta/ +|- lib/ +|- opt/ +|- run/ +|- sys/ +|- usr/ +``` +::: +::: + +:::: + +## What can I use Docker for? + +::: {.fragment} +- As an advanced [environment]{.green} manager + +```bash +docker run -it -v $PWD:/home my_image /bin/bash +``` +::: + +::: {.fragment} +- To package your [code]{.green} with the environment it needs + +```bash +docker run \ + -v $PWD/data:/home/data \ + -v $PWD/results:/home/results \ + my_image snakemake report.pdf +``` +::: + +::: {.fragment} +- To package a [whole project]{.green} with environment, code and data (_e.g._ + to accompany a manuscript). + +```bash +docker run \ + -v $PWD/results:/home/results \ + my_image snakemake report.pdf +``` +::: + +## What is Singularity? + +![](https://sylabs.io/guides/3.0/user-guide/_static/logo.png){height=250 fig-align=center} + +::: {.incremental} +- Another software for working with containers, similar to Docker +- A Singularity image is contained in a single file, facilitating portability +- Does not require `root` access, making it suitable for work at HPCs +::: + +## What is Apptainer? + +![](http://apptainer.org/static/hero-img-e614806cf573fd0cbbe09a25d9e69637.png){fig-align=center} + +::: {.incremental} +- In 2021 Singularity joined the Linux Foundation and became + [Apptainer]{.green}. +- The company [Sylabs]{.green} still maintains Singularity as a commercial +piece of software. +- [SingularityCE]{.green} is an Open Source project supported by Sylabs. +- The three versions are similar today, but will diverge with time. +- [Apptainer]{.green} is the most commonly adopted version of the scientific + community. +::: + +## Docker vs. Apptainer + +:::: {.columns} +::: {.column} +![](images/docker-logo.webp){height=150 fig-align=center} +::: +::: {.column} +![](https://apptainer.org/docs/user/main/_static/logo.png){height=150 fig-align=center} +::: +:::: + +| | | +| ------------------------------------------------------- | -------------------------------------------------------- | +| Runs as a daemon process with superuser privileges | Runs as a regular user | +| Images stored centrally | Image files that you can move around. | +| Isolates the host and container file system by default | Containers have access to host file system | +| Well-supported on Mac, Linux and Windows | Limited support on Mac and Windows | + +# Questions? diff --git a/lectures/containers/images/docker-logo.webp b/lectures/containers/images/docker-logo.webp new file mode 100644 index 00000000..88bd2d7d Binary files /dev/null and b/lectures/containers/images/docker-logo.webp differ diff --git a/lectures/containers/template.css b/lectures/containers/template.css deleted file mode 100644 index 47a1ac43..00000000 --- a/lectures/containers/template.css +++ /dev/null @@ -1,124 +0,0 @@ -/* CSS style general NBIS presentations using Xaringen and RMarkdown */ - -/* Import Google font */ -@import url('https://fonts.googleapis.com/css?family=Open+Sans&display=swap'); - -/* Fonts */ -body,p,h1,h2,h3,h4,h5,h6 { - font-family: 'Open Sans', sans-serif; - font-weight: 400; - font-size: 20px; - color: #333333; -} - -/* Header 1 font size */ -h1 { - font-size: 1.7em !important; -} - -/* Header 2 font size */ -h2 { - font-size: 1.2em !important; -} - -/* Header 3 font size */ -h3 { - font-size: 0.9em !important; -} - -/* Custom font sizes */ -.micro { font-size: 25% } -.tiny { font-size: 50% } -.small { font-size: 75% } -.large { font-size: 125% } -.huge { font-size: 150% } -.Huge { font-size: 200% } -.HUGE { font-size: 250% } - -.remark-code { font-size: 66% } - -/* Custom colours */ -.green { color: #85be42; font-weight: bold } -.dark-green { color: #487a2d; font-weight: bold } -.blue { color: #0099c2; font-weight: bold } -.dark-blue { color: #015491; font-weight: bold } -.light-grey { color: #999999 } -.grey { color: #666666 } -.dark-grey { color: #333333 } - -/* Set the maximum figure size to 100% of the slide */ -img, video, iframe { - max-width: 100%; - max-height: 100%; -} - -table_rot { - color: black; - text-transform: uppercase; - @include transform(rotate(270deg)) - @include transform-origin(0 0); -} - -/* SciLifeLab logo in bottom left corner */ -div.scilife-logo { - content: ""; - background-image: url("https://www.scilifelab.se/wp-content/uploads/2020/09/SciLifeLab_Logotype_Green_POS_large-copy-1536x334.png"); - background-repeat: no-repeat; - background-size: contain; - position: absolute; - bottom: 10px; - left: 15px; - height: 50px; - width: 200px; -} - -/* NBIS logo in bottom right corner */ -div.nbis-logo { - content: ""; - background-image: url("https://nbis.se/assets/img/logos/nbislogo-green.svg"); - background-repeat: no-repeat; - background-size: contain; - position: absolute; - bottom: 23px; - right: 27px; - height: 35px; - width: 100px; -} - -/* Blockquotes */ -blockquote { - border-left: 5px solid #85be42; - background: #f9f9f9; - font-style: italic; - padding: 0.01em; - padding-left: 12px; - padding-right: 12px; - margin-left: 12px; - margin-right: 12px; - border-radius: 0 5px 5px 0; -} - -/* Link colour */ -a { - color: #85be42; - text-decoration: none; -} - -/* Underline links when hovering */ -a:hover { - text-decoration: underline; -} - -/* Inline code color and border */ -.remark-inline-code { - background: #f2f2f2; - border-radius: 3px; - margin-right: 2px; - padding: 2px 3px 1px; - font-size: 80% !important; -} - -.image-with-text { - display: inline-block; - vertical-align: middle; -}