Appendix

Apptainer’s environment variables

Apptainer comes with some environment variables you can set or modify depending on your needs. You can see them listed alphabetically below with their respective functionality.

`A`

APPTAINER_ADD_CAPS: To specify a list (comma separated string) of capabilities to be added. Default is an empty string.
APPTAINER_ALL: List all the users and groups capabilities.
APPTAINER_ALLOW_SETUID: To specify that setuid binaries should or not be allowed in the container. (root only) Default is set to false.
APPTAINER_APP and APPTAINER_APPNAME: Sets the name of an application to be run inside a container.
APPTAINER_APPLY_CGROUPS: Used to apply cgroups from an input file for container processes. (it requires root privileges)

`B`

APPTAINER_BIND and APPTAINER_BINDPATH: Comma separated string source:<dest> list of paths to bind between the host and the container.
APPTAINER_BOOT: Set to false by default, considers if executing /sbin/init when container boots (root only).

`C`

APPTAINER_CACHEDIR: Specifies the directory for image downloads to be cached in. See Cache Folders.
APPTAINER_CLEANENV: Specifies if the environment should be cleaned or not before running the container. Default is set to false.
APPTAINER_CONTAIN: To use minimal /dev and empty other directories (e.g. /tmp and $HOME) instead of sharing filesystems from your host. Default is set to false.
APPTAINER_CONTAINALL: To contain not only file systems, but also PID, IPC, and environment. Default is set to false.
APPTAINER_CONTAINLIBS: Used to specify a string of file names (comma separated string) to bind to the /.singularity.d/libs directory.

`D`

APPTAINER_DEFFILE: Shows the Apptainer recipe that was used to generate the image.
APPTAINER_DESC: Contains a description of the capabilities.
APPTAINER_DISABLE_CACHE: To disable all caching of docker/oci, library, oras, etc. downloads and built SIFs. Default is set to false.
APPTAINER_DNS: A list of the DNS server addresses separated by commas to be added in resolv.conf.
APPTAINER_DOCKER_LOGIN: To specify the interactive prompt for docker authentication.
APPTAINER_DOCKER_USERNAME: To specify a username for docker authentication.
APPTAINER_DOCKER_PASSWORD: To specify the password for docker authentication.
APPTAINER_DOWNLOAD_CONCURRENCY: To specify how many concurrent streams when downloading (pulling) an image from cloud library.
APPTAINER_DOWNLOAD_PART_SIZE: To specify the size of each part (bytes) when concurrent downloads are enabled.
APPTAINER_DOWNLOAD_BUFFER_SIZE: To specify the transfer buffer size (bytes) when concurrent downloads are enabled.
APPTAINER_DROP_CAPS: To specify a list (comma separated string) of capabilities to be dropped. Default is an empty string.

`E`

APPTAINER_ENVIRONMENT: Contains all the environment variables that have been exported in your container.
APPTAINER_ENCRYPTION_PASSPHRASE: Used to specify the plaintext passphrase to encrypt the container.
APPTAINER_ENCRYPTION_PEM_PATH: Used to specify the path of the file containing public or private key to encrypt the container in PEM format.
APPTAINERENV_*: Allows you to transpose variables into the container at runtime. You can see more in detail how to use this variable in our environment and metadata section.
APPTAINERENV_APPEND_PATH: Used to append directories to the end of the $PATH environment variable. You can see more in detail on how to use this variable in our environment and metadata section.
APPTAINERENV_PATH: A specified path to override the $PATH environment variable within the container. You can see more in detail on how to use this variable in our environment and metadata section.
APPTAINERENV_PREPEND_PATH: Used to prepend directories to the beginning of $PATH` environment variable. You can see more in detail on how to use this variable in our environment and metadata section.

`F`

APPTAINER_FAKEROOT: Set to false by default, considers running the container in a new user namespace as uid 0 (experimental).
APPTAINER_FORCE: Forces to kill the instance.

`G`

APPTAINER_GROUP: Used to specify a string of capabilities for the given group.

`H`

APPTAINER_HELPFILE: Specifies the runscript helpfile, if it exists.
APPTAINER_HOME : A home directory specification, it could be a source or destination path. The source path is the home directory outside the container and the destination overrides the home directory within the container.
APPTAINER_HOSTNAME: The container’s hostname.

`I`

APPTAINER_IMAGE: Filename of the container.

`J`

APPTAINER_JSON: Specifies the structured json of the def file, every node as each section in the def file.

`K`

APPTAINER_KEEP_PRIVS: To let root user keep privileges in the container. Default is set to false.

`L`

APPTAINER_LABELS: Specifies the labels associated with the image.
APPTAINER_LIBRARY: Specifies the library to pull from. Default is set to our Cloud Library.

`M`

APPTAINER_MOUNT: To specify host to container mounts, using the syntax understood by the --mount flag. Multiple mounts should be separated by newline characters.

`N`

APPTAINER_NAME: Specifies a custom image name.
APPTAINER_NETWORK: Used to specify a desired network. If more than one parameters is used, addresses should be separated by commas, where each network will bring up a dedicated interface inside the container.
APPTAINER_NETWORK_ARGS: To specify the network arguments to pass to CNI plugins.
APPTAINER_NOCLEANUP: To not clean up the bundle after a failed build, this can be helpful for debugging. Default is set to false.
APPTAINER_NOHTTPS: Sets to either false or true to avoid using HTTPS for communicating with the local docker registry. Default is set to false.
APPTAINER_NO_HOME: Considers not mounting users home directory if home is not the current working directory. Default is set to false.
APPTAINER_NO_INIT and APPTAINER_NOSHIMINIT: Considers not starting the shim process with --pid.
APPTAINER_NO_NV: Flag to disable Nvidia support. Opposite of APPTAINER_NV.
APPTAINER_NO_PRIVS: To drop all the privileges from root user in the container. Default is set to false.
APPTAINER_NV: To enable Nvidia GPU support. Default is set to false.
APPTAINER_NVCCLI: To use nvidia-container-cli for container GPU setup (experimental, only unprivileged).

`O`

APPTAINER_OVERLAY and APPTAINER_OVERLAYIMAGE: To indicate the use of an overlay file system image for persistent data storage or as read-only layer of container.

`P`

APPTAINER_PWD and APPTAINER_TARGET_PWD: The initial working directory for payload process inside the container.

`R`

APPTAINER_ROOTFS: To reference the system file location.
APPTAINER_RUNSCRIPT: Specifies the runscript of the image.

`S`

APPTAINER_SANDBOX: To specify that the format of the image should be a sandbox. Default is set to false.
APPTAINER_SCRATCH and APPTAINER_SCRATCHDIR: Used to include a scratch directory within the container that is linked to a temporary directory. (use -W to force location)
APPTAINER_SECTION: To specify a comma separated string of all the sections to be run from the deffile (setup, post, files, environment, test, labels, none)
APPTAINER_SECURITY: Used to enable security features. (SELinux, Apparmor, Seccomp)
APPTAINER_SECRET: Lists all the private keys instead of the default which display the public ones.
APPTAINER_SHELL: The path to the program to be used as an interactive shell.
APPTAINER_SIGNAL: Specifies a signal sent to the instance.

`T`

APPTAINER_TEST: Specifies the test script for the image.
APPTAINER_TMPDIR: Used with the build command, to consider a temporary location for the build. See Temporary Folders.

`U`

APPTAINER_UNSHARE_PID: To specify that the container will run in a new PID namespace. Default is set to false.
APPTAINER_UNSHARE_IPC: To specify that the container will run in a new IPC namespace. Default is set to false.
APPTAINER_UNSHARE_NET: To specify that the container will run in a new network namespace (sets up a bridge network interface by default). Default is set to false.
APPTAINER_UNSHARE_UTS: To specify that the container will run in a new UTS namespace. Default is set to false.
APPTAINER_UPDATE: To run the definition over an existing container (skips the header). Default is set to false.
APPTAINER_URL: Specifies the key server URL.
APPTAINER_USER: Used to specify a string of capabilities for the given user.
APPTAINER_USERNS and APPTAINER_UNSHARE_USERNS: To specify that the container will run in a new user namespace, allowing Apptainer to run completely unprivileged on recent kernels. This may not support every feature of Apptainer. (Sandbox image only). Default is set to false.

`W`

APPTAINER_WORKDIR: The working directory to be used for /tmp, /var/tmp and $HOME (if -c or --contain was also used)
APPTAINER_WRITABLE: By default, all Apptainer containers are available as read only, this option makes the file system accessible as read/write. Default set to false.
APPTAINER_WRITABLE_TMPFS: Makes the file system accessible as read-write with non-persistent data (with overlay support only). Default is set to false.

Build Modules

`library` bootstrap agent

Overview

You can use an existing container on the Container Library as your “base,” and then add customization. This allows you to build multiple images from the same starting point. For example, you may want to build several containers with the same custom python installation, the same custom compiler toolchain, or the same base MPI installation. Instead of building these from scratch each time, you could create a base container on the Container Library and then build new containers from that existing base container adding customizations in %post, %environment, %runscript, etc.

Keywords

Bootstrap: library

The Bootstrap keyword is always mandatory. It describes the bootstrap module to use.

From: <entity>/<collection>/<container>:<tag>

The From keyword is mandatory. It specifies the container to use as a base. entity is optional and defaults to library. collection is optional and defaults to default. This is the correct namespace to use for some official containers (alpine for example). tag is also optional and will default to latest.

Library: http://custom/library

The Library keyword is optional. It will default to https://library.sylabs.io.

Fingerprints: 22045C8C0B1004D058DE4BEDA20C27EE7FF7BA84

The Fingerprints keyword is optional. It specifies one or more comma separated fingerprints corresponding to PGP public keys. If present, the bootstrap image will be verified and the build will only proceed if it is signed by keys matching all of the specified fingerprints.

`docker` bootstrap agent

Overview

Docker images are comprised of layers that are assembled at runtime to create an image. You can use Docker layers to create a base image, and then add your own custom software. For example, you might use Docker’s Ubuntu image layers to create an Ubuntu Apptainer container. You could do the same with CentOS, Debian, Arch, Suse, Alpine, BusyBox, etc.

Or maybe you want a container that already has software installed. For instance, maybe you want to build a container that uses CUDA and cuDNN to leverage the GPU, but you don’t want to install from scratch. You can start with one of the nvidia/cuda containers and install your software on top of that.

Or perhaps you have already invested in Docker and created your own Docker containers. If so, you can seamlessly convert them to Apptainer with the docker bootstrap module.

Keywords

Bootstrap: docker

The Bootstrap keyword is always mandatory. It describes the bootstrap module to use.

From: <registry>/<namespace>/<container>:<tag>@<digest>

The From keyword is mandatory. It specifies the container to use as a base. registry is optional and defaults to index.docker.io. namespace is optional and defaults to library. This is the correct namespace to use for some official containers (ubuntu for example). tag is also optional and will default to latest

See Apptainer and Docker for more detailed info on using Docker registries.

Registry: http://custom_registry

The Registry keyword is optional. It will default to index.docker.io.

Namespace: namespace

The Namespace keyword is optional. It will default to library.

Notes

Docker containers are stored as a collection of tarballs called layers. When building from a Docker container the layers must be downloaded and then assembled in the proper order to produce a viable file system. Then the file system must be converted to Singularity Image File (sif) format.

Building from Docker Hub is not considered reproducible because if any of the layers of the image are changed, the container will change. If reproducibility is important to your workflow, consider hosting a base container on the Container Library and building from it instead.

For detailed information about setting your build environment see Build Customization.

`shub` bootstrap agent

Overview

You can use an existing container on Singularity Hub as your “base,” and then add customization. This allows you to build multiple images from the same starting point. For example, you may want to build several containers with the same custom python installation, the same custom compiler toolchain, or the same base MPI installation. Instead of building these from scratch each time, you could create a base container on Singularity Hub and then build new containers from that existing base container adding customizations in %post , %environment, %runscript, etc.

Keywords

Bootstrap: shub

The Bootstrap keyword is always mandatory. It describes the bootstrap module to use.

From: shub://<registry>/<username>/<container-name>:<tag>@digest

The From keyword is mandatory. It specifies the container to use as a base. registry is optional and defaults to ``singularity-hub.org. tag and digest are also optional. tag defaults to latest and digest can be left blank if you want the latest build.

Notes

When bootstrapping from a Singularity Hub image, all previous definition files that led to the creation of the current image will be stored in a directory within the container called /.singularity.d/bootstrap_history. Apptainer will also alert you if environment variables have been changed between the base image and the new image during bootstrap.

`oras` bootstrap agent

Overview

Using, this module, a container from supporting OCI Registries - Eg: ACR (Azure Container Registry), local container registries, etc can be used as your “base” image and later customized. This allows you to build multiple images from the same starting point. For example, you may want to build several containers with the same custom python installation, the same custom compiler toolchain, or the same base MPI installation. Instead of building these from scratch each time, you could make use of oras to pull an appropriate base container and then build new containers by adding customizations in %post , %environment, %runscript, etc.

Keywords

Bootstrap: oras

The Bootstrap keyword is always mandatory. It describes the bootstrap module to use.

From: oras://registry/namespace/image:tag

The From keyword is mandatory. It specifies the container to use as a base. Also,``tag`` is mandatory that refers to the version of image you want to use.

`localimage` bootstrap agent

This module allows you to build a container from an existing Apptainer container on your host system. The name is somewhat misleading because your container can be in either image or directory format.

Overview

You can use an existing container image as your “base”, and then add customization. This allows you to build multiple images from the same starting point. For example, you may want to build several containers with the same custom python installation, the same custom compiler toolchain, or the same base MPI installation. Instead of building these from scratch each time, you could start with the appropriate local base container and then customize the new container in %post, %environment, %runscript, etc.

Keywords

Bootstrap: localimage

The Bootstrap keyword is always mandatory. It describes the bootstrap module to use.

From: /path/to/container/file/or/directory

The From keyword is mandatory. It specifies the local container to use as a base.

Fingerprints: 22045C8C0B1004D058DE4BEDA20C27EE7FF7BA84

The Fingerprints keyword is optional. It specifies one or more comma separated fingerprints corresponding to PGP public keys. If present, and the From: keyword points to a SIF format image, it will be verified and the build will only proceed if it is signed by keys matching all of the specified fingerprints.

Notes

When building from a local container, all previous definition files that led to the creation of the current container will be stored in a directory within the container called /.singularity.d/bootstrap_history. Apptainer will also alert you if environment variables have been changed between the base image and the new image during bootstrap.

`yum` bootstrap agent

This module allows you to build a Red Hat/CentOS/Scientific Linux style container from a mirror URI.

Overview

Use the yum module to specify a base for a CentOS-like container. You must also specify the URI for the mirror you would like to use.

Keywords

Bootstrap: yum

The Bootstrap keyword is always mandatory. It describes the bootstrap module to use.

OSVersion: 7

The OSVersion keyword is optional. It specifies the OS version you would like to use. It is only required if you have specified a %{OSVERSION} variable in the MirrorURL keyword.

MirrorURL: http://mirror.centos.org/centos-%{OSVERSION}/%{OSVERSION}/os/$basearch/

The MirrorURL keyword is mandatory. It specifies the URI to use as a mirror to download the OS. If you define the OSVersion keyword, then you can use it in the URI as in the example above.

Include: yum

The Include keyword is optional. It allows you to install additional packages into the core operating system. It is a best practice to supply only the bare essentials such that the %post section has what it needs to properly complete the build. One common package you may want to install when using the yum build module is YUM itself.

Notes

There is a major limitation with using YUM to bootstrap a container. The RPM database that exists within the container will be created using the RPM library and Berkeley DB implementation that exists on the host system. If the RPM implementation inside the container is not compatible with the RPM database that was used to create the container, RPM and YUM commands inside the container may fail. This issue can be easily demonstrated by bootstrapping an older RHEL compatible image by a newer one (e.g. bootstrap a Centos 5 or 6 container from a Centos 7 host).

In order to use the yum build module, you must have yum installed on your system. It may seem counter-intuitive to install YUM on a system that uses a different package manager, but you can do so. For instance, on Ubuntu you can install it like so:

$ sudo apt-get update && sudo apt-get install yum

`debootstrap` build agent

This module allows you to build a Debian/Ubuntu style container from a mirror URI.

Overview

Use the debootstrap module to specify a base for a Debian-like container. You must also specify the OS version and a URI for the mirror you would like to use.

Keywords

Bootstrap: debootstrap

The Bootstrap keyword is always mandatory. It describes the bootstrap module to use.

OSVersion: xenial

The OSVersion keyword is mandatory. It specifies the OS version you would like to use. For Ubuntu you can use code words like trusty (14.04), xenial (16.04), and yakkety (17.04). For Debian you can use values like stable, oldstable, testing, and unstable or code words like wheezy (7), jesse (8), and stretch (9).

MirrorURL:  http://us.archive.ubuntu.com/ubuntu/

The MirrorURL keyword is mandatory. It specifies a URI to use as a mirror when downloading the OS.

Include: somepackage

The Include keyword is optional. It allows you to install additional packages into the core operating system. It is a best practice to supply only the bare essentials such that the %post section has what it needs to properly complete the build.

Notes

In order to use the debootstrap build module, you must have debootstrap installed on your system. On Ubuntu you can install it like so:

$ sudo apt-get update && sudo apt-get install debootstrap

On CentOS you can install it from the epel repos like so:

$ sudo yum update && sudo yum install epel-release && sudo yum install debootstrap.noarch

`arch` bootstrap agent

This module allows you to build a Arch Linux based container.

Overview

Use the arch module to specify a base for an Arch Linux based container. Arch Linux uses the aptly named pacman package manager (all puns intended).

Keywords

Bootstrap: arch

The Bootstrap keyword is always mandatory. It describes the bootstrap module to use.

The Arch Linux bootstrap module does not name any additional keywords at this time. By defining the arch module, you have essentially given all of the information necessary for that particular bootstrap module to build a core operating system.

Notes

Arch Linux is, by design, a very stripped down, light-weight OS. You may need to perform a significant amount of configuration to get a usable OS. Please refer to this README.md and the Arch Linux example for more info.

`busybox` bootstrap agent

This module allows you to build a container based on BusyBox.

Overview

Use the busybox module to specify a BusyBox base for container. You must also specify a URI for the mirror you would like to use.

Keywords

Bootstrap: busybox

The Bootstrap keyword is always mandatory. It describes the bootstrap module to use.

MirrorURL: https://www.busybox.net/downloads/binaries/1.35.0-x86_64-linux-musl/busybox

The MirrorURL keyword is mandatory. It specifies a URI to use as a mirror when downloading the OS.

Notes

You can build a fully functional BusyBox container that only takes up ~700kB of disk space!

`zypper` bootstrap agent

This module allows you to build a Suse style container from a mirror URI.

Note

zypper version 1.11.20 or greater is required on the host system, as Apptainer requires the --releasever flag.

Overview

Use the zypper module to specify a base for a Suse-like container. You must also specify a URI for the mirror you would like to use.

Keywords

Bootstrap: zypper

The Bootstrap keyword is always mandatory. It describes the bootstrap module to use.

OSVersion: 42.2

The OSVersion keyword is optional. It specifies the OS version you would like to use. It is only required if you have specified a %{OSVERSION} variable in the MirrorURL keyword.

Include: somepackage

The Include keyword is optional. It allows you to install additional packages into the core operating system. It is a best practice to supply only the bare essentials such that the %post section has what it needs to properly complete the build. One common package you may want to install when using the zypper build module is zypper itself.

`docker-daemon` and `docker-archive` bootstrap agents

If you are using docker locally there are two options for creating Apptainer images without the need for a repository. You can either build a SIF from a docker-save tar file or you can convert any docker image present in docker’s daemon internal storage.

Overview

docker-daemon allows you to build a SIF from any docker image currently residing in docker’s daemon internal storage:

$ docker images alpine
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
alpine              latest              965ea09ff2eb        7 weeks ago         5.55MB

$ apptainer run docker-daemon:alpine:latest
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
Getting image source signatures
Copying blob 77cae8ab23bf done
Copying config 759e71f0d3 done
Writing manifest to image destination
Storing signatures
2019/12/11 14:53:24  info unpack layer: sha256:eb7c47c7f0fd0054242f35366d166e6b041dfb0b89e5f93a82ad3a3206222502
INFO:    Creating SIF file...
Apptainer>

while docker-archive permits you to do the same thing starting from a docker image stored in a docker-save formatted tar file:

$ docker save -o alpine.tar alpine:latest

$ apptainer run docker-archive:$(pwd)/alpine.tar
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
Getting image source signatures
Copying blob 77cae8ab23bf done
Copying config 759e71f0d3 done
Writing manifest to image destination
Storing signatures
2019/12/11 15:25:09  info unpack layer: sha256:eb7c47c7f0fd0054242f35366d166e6b041dfb0b89e5f93a82ad3a3206222502
INFO:    Creating SIF file...
Apptainer>

Keywords

The docker-daemon bootstrap agent can be used in an Apptainer definition file as follows:

From: docker-daemon:<image>:<tag>

where both <image> and <tag> are mandatory fields that must be written explicitly. The docker-archive bootstrap agent requires instead the path to the tar file containing the image:

From: docker-archive:<path-to-tar-file>

Note that differently from the docker:// bootstrap agent both docker-daemon and docker-archive don’t require a double slash // after the colon in the agent name.

`scratch` bootstrap agent

The scratch bootstrap agent allows you to start from a completely empty container. You are then responsible for adding any and all executables, libraries etc. that are required. Starting with a scratch container can be useful when you are aiming to minimize container size, and have a simple application / static binaries.

Overview

A minimal container providing a shell can be created by copying the busybox static binary into an empty scratch container:

Bootstrap: scratch

%setup
    # Runs on host - fetch static busybox binary
    curl -o /tmp/busybox https://www.busybox.net/downloads/binaries/1.31.0-i686-uclibc/busybox
    # It needs to be executable
    chmod +x /tmp/busybox

%files
    # Copy from host into empty container
    /tmp/busybox /bin/sh

%runscript
   /bin/sh

The resulting container provides a shell, and is 696KiB in size:

$ ls -lah scratch.sif
-rwxr-xr-x. 1 dave dave 696K May 28 13:29 scratch.sif

$ apptainer run scratch.sif
WARNING: passwd file doesn't exist in container, not updating
WARNING: group file doesn't exist in container, not updating
Apptainer> echo "Hello from a 696KiB container"
Hello from a 696KiB container

Keywords

Bootstrap: scratch

There are no additional keywords for the scratch bootstrap agent.

Appendix

Apptainer’s environment variables

A

B

C

D

E

F

G

H

I

J

K

L

M

N

O

P

R

S

T

U

W

Build Modules

library bootstrap agent

Overview

Keywords

docker bootstrap agent

Overview

Keywords

Notes

shub bootstrap agent

Overview

Keywords

Notes

oras bootstrap agent

Overview

Keywords

localimage bootstrap agent

Overview

Keywords

Notes

yum bootstrap agent

Overview

Keywords

Notes

debootstrap build agent

Overview

Keywords

Notes

arch bootstrap agent

Overview

Keywords

Notes

busybox bootstrap agent

Overview

Keywords

Notes

zypper bootstrap agent

Overview

Keywords

docker-daemon and docker-archive bootstrap agents

Overview

Keywords

scratch bootstrap agent

Overview

`A`

`B`

`C`

`D`

`E`

`F`

`G`

`H`

`I`

`J`

`K`

`L`

`M`

`N`

`O`

`P`

`R`

`S`

`T`

`U`

`W`

`library` bootstrap agent

`docker` bootstrap agent

`shub` bootstrap agent

`oras` bootstrap agent

`localimage` bootstrap agent

`yum` bootstrap agent

`debootstrap` build agent

`arch` bootstrap agent

`busybox` bootstrap agent

`zypper` bootstrap agent

`docker-daemon` and `docker-archive` bootstrap agents

`scratch` bootstrap agent