Build Environment¶

Overview¶

You may wish to customize your build environment by doing things such as specifying a custom cache directory for images or sending your Docker Credentials to the registry endpoint. Here we will discuss these and other topics related to the build environment.

Cache Folders¶

To make downloading images for build and pull faster and less redundant, Singularity uses a caching strategy. By default, Singularity will create a set of folders in your $HOME directory for docker layers, Cloud library images, and metadata, respectively:

$HOME/.singularity/cache/library
$HOME/.singularity/cache/oci
$HOME/.singularity/cache/oci-tmp

If you want to cache in a different directory, set SINGULARITY_CACHEDIR to the desired path. By using the -E option with the sudo command, SINGULARITY_CACHEDIR will be passed along to root’s environment and respected during the build. Remember that when you run commands as root images will be cached in root’s home at /root and not your user’s home.

Cache commands¶

Singularity 3.1 comes with new commands for cleaning and listing the cache image files generated.

Note

Running the cache commands with sudo privilege will consider cache location as /root/.singularity/cache. The default location for cache without sudo privilege is ~/.singularity/cache. Make sure that if you build a container with sudo privilege, you will need to consider the sudo location from the cache, and not the default.

For example, running the following command with sudo privilege (considering the sudo privilege location for cache /root/.singularity/cache):

$ sudo singularity cache list
NAME                   DATE CREATED           SIZE             TYPE
ubuntu_latest.sif      2019-01-31 14:59:32    28.11 Mb         library
ubuntu_18.04.sif       2019-01-31 14:58:44    27.98 Mb         library

and then cleaning the cache without sudo privilege (singularity cache clean -a) will not work, since the default cache location is ~/.singularity/cache. In this case you would need to run the clean command with sudo privilege:

$ sudo singularity cache clean -a
NAME                     DATE CREATED           SIZE             TYPE

There 0 containers using: 0.00 kB, 0 oci blob file(s) using 0.00 kB of space.
Total space used: 0.00 kB

Listing Cache¶

For example, you can list cache image files and check which type they belong to: Library or oci.

$ singularity cache list
NAME                   DATE CREATED           SIZE             TYPE
ubuntu_latest.sif      2019-01-31 14:59:32    28.11 Mb         library
ubuntu_18.04.sif       2019-01-31 14:58:44    27.98 Mb         library
alpine_latest.sif      2019-01-31 14:58:24    2.18 Mb          library
centos_latest.sif      2019-01-31 14:59:07    72.96 Mb         library
centos_latest.sif      2019-01-31 14:59:26    73.45 Mb         oci
ubuntu_18.04.sif       2019-01-31 14:58:58    27.99 Mb         oci
ubuntu_latest.sif      2019-01-31 14:59:41    27.99 Mb         oci
alpine_latest.sif      2019-01-31 14:58:30    2.72 Mb          oci

There are 15 oci blob file(s) using 112.51 Mb of space. Use: '-T=blob' to list

You can also clean a specific cache type, choosing between: library, oci, blob (separated by commas)

# clean only library cache
$ singularity cache clean --type=library

# clean only oci cache
$ singularity cache clean --type=oci

# clean only blob cache
$ singularity cache clean --type=blob

# clean only library, and oci cache
$ singularity cache clean --type=library,oci

Note

This feature of passing additional flags with comma-separated arguments can also be used with the singularity cache clean command we will see below.

Cleaning the Cache¶

Most of the cache clean and cache list flags can be interchanged, (--name is only reserved for cache clean).

It’s worth noting that by running the following command: (with no flags)

$ singularity cache clean

By default will just clean the blob cache, but if you do:

$ singularity cache clean --all

It will clean all the cache.

Temporary Folders¶

Singularity uses a temporary directory to build the squashfs filesystem, and this temp space needs to be large enough to hold the entire resulting Singularity image. By default this happens in /tmp but the location can be configured by setting SINGULARITY_TMPDIR to the full path where you want the sandbox and squashfs temp files to be stored. Remember to use -E option to pass the value of SINGULARITY_TMPDIR to root’s environment when executing the build command with sudo.

When you run one of the action commands (i.e. run, exec, or shell) with a container from the container library or an OCI registry, Singularity builds the container in the temporary directory caches it and runs it from the cached location.

Consider the following command:

$ singularity exec docker://busybox /bin/sh

This container is first built in /tmp. Since all the oci blobs are converted into SIF format, by default a temporary runtime directory is created in:

$HOME/.singularity/cache/oci-tmp/<sha256-code>/busybox_latest.sif

In this case, the SINGULARITY_TMPDIR and SINGULARITY_CACHEDIR variables will also be respected.

Pull Folder¶

To customize your pull default location you can do so by specifying Singularity in which folder to pull the image, assuming you own a folder called mycontainers inside your $HOME folder , you would need to do something like the following:

$ singularity pull $HOME/mycontainers library://library/default/alpine

Singularity also allows you to modify the default cache location for pulling images. By default, the location of the pull folder is given by the environment variable SINGULARITY_CACHEDIR. SINGULARITY_CACHEDIR by default points to $HOME/.singularity/cache but this path can be modified. You would need to set and export the SINGULARITY_CACHEDIR environment variable before pulling the image, like so:

$ export SINGULARITY_CACHEDIR=$HOME/mycontainers
$ singularity pull library://library/default/alpine

And that will successfully pull that container image inside your new SINGULARITY_CACHEDIR location.

Encrypted Containers¶

Beginning in Singularity 3.4.0 it is possible to build and run encrypted containers. The containers are decrypted at runtime entirely in kernel space, meaning that no intermediate decrypted data is ever present on disk or in memory. See encrypted containers for more details.

Environment Variables¶

If a flag is represented by both a CLI option and an environment variable, and both are set, the CLI option will always take precedence. This is true for all environment variables except for SINGULARITY_BIND and SINGULARITY_BINDPATH which is combined with the --bind option, argument pair if both are present.
Environment variables overwrite default values in the CLI code
Any defaults in the CLI code are applied.

Defaults¶

The following variables have defaults that can be customized by you via environment variables at runtime.

Docker¶

SINGULARITY_DOCKER_LOGIN Used for the interactive login for Docker Hub.

SINGULARITY_DOCKER_USERNAME Your Docker username.

SINGULARITY_DOCKER_PASSWORD Your Docker password.

RUNSCRIPT_COMMAND Is not obtained from the environment, but is a hard coded default (“/bin/bash”). This is the fallback command used in the case that the docker image does not have a CMD or ENTRYPOINT. TAG Is the default tag, latest.

SINGULARITY_NOHTTPS This is relevant if you want to use a registry that doesn’t have https, and it speaks for itself. If you export the variable SINGULARITY_NOHTTPS you can force the software to not use https when interacting with a Docker registry. This use case is typically for use of a local registry.

Library¶

SINGULARITY_BUILDER Used to specify the remote builder service URL. The default value is our remote builder.

SINGULARITY_LIBRARY Used to specify the library to pull from. Default is set to our Cloud Library.

SINGULARITY_REMOTE Used to build an image remotely (This does not require root). The default is set to false.

Encryption¶

SINGULARITY_ENCRYPTION_PASSPHRASE Used to pass a plaintext passphrase to encrypt a container file system (with the --encrypt flag). The default is empty.

SINGULARITY_ENCRYPTION_PEM_PATH Used to specify the location of a public key to use for container encryption (with the --encrypt flag). The default is empty.