Support for Docker and OCI¶
Overview¶
Effort has been expended in developing Docker containers. Deconstructed into one or more compressed archives (typically split across multiple segments, or layers as they are known in Docker parlance) plus some metadata, images for these containers are built from specifications known as Dockerfiles. The public Docker Hub, as well as various private registries, host images for use as Docker containers. Singularity has from the outset emphasized the importance of interoperability with Docker. As a consequence, this section of the Singularity User Docs first makes its sole focus interoperabilty with Docker. In so doing, the following topics receive attention here:
Application of Singularity action commands on ephemeral containers derived from public Docker images
Converting public Docker images into Singularity’s native format for containerization, namely the Singularity Image Format (SIF)
Authenticated application of Singularity commands to containers derived from private Docker images
Authenticated application of Singularity commands to containers derived from private Docker images originating from private registries
Building SIF containers for Singularity via the command line or definition files from a variety of sources for Docker images and image archives
The second part of this section places emphasis upon Singularity’s interoperability with open standards emerging from the Open Containers Initiative (OCI). Specifically, in documenting Singularity interoperability as it relates to the OCI Image Specification, the following topics are covered:
Compliance with the OCI Image Layout Specification
OCI-compliant caching in Singularity
Acquiring OCI images and image archives via Singularity
Building SIF containers for Singularity via the command line or definition files from a variety of sources for OCI images and image archives
The section closes with a brief enumeration of emerging best practices plus consideration of troubleshooting common issues.
Running action commands on public images from Docker Hub¶
godlovedc/lolcow is a whimsical example of a publicly accessible image hosted via Docker Hub. Singularity can execute this image as follows:
$ singularity run docker://godlovedc/lolcow
INFO: Converting OCI blobs to SIF format
INFO: Starting build...
Getting image source signatures
Copying blob sha256:9fb6c798fa41e509b58bccc5c29654c3ff4648b608f5daa67c1aab6a7d02c118
45.33 MiB / 45.33 MiB [====================================================] 1s
Copying blob sha256:3b61febd4aefe982e0cb9c696d415137384d1a01052b50a85aae46439e15e49a
848 B / 848 B [============================================================] 0s
Copying blob sha256:9d99b9777eb02b8943c0e72d7a7baec5c782f8fd976825c9d3fb48b3101aacc2
621 B / 621 B [============================================================] 0s
Copying blob sha256:d010c8cf75d7eb5d2504d5ffa0d19696e8d745a457dd8d28ec6dd41d3763617e
853 B / 853 B [============================================================] 0s
Copying blob sha256:7fac07fb303e0589b9c23e6f49d5dc1ff9d6f3c8c88cabe768b430bdb47f03a9
169 B / 169 B [============================================================] 0s
Copying blob sha256:8e860504ff1ee5dc7953672d128ce1e4aa4d8e3716eb39fe710b849c64b20945
53.75 MiB / 53.75 MiB [====================================================] 2s
Copying config sha256:73d5b1025fbfa138f2cacf45bbf3f61f7de891559fa25b28ab365c7d9c3cbd82
3.33 KiB / 3.33 KiB [======================================================] 0s
Writing manifest to image destination
Storing signatures
INFO: Creating SIF file...
INFO: Build complete: /home/vagrant/.singularity/cache/oci-tmp/a692b57abc43035b197b10390ea2c12855d21649f2ea2cc28094d18b93360eeb/lolcow_latest.sif
INFO: Image cached as SIF at /home/vagrant/.singularity/cache/oci-tmp/a692b57abc43035b197b10390ea2c12855d21649f2ea2cc28094d18b93360eeb/lolcow_latest.sif
___________________________________
/ Repartee is something we think of \
| twenty-four hours too late. |
| |
\ -- Mark Twain /
-----------------------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
Here docker is prepended to ensure that the run command of Singularity is instructed to boostrap container creation based upon this Docker image, thus creating a complete URI for Singularity. Singularity subsequently downloads all the OCI blobs that comprise this image, and converts them into a single SIF file - the native format for Singularity containers. Because this image from Docker Hub is cached locally in the $HOME/.singularity/cache/oci-tmp/<org.opencontainers.image.ref.name>/lolcow_latest.sif directory, where <org.opencontainers.image.ref.name> will be replaced by the appropriate hash for the container, the image does not need to be downloaded again (from Docker Hub) the next time a Singularity run is executed. In other words, the cached copy is sensibly reused:
$ singularity run docker://godlovedc/lolcow
_________________________________________
/ Soap and education are not as sudden as \
| a massacre, but they are more deadly in |
| the long run. |
| |
\ -- Mark Twain /
-----------------------------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
Note
Image caching is documented in detail below.
Note
Use is made of the $HOME/.singularity directory by default to cache images. To cache images elsewhere, use of the environment variable SINGULARITY_CACHEDIR can be made.
As the runtime of this container is encapsulated as a single SIF file, it is possible to
cd /home/vagrant/.singularity/cache/oci-tmp/a692b57abc43035b197b10390ea2c12855d21649f2ea2cc28094d18b93360eeb/
and then execute the SIF file directly:
./lolcow_latest.sif
_______________________________________
/ The secret source of humor is not joy \
| but sorrow; there is no humor in |
| Heaven. |
| |
\ -- Mark Twain /
---------------------------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
Note
SIF files abstract Singularity containers as a single file. As with any executable, a SIF file can be executed directly.
fortune | cowsay | lolcat is executed by default when this container is run by Singularity. Singularity’s exec command allows a different command to be executed; for example:
$ singularity exec docker://godlovedc/lolcow fortune
Don't go around saying the world owes you a living. The world owes you
nothing. It was here first.
-- Mark Twain
Note
The same cached copy of the lolcow container is reused here by Singularity exec, and immediately below here by shell.
Note
Execution defaults are documented below - see Directing Execution and Container Metadata.
In addition to non-interactive execution of an image from Docker Hub, Singularity provides support for an interactive shell session:
$ singularity shell docker://godlovedc/lolcow
Singularity lolcow_latest.sif:~> cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.3 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.3 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
Singularity lolcow_latest.sif:~>
From this it is evident that use is being made of Ubuntu 16.04 within this container, whereas the shell external to the container is running a more recent release of Ubuntu (not illustrated here).
inspect reveals the metadata for a Singularity container encapsulated via SIF; Container Metadata is documented below.
Note
singularity search [search options...] <search query> does not support Docker registries like Docker Hub. Use the search box at Docker Hub to locate Docker images. Docker pull commands, e.g., docker pull godlovedc/lolcow, can be easily translated into the corresponding command for Singularity. The Docker pull command is available under “DETAILS” for a given image on Docker Hub.
Making use of public images from Docker Hub¶
Singularity can make use of public images available from the Docker Hub. By specifying the docker:// URI for an image that has already been located, Singularity can pull it - e.g.:
$ singularity pull docker://godlovedc/lolcow
WARNING: Authentication token file not found : Only pulls of public images will succeed
INFO: Starting build...
Getting image source signatures
Copying blob sha256:9fb6c798fa41e509b58bccc5c29654c3ff4648b608f5daa67c1aab6a7d02c118
45.33 MiB / 45.33 MiB [====================================================] 2s
Copying blob sha256:3b61febd4aefe982e0cb9c696d415137384d1a01052b50a85aae46439e15e49a
848 B / 848 B [============================================================] 0s
Copying blob sha256:9d99b9777eb02b8943c0e72d7a7baec5c782f8fd976825c9d3fb48b3101aacc2
621 B / 621 B [============================================================] 0s
Copying blob sha256:d010c8cf75d7eb5d2504d5ffa0d19696e8d745a457dd8d28ec6dd41d3763617e
853 B / 853 B [============================================================] 0s
Copying blob sha256:7fac07fb303e0589b9c23e6f49d5dc1ff9d6f3c8c88cabe768b430bdb47f03a9
169 B / 169 B [============================================================] 0s
Copying blob sha256:8e860504ff1ee5dc7953672d128ce1e4aa4d8e3716eb39fe710b849c64b20945
53.75 MiB / 53.75 MiB [====================================================] 3s
Copying config sha256:73d5b1025fbfa138f2cacf45bbf3f61f7de891559fa25b28ab365c7d9c3cbd82
3.33 KiB / 3.33 KiB [======================================================] 0s
Writing manifest to image destination
Storing signatures
INFO: Creating SIF file...
INFO: Build complete: lolcow_latest.sif
This pull results in a local copy of the Docker image in SIF, the Singularity Image Format:
$ file lolcow_latest.sif
lolcow_latest.sif: a /usr/bin/env run-singularity script executable (binary data)
In converting to SIF, individual layers of the Docker image have been combined into a single, native file for use by Singularity; there is no need to subsequently build the image for Singularity. For example, you can now exec, run or shell into the SIF version via Singularity, as described above.
Note
The above authentication warning originates from a check for the existence of ${HOME}/.singularity/sylabs-token. It can be ignored when making use of Docker Hub, or silenced by issuing touch ${HOME}/.singularity/sylabs-token once.
inspect reveals metadata for the container encapsulated via SIF:
$ singularity inspect lolcow_latest.sif
{
"org.label-schema.build-date": "Thursday_6_December_2018_17:29:48_UTC",
"org.label-schema.schema-version": "1.0",
"org.label-schema.usage.singularity.deffile.bootstrap": "docker",
"org.label-schema.usage.singularity.deffile.from": "godlovedc/lolcow",
"org.label-schema.usage.singularity.version": "3.0.1-40.g84083b4f"
}
Note
Container Metadata is documented below.
SIF files built from Docker images are not crytographically signed:
$ singularity verify lolcow_latest.sif
Verifying image: lolcow_latest.sif
ERROR: verification failed: error while searching for signature blocks: no signatures found for system partition
The sign command allows a cryptographic signature to be added. Refer to
Signing and Verifying Containers for details. But caution
should be exercised in signing images from Docker Hub because, unless you build
an image from scratch (OS mirrors) you are probably not really sure about the
complete contents of that image.
Note
pull is a one-time-only operation that builds a SIF file corresponding to the image retrieved from Docker Hub. Updates to the image on Docker Hub will not be reflected in the local copy.
In our example docker://godlovedc/lolcow, godlovedc specifies a Docker Hub user, whereas lolcow is the name of the repository. Adding the option to specifiy an image tag, the generic version of the URI is docker://<user>/<repo-name>[:<tag>]. Repositories on Docker Hub provides additional details.
Making use of private images from Docker Hub¶
After successful authentication, Singularity can also make use of private images available from the Docker Hub. The two means available for authentication follow here. Before describing these means, it is instructive to illustate the error generated when attempting access a private image without credentials:
$ singularity pull docker://ilumb/mylolcow
INFO: Starting build...
FATAL: Unable to pull docker://ilumb/mylolcow: conveyor failed to get: Error reading manifest latest in docker.io/ilumb/mylolcow: errors:
denied: requested access to the resource is denied
unauthorized: authentication required
In this case, the mylolcow repository of user ilumb requires authentication through specification of a valid username and password.
Authentication via Interactive Login¶
Interactive login is the first of two means provided for authentication with Docker Hub. It is enabled through use of the --docker-login option of Singularity’s pull command; for example:
$ singularity pull --docker-login docker://ilumb/mylolcow
Enter Docker Username: ilumb
Enter Docker Password:
INFO: Starting build...
Getting image source signatures
Skipping fetch of repeat blob sha256:7b8b6451c85f072fd0d7961c97be3fe6e2f772657d471254f6d52ad9f158a580
Skipping fetch of repeat blob sha256:ab4d1096d9ba178819a3f71f17add95285b393e96d08c8a6bfc3446355bcdc49
Skipping fetch of repeat blob sha256:e6797d1788acd741d33f4530106586ffee568be513d47e6e20a4c9bc3858822e
Skipping fetch of repeat blob sha256:e25c5c290bded5267364aa9f59a18dd22a8b776d7658a41ffabbf691d8104e36
Skipping fetch of repeat blob sha256:258e068bc5e36969d3ba4b47fd3ca0d392c6de465726994f7432b14b0414d23b
Copying config sha256:8a8f815257182b770d32dffff7f185013b4041d076e065893f9dd1e89ad8a671
3.12 KiB / 3.12 KiB [======================================================] 0s
Writing manifest to image destination
Storing signatures
INFO: Creating SIF file...
INFO: Build complete: mylolcow_latest.sif
After successful authentication, the private Docker image is pulled and converted to SIF as described above.
Note
For interactive sessions, --docker-login is recommended as use of plain-text passwords in your environment is avoided. Encoded authentication data is communicated with Docker Hub via secure HTTP.
Authentication via Environment Variables¶
Environment variables offer an alternative means for authentication with Docker Hub. The required exports are as follows:
export SINGULARITY_DOCKER_USERNAME=ilumb
export SINGULARITY_DOCKER_PASSWORD=<redacted>
Of course, the <redacted> plain-text password needs to be replaced by a valid one to be of practical use.
Based upon these exports, $ singularity pull docker://ilumb/mylolcow allows for the retrieval of this private image.
Note
This approach for authentication supports both interactive and non-interactive sessions. However, the requirement for a plain-text password assigned to an envrionment variable, is the security compromise for this flexibility.
Note
When specifying passwords, ‘special characters’ (e.g., $, #, .) need to be ‘escaped’ to avoid interpretation by the shell.
Making use of private images from Private Registries¶
Authentication is required to access private images that reside in Docker Hub. Of course, private images can also reside in private registries. Accounting for locations other than Docker Hub is easily achieved.
In the complete command line specification
docker://<registry>/<user>/<repo-name>[:<tag>]
registry defaults to index.docker.io. In other words,
$ singularity pull docker://godlovedc/lolcow
is functionally equivalent to
$ singularity pull docker://index.docker.io/godlovedc/lolcow
From the above example, it is evident that
$ singularity pull docker://nvcr.io/nvidia/pytorch:18.11-py3
INFO: Starting build...
Getting image source signatures
Skipping fetch of repeat blob sha256:18d680d616571900d78ee1c8fff0310f2a2afe39c6ed0ba2651ff667af406c3e
<blob fetching details deleted>
Skipping fetch of repeat blob sha256:c71aeebc266c779eb4e769c98c935356a930b16d881d7dde4db510a09cfa4222
Copying config sha256:b77551af8073c85588088ab2a39007d04bc830831ba1eef4127b2d39aaf3a6b1
21.28 KiB / 21.28 KiB [====================================================] 0s
Writing manifest to image destination
Storing signatures
INFO: Creating SIF file...
INFO: Build complete: pytorch_18.11-py3.sif
will retrieve a specific version of the PyTorch platform for Deep Learning from the NVIDIA GPU Cloud (NGC). Because NGC is a private registry, the above pull assumes authentication via environment variables when the blobs that collectively comprise the Docker image have not already been cached locally. In the NGC case, the required environment variable are set as follows:
export SINGULARITY_DOCKER_USERNAME='$oauthtoken'
export SINGULARITY_DOCKER_PASSWORD=<redacted>
Upon use, these environment-variable settings allow for authentication with NGC.
Note
$oauthtoken is to be taken literally - it is not, for example, an environment variable.
The password provided via these means is actually an API token. This token is generated via your NGC account, and is required for use of the service.
For additional details regarding authentication with NGC, and much more, please consult the NGC Getting Started documentation.
Alternatively, for purely interactive use, --docker-login is recommended:
$ singularity pull --docker-login docker://nvcr.io/nvidia/pytorch:18.11-py3
Enter Docker Username: $oauthtoken
Enter Docker Password:
INFO: Starting build...
Getting image source signatures
Skipping fetch of repeat blob sha256:18d680d616571900d78ee1c8fff0310f2a2afe39c6ed0ba2651ff667af406c3e
<blob fetching details deleted>
Skipping fetch of repeat blob sha256:c71aeebc266c779eb4e769c98c935356a930b16d881d7dde4db510a09cfa4222
Copying config sha256:b77551af8073c85588088ab2a39007d04bc830831ba1eef4127b2d39aaf3a6b1
21.28 KiB / 21.28 KiB [====================================================] 0s
Writing manifest to image destination
Storing signatures
INFO: Creating SIF file...
INFO: Build complete: pytorch_18.11-py3.sif
Authentication aside, the outcome of the pull command is the Singularity container pytorch_18.11-py3.sif - i.e., a locally stored copy, that has been coverted to SIF.
Building images for Singularity from Docker Registries¶
The build command is used to create Singularity containers. Because it is documented extensively elsewhere in this manual, only specifics relevant to Docker are provided here - namely, working with Docker Hub via the Singularity command line and through Singularity definition files.
Working from the Singularity Command Line¶
Remotely Hosted Images¶
In the simplest case, build is functionally equivalent to pull:
$ singularity build mylolcow_latest.sif docker://godlovedc/lolcow
INFO: Starting build...
Getting image source signatures
Skipping fetch of repeat blob sha256:9fb6c798fa41e509b58bccc5c29654c3ff4648b608f5daa67c1aab6a7d02c118
Skipping fetch of repeat blob sha256:3b61febd4aefe982e0cb9c696d415137384d1a01052b50a85aae46439e15e49a
Skipping fetch of repeat blob sha256:9d99b9777eb02b8943c0e72d7a7baec5c782f8fd976825c9d3fb48b3101aacc2
Skipping fetch of repeat blob sha256:d010c8cf75d7eb5d2504d5ffa0d19696e8d745a457dd8d28ec6dd41d3763617e
Skipping fetch of repeat blob sha256:7fac07fb303e0589b9c23e6f49d5dc1ff9d6f3c8c88cabe768b430bdb47f03a9
Skipping fetch of repeat blob sha256:8e860504ff1ee5dc7953672d128ce1e4aa4d8e3716eb39fe710b849c64b20945
Copying config sha256:73d5b1025fbfa138f2cacf45bbf3f61f7de891559fa25b28ab365c7d9c3cbd82
3.33 KiB / 3.33 KiB [======================================================] 0s
Writing manifest to image destination
Storing signatures
INFO: Creating SIF file...
INFO: Build complete: mylolcow_latest.sif
This build results in a local copy of the Docker image in SIF, as did pull above. Here, build has named the Singularity container mylolcow_latest.sif.
Note
docker://godlovedc/lolcow is the target provided as input for build. Armed with this target, build applies the appropriate boostrap agent to create the container - in this case, one appropriate for Docker Hub.
In addition to a read-only container image in SIF (default), build allows for the creation of a writable (ch)root directory called a sandbox for interactive development via the --sandbox option:
$ singularity build --sandbox mylolcow_latest_sandbox docker://godlovedc/lolcow
INFO: Starting build...
Getting image source signatures
Skipping fetch of repeat blob sha256:9fb6c798fa41e509b58bccc5c29654c3ff4648b608f5daa67c1aab6a7d02c118
Skipping fetch of repeat blob sha256:3b61febd4aefe982e0cb9c696d415137384d1a01052b50a85aae46439e15e49a
Skipping fetch of repeat blob sha256:9d99b9777eb02b8943c0e72d7a7baec5c782f8fd976825c9d3fb48b3101aacc2
Skipping fetch of repeat blob sha256:d010c8cf75d7eb5d2504d5ffa0d19696e8d745a457dd8d28ec6dd41d3763617e
Skipping fetch of repeat blob sha256:7fac07fb303e0589b9c23e6f49d5dc1ff9d6f3c8c88cabe768b430bdb47f03a9
Skipping fetch of repeat blob sha256:8e860504ff1ee5dc7953672d128ce1e4aa4d8e3716eb39fe710b849c64b20945
Copying config sha256:73d5b1025fbfa138f2cacf45bbf3f61f7de891559fa25b28ab365c7d9c3cbd82
3.33 KiB / 3.33 KiB [======================================================] 0s
Writing manifest to image destination
Storing signatures
INFO: Creating sandbox directory...
INFO: Build complete: mylolcow_latest_sandbox
After successful execution, the above command results in creation of the mylolcow_latest_sandbox directory with contents:
bin boot core dev environment etc home lib lib64 media mnt opt proc root run sbin singularity srv sys tmp usr var
The build command of Singularity allows (e.g., development) sandbox containers to be converted into (e.g., production) read-only SIF containers, and vice-versa. Consult the Build a container documentation for the details.
Implicit in the above command-line interactions is use of public images from Docker Hub. To make use of private images from Docker Hub, authentication is required. Available means for authentication were described above. Use of environment variables is functionally equivalent for Singularity build as it is for pull; see Authentication via Environment Variables above. For purely interactive use, authentication can be added to the build command as follows:
singularity build --docker-login mylolcow_latest_il.sif docker://ilumb/mylolcow
(Recall that docker://ilumb/mylolcow is a private image available via Docker Hub.) See Authentication via Interactive Login above regarding use of --docker-login.
Building Containers Remotely¶
By making use of the Sylabs Cloud Remote Builder, it is possible to build SIF containers remotely from images hosted at Docker Hub. The Sylabs Cloud Remote Builder is a service that can be used from the Singularity command line or via its Web interface. Here use of the Singularity CLI is emphasized.
Once you have an account for Sylabs Cloud, and have logged in to the portal, select Remote Builder. The right-hand side of this page is devoted to use of the Singularity CLI. Self-generated API tokens are used to enable authenticated access to the Remote Builder. To create a token, follow the instructions provided. Once the token has been created, store it in the file $HOME/.singularity/sylabs-token.
The above token provides authenticated use of the Sylabs Cloud Remote Builder when --remote is appended to the Singularity build command. For example, for remotely hosted images:
$ singularity build --remote lolcow_rb.sif docker://godlovedc/lolcow
searching for available build agent.........INFO: Starting build...
Getting image source signatures
Copying blob sha256:9fb6c798fa41e509b58bccc5c29654c3ff4648b608f5daa67c1aab6a7d02c118
45.33 MiB / 45.33 MiB 0s
Copying blob sha256:3b61febd4aefe982e0cb9c696d415137384d1a01052b50a85aae46439e15e49a
848 B / 848 B 0s
Copying blob sha256:9d99b9777eb02b8943c0e72d7a7baec5c782f8fd976825c9d3fb48b3101aacc2
621 B / 621 B 0s
Copying blob sha256:d010c8cf75d7eb5d2504d5ffa0d19696e8d745a457dd8d28ec6dd41d3763617e
853 B / 853 B 0s
Copying blob sha256:7fac07fb303e0589b9c23e6f49d5dc1ff9d6f3c8c88cabe768b430bdb47f03a9
169 B / 169 B 0s
Copying blob sha256:8e860504ff1ee5dc7953672d128ce1e4aa4d8e3716eb39fe710b849c64b20945
53.75 MiB / 53.75 MiB 0s
Copying config sha256:73d5b1025fbfa138f2cacf45bbf3f61f7de891559fa25b28ab365c7d9c3cbd82
3.33 KiB / 3.33 KiB 0s
Writing manifest to image destination
Storing signatures
INFO: Creating SIF file...
INFO: Build complete: /tmp/image-341891107
INFO: Now uploading /tmp/image-341891107 to the library
87.94 MiB / 87.94 MiB 100.00% 38.96 MiB/s 2s
INFO: Setting tag latest
87.94 MiB / 87.94 MiB [===============================================================================] 100.00% 17.23 MiB/s 5s
Note
Elevated privileges (e.g., via sudo) are not required when use is made of the Sylabs Cloud Remote Builder.
During the build process, progress can be monitored in the Sylabs Cloud portal on the Remote Builder page - as illustrated upon completion by the screenshot below. Once complete, this results in a local copy of the SIF file lolcow_rb.sif. From the Sylabs Cloud Singularity Library it is evident that the ‘original’ SIF file remains available via this portal.
Locally Available Images: Cached by Docker¶
Singularity containers can be built at the command line from images cached locally by Docker. Suppose, for example:
$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
godlovedc/lolcow latest 577c1fe8e6d8 16 months ago 241MB
This indicates that godlovedc/lolcow:latest has been cached locally by Docker. Then
$ sudo singularity build lolcow_from_docker_cache.sif docker-daemon://godlovedc/lolcow:latest
INFO: Starting build...
Getting image source signatures
Copying blob sha256:a2022691bf950a72f9d2d84d557183cb9eee07c065a76485f1695784855c5193
119.83 MiB / 119.83 MiB [==================================================] 6s
Copying blob sha256:ae620432889d2553535199dbdd8ba5a264ce85fcdcd5a430974d81fc27c02b45
15.50 KiB / 15.50 KiB [====================================================] 0s
Copying blob sha256:c561538251751e3685c7c6e7479d488745455ad7f84e842019dcb452c7b6fecc
14.50 KiB / 14.50 KiB [====================================================] 0s
Copying blob sha256:f96e6b25195f1b36ad02598b5d4381e41997c93ce6170cab1b81d9c68c514db0
5.50 KiB / 5.50 KiB [======================================================] 0s
Copying blob sha256:7f7a065d245a6501a782bf674f4d7e9d0a62fa6bd212edbf1f17bad0d5cd0bfc
3.00 KiB / 3.00 KiB [======================================================] 0s
Copying blob sha256:70ca7d49f8e9c44705431e3dade0636a2156300ae646ff4f09c904c138728839
116.56 MiB / 116.56 MiB [==================================================] 6s
Copying config sha256:73d5b1025fbfa138f2cacf45bbf3f61f7de891559fa25b28ab365c7d9c3cbd82
3.33 KiB / 3.33 KiB [======================================================] 0s
Writing manifest to image destination
Storing signatures
INFO: Creating SIF file...
INFO: Build complete: lolcow_from_docker_cache.sif
results in lolcow_from_docker_cache.sif for native use by Singularity. There are two important differences in syntax evident in the above build command:
The
dockerpart of the URI has been appended bydaemon. This ensures Singularity seek an image locally cached by Docker to boostrap the conversion process to SIF, as opposed to attempting to retrieve an image remotely hosted via Docker Hub.
sudois prepended to thebuildcommand for Singularity; this is required as the Docker daemon executes asroot. However, if the user issuing thebuildcommand is a member of thedockerLinux group, thensudoneed not be prepended.
Note
The image tag, in this case latest, is required when bootstrapping creation of a container for Singularity from an image locally cached by Docker.
Note
The Sylabs Cloud Remote Builder does not interoperate with local Docker daemons; therefore, images cached locally by Docker, cannot be used to bootstrap creation of SIF files via the Remote Builder service. Of course, a SIF file could be created locally as detailed above. Then, in a separate, manual step, pushed to the Sylabs Cloud Singularity Library.
Locally Available Images: Stored Archives¶
Singularity containers can also be built at the command line from Docker images stored locally as tar files.
The lolcow.tar file employed below in this example can be produced by making use of an environment in which Docker is available as follows:
Obtain a local copy of the image from Docker Hub via
sudo docker pull godlovedc/lolcow. Issuing the following command confirms that a copy of the desired image is available locally:$ sudo docker images REPOSITORY TAG IMAGE ID CREATED SIZE godlovedc/lolcow latest 577c1fe8e6d8 17 months ago 241MB
Noting that the image identifier above is
577c1fe8e6d8, the required archive can be created bysudo docker save 577c1fe8e6d8 -o lolcow.tar.
Thus lolcow.tar is a locally stored archive in the current working directory with contents:
$ sudo tar tvf lolcow.tar
drwxr-xr-x 0/0 0 2017-09-21 19:37 02aefa059d08482d344293d0ad27182a0a9d330ebc73abd92a1f9744844f91e9/
-rw-r--r-- 0/0 3 2017-09-21 19:37 02aefa059d08482d344293d0ad27182a0a9d330ebc73abd92a1f9744844f91e9/VERSION
-rw-r--r-- 0/0 1417 2017-09-21 19:37 02aefa059d08482d344293d0ad27182a0a9d330ebc73abd92a1f9744844f91e9/json
-rw-r--r-- 0/0 122219008 2017-09-21 19:37 02aefa059d08482d344293d0ad27182a0a9d330ebc73abd92a1f9744844f91e9/layer.tar
drwxr-xr-x 0/0 0 2017-09-21 19:37 3762e087ebbb895fd9c38981c1f7bfc76c9879fd3fdadef64df49e92721bb527/
-rw-r--r-- 0/0 3 2017-09-21 19:37 3762e087ebbb895fd9c38981c1f7bfc76c9879fd3fdadef64df49e92721bb527/VERSION
-rw-r--r-- 0/0 482 2017-09-21 19:37 3762e087ebbb895fd9c38981c1f7bfc76c9879fd3fdadef64df49e92721bb527/json
-rw-r--r-- 0/0 14848 2017-09-21 19:37 3762e087ebbb895fd9c38981c1f7bfc76c9879fd3fdadef64df49e92721bb527/layer.tar
-rw-r--r-- 0/0 4432 2017-09-21 19:37 577c1fe8e6d84360932b51767b65567550141af0801ff6d24ad10963e40472c5.json
drwxr-xr-x 0/0 0 2017-09-21 19:37 5bad884501c0e760bc0c9ca3ae3dca3f12c4abeb7d18194c364fec522b91b4f9/
-rw-r--r-- 0/0 3 2017-09-21 19:37 5bad884501c0e760bc0c9ca3ae3dca3f12c4abeb7d18194c364fec522b91b4f9/VERSION
-rw-r--r-- 0/0 482 2017-09-21 19:37 5bad884501c0e760bc0c9ca3ae3dca3f12c4abeb7d18194c364fec522b91b4f9/json
-rw-r--r-- 0/0 3072 2017-09-21 19:37 5bad884501c0e760bc0c9ca3ae3dca3f12c4abeb7d18194c364fec522b91b4f9/layer.tar
drwxr-xr-x 0/0 0 2017-09-21 19:37 81ce2fd011bc8241ae72eaee9146116b7c289e941467ff276397720171e6c576/
-rw-r--r-- 0/0 3 2017-09-21 19:37 81ce2fd011bc8241ae72eaee9146116b7c289e941467ff276397720171e6c576/VERSION
-rw-r--r-- 0/0 406 2017-09-21 19:37 81ce2fd011bc8241ae72eaee9146116b7c289e941467ff276397720171e6c576/json
-rw-r--r-- 0/0 125649920 2017-09-21 19:37 81ce2fd011bc8241ae72eaee9146116b7c289e941467ff276397720171e6c576/layer.tar
drwxr-xr-x 0/0 0 2017-09-21 19:37 a10239905b060fd8b17ab31f37957bd126774f52f5280767d3b2639692913499/
-rw-r--r-- 0/0 3 2017-09-21 19:37 a10239905b060fd8b17ab31f37957bd126774f52f5280767d3b2639692913499/VERSION
-rw-r--r-- 0/0 482 2017-09-21 19:37 a10239905b060fd8b17ab31f37957bd126774f52f5280767d3b2639692913499/json
-rw-r--r-- 0/0 15872 2017-09-21 19:37 a10239905b060fd8b17ab31f37957bd126774f52f5280767d3b2639692913499/layer.tar
drwxr-xr-x 0/0 0 2017-09-21 19:37 ab6e1ca3392b2f4dbb60157cf99434b6975f37a767f530e293704a7348407634/
-rw-r--r-- 0/0 3 2017-09-21 19:37 ab6e1ca3392b2f4dbb60157cf99434b6975f37a767f530e293704a7348407634/VERSION
-rw-r--r-- 0/0 482 2017-09-21 19:37 ab6e1ca3392b2f4dbb60157cf99434b6975f37a767f530e293704a7348407634/json
-rw-r--r-- 0/0 5632 2017-09-21 19:37 ab6e1ca3392b2f4dbb60157cf99434b6975f37a767f530e293704a7348407634/layer.tar
-rw-r--r-- 0/0 574 1970-01-01 01:00 manifest.json
In other words, it is evident that this ‘tarball’ is a Docker-format image comprised of multiple layers along with metadata in a JSON manifest.
Through use of the docker-archive bootstrap agent, a SIF file (lolcow_tar.sif) for use by Singularity can be created via the following build command:
$ singularity build lolcow_tar.sif docker-archive://lolcow.tar
INFO: Starting build...
Getting image source signatures
Copying blob sha256:a2022691bf950a72f9d2d84d557183cb9eee07c065a76485f1695784855c5193
119.83 MiB / 119.83 MiB [==================================================] 6s
Copying blob sha256:ae620432889d2553535199dbdd8ba5a264ce85fcdcd5a430974d81fc27c02b45
15.50 KiB / 15.50 KiB [====================================================] 0s
Copying blob sha256:c561538251751e3685c7c6e7479d488745455ad7f84e842019dcb452c7b6fecc
14.50 KiB / 14.50 KiB [====================================================] 0s
Copying blob sha256:f96e6b25195f1b36ad02598b5d4381e41997c93ce6170cab1b81d9c68c514db0
5.50 KiB / 5.50 KiB [======================================================] 0s
Copying blob sha256:7f7a065d245a6501a782bf674f4d7e9d0a62fa6bd212edbf1f17bad0d5cd0bfc
3.00 KiB / 3.00 KiB [======================================================] 0s
Copying blob sha256:70ca7d49f8e9c44705431e3dade0636a2156300ae646ff4f09c904c138728839
116.56 MiB / 116.56 MiB [==================================================] 6s
Copying config sha256:73d5b1025fbfa138f2cacf45bbf3f61f7de891559fa25b28ab365c7d9c3cbd82
3.33 KiB / 3.33 KiB [======================================================] 0s
Writing manifest to image destination
Storing signatures
INFO: Creating SIF file...
INFO: Build complete: lolcow_tar.sif
There are two important differences in syntax evident in the above build command:
The
dockerpart of the URI has been appended byarchive. This ensures Singularity seek a Docker-format image archive stored locally aslolcow.tarto boostrap the conversion process to SIF, as opposed to attempting to retrieve an image remotely hosted via Docker Hub.
sudois not prepended to thebuildcommand for Singularity. This is not required if the executing user has the appropriate access privileges to the stored file.
Note
The docker-archive bootstrap agent handles archives (.tar files) as well as compressed archives (.tar.gz) when containers are built for Singularity via its build command.
Note
The Sylabs Cloud Remote Builder does not interoperate with locally stored Docker-format images; therefore, images cached locally by Docker, cannot be used to bootstrap creation of SIF files via the Remote Builder service. Of course, a SIF file could be created locally as detailed above. Then, in a separate, manual step, pushed to the Sylabs Cloud Singularity Library.
Pushing Locally Available Images to a Library¶
The outcome of bootstrapping from an image cached locally by Docker, or one stored locally as an archive, is of course a locally stored SIF file. As noted above, this is the only option available, as the Sylabs Cloud Remote Builder does not interoperate with the Docker daemon or locally stored archives in the Docker image format. Once produced, however, it may be desirable to make the resulting SIF file available through the Sylabs Cloud Singularity Library; therefore, the procedure to push a locally available SIF file to the Library is detailed here.
From the Sylabs Cloud Singularity Library, select Create a new Project. In this first of two steps, the publicly accessible project is created as illustrated below:
Because an access token for the cloud service already exists, attention can be focused on the push command prototyped towards the bottom of the following screenshot:
In fact, by simply replacing image.sif with lolcow_tar.sif, the following upload is executed:
$ singularity push lolcow_tar.sif library://ilumb/default/lolcow_tar
INFO: Now uploading lolcow_tar.sif to the library
87.94 MiB / 87.94 MiB [=============================================================================] 100.00% 1.25 MiB/s 1m10s
INFO: Setting tag latest
Finally, from the perspective of the Library, the hosted version of the SIF file appears as illustrated below. Directions on how to pull this file are included from the portal.
Note
The hosted version of the SIF file in the Sylabs Cloud Singularity Library is maintainable. In other words, if the image is updated locally, the update can be pushed to the Library and tagged appropriately.
Working with Definition Files¶
Mandatory Header Keywords: Remotely Boostrapped¶
Akin to a set of blueprints that explain how to build a custom container, Singularity definition files (or “def files”) are considered in detail elsewhere in this manual. Therefore, only def file nuances specific to interoperability with Docker receive consideration here.
Singularity definition files are comprised of two parts - a header plus sections.
When working with repositories such as Docker Hub, Bootstrap and From are mandatory keywords within the header; for example, if the file lolcow.def has contents
Bootstrap: docker
From: godlovedc/lolcow
then
sudo singularity build lolcow.sif lolcow.def
creates a Singularity container in SIF by bootstrapping from the public godlovedc/lolcow image from Docker Hub.
In the above definition file, docker is one of numerous, possible bootstrap agents; this, and other bootstrap agents receive attention in the appendix.
Through the means for authentication described above, definition files permit use of private images hosted via Docker Hub. For example, if the file mylolcow.def has contents
Bootstrap: docker
From: ilumb/mylolcow
then
sudo singularity build --docker-login mylolcow.sif mylolcow.def
creates a Singularity container in SIF by bootstrapping from the private ilumb/mylolcow image from Docker Hub after successful interactive authentication.
Alternatively, if environment variables have been set as above, then
$ sudo -E singularity build mylolcow.sif mylolcow.def
enables authenticated use of the private image.
Note
The -E option is required to preserve the user’s existing environment variables upon sudo invocation - a priviledge escalation required to create Singularity containers via the build command.
Remotely Bootstrapped and Built Containers¶
Consider again the definition file used the outset of the section above:
Bootstrap: docker
From: godlovedc/lolcow
With two small adjustments to the Singularity build command, the Sylabs Cloud Remote Builder can be utilized:
$ singularity build --remote lolcow_rb_def.sif lolcow.def
searching for available build agent......INFO: Starting build...
Getting image source signatures
Copying blob sha256:9fb6c798fa41e509b58bccc5c29654c3ff4648b608f5daa67c1aab6a7d02c118
45.33 MiB / 45.33 MiB 0s
Copying blob sha256:3b61febd4aefe982e0cb9c696d415137384d1a01052b50a85aae46439e15e49a
848 B / 848 B 0s
Copying blob sha256:9d99b9777eb02b8943c0e72d7a7baec5c782f8fd976825c9d3fb48b3101aacc2
621 B / 621 B 0s
Copying blob sha256:d010c8cf75d7eb5d2504d5ffa0d19696e8d745a457dd8d28ec6dd41d3763617e
853 B / 853 B 0s
Copying blob sha256:7fac07fb303e0589b9c23e6f49d5dc1ff9d6f3c8c88cabe768b430bdb47f03a9
169 B / 169 B 0s
Copying blob sha256:8e860504ff1ee5dc7953672d128ce1e4aa4d8e3716eb39fe710b849c64b20945
53.75 MiB / 53.75 MiB 0s
Copying config sha256:73d5b1025fbfa138f2cacf45bbf3f61f7de891559fa25b28ab365c7d9c3cbd82
3.33 KiB / 3.33 KiB 0s
Writing manifest to image destination
Storing signatures
INFO: Creating SIF file...
INFO: Build complete: /tmp/image-994007654
INFO: Now uploading /tmp/image-994007654 to the library
87.94 MiB / 87.94 MiB 100.00% 41.76 MiB/s 2s
INFO: Setting tag latest
87.94 MiB / 87.94 MiB [===============================================================================] 100.00% 19.08 MiB/s 4s
In the above, --remote has been added as the build option that causes use of the Remote Builder service. A much more subtle change, however, is the absence of sudo ahead of singularity build. Though subtle here, this absence is notable, as users can build containers via the Remote Builder with escalated privileges; in other words, steps in container creation that require root access are enabled via the Remote Builder even for (DevOps) users without admninistrative privileges locally.
In addition to the command-line support described above, the Sylabs Cloud Remote Builder also allows definition files to be copied and pasted into its Graphical User Interface (GUI). After pasting a definition file, and having that file validated by the service, the build-centric part of the GUI appears as illustrated below. By clicking on the Build button, creation of the container is initiated.
Once the build process has been completed, the corresponding SIF file can be retrieved from the service - as shown below. A log file for the build process is provided by the GUI, and made available for download as a text file (not shown here).
A copy of the SIF file created by the service remains in the Sylabs Cloud Singularity Library as illustrated below.
Note
The Sylabs Cloud is currently available as an Alpha Preview. In addition to the Singularity Library and Remote Builder, a Keystore service is also available. All three services make use of a freemium pricing model in supporting Singularity Community Edition. In contrast, all three services are included in SingularityPRO - an enterprise grade subscription for Singularity that is offered for a fee from Sylabs. For addtional details regarding the different offerings available for Singularity, please consult the Sylabs website.
Mandatory Header Keywords: Locally Boostrapped¶
When docker-daemon is the bootstrap agent in a Singularity definition file, SIF containers can be created from images cached locally by Docker. Suppose the definition file lolcow-d.def has contents:
Bootstrap: docker-daemon
From: godlovedc/lolcow:latest
Note
Again, the image tag latest is required when bootstrapping creation of a container for Singularity from an image locally cached by Docker.
Then,
$ sudo singularity build lolcow_from_docker_cache.sif lolcow-d.def
Build target already exists. Do you want to overwrite? [N/y] y
INFO: Starting build...
Getting image source signatures
Copying blob sha256:a2022691bf950a72f9d2d84d557183cb9eee07c065a76485f1695784855c5193
119.83 MiB / 119.83 MiB [==================================================] 6s
Copying blob sha256:ae620432889d2553535199dbdd8ba5a264ce85fcdcd5a430974d81fc27c02b45
15.50 KiB / 15.50 KiB [====================================================] 0s
Copying blob sha256:c561538251751e3685c7c6e7479d488745455ad7f84e842019dcb452c7b6fecc
14.50 KiB / 14.50 KiB [====================================================] 0s
Copying blob sha256:f96e6b25195f1b36ad02598b5d4381e41997c93ce6170cab1b81d9c68c514db0
5.50 KiB / 5.50 KiB [======================================================] 0s
Copying blob sha256:7f7a065d245a6501a782bf674f4d7e9d0a62fa6bd212edbf1f17bad0d5cd0bfc
3.00 KiB / 3.00 KiB [======================================================] 0s
Copying blob sha256:70ca7d49f8e9c44705431e3dade0636a2156300ae646ff4f09c904c138728839
116.56 MiB / 116.56 MiB [==================================================] 6s
Copying config sha256:73d5b1025fbfa138f2cacf45bbf3f61f7de891559fa25b28ab365c7d9c3cbd82
3.33 KiB / 3.33 KiB [======================================================] 0s
Writing manifest to image destination
Storing signatures
INFO: Creating SIF file...
INFO: Build complete: lolcow_from_docker_cache.sif
In other words, this is the definition-file counterpart to the command-line invocation provided above.
Note
The sudo requirement in the above build request originates from Singularity; it is the standard requirement when use is made of definition files. In other words, membership of the issuing user in the docker Linux group is of no consequence in this context.
Alternatively when docker-archive is the bootstrap agent in a Singularity definition file, SIF containers can be created from images stored locally by Docker. Suppose the definition file lolcow-da.def has contents:
Bootstrap: docker-archive
From: lolcow.tar
Then,
$ sudo singularity build lolcow_tar_def.sif lolcow-da.def
INFO: Starting build...
Getting image source signatures
Copying blob sha256:a2022691bf950a72f9d2d84d557183cb9eee07c065a76485f1695784855c5193
119.83 MiB / 119.83 MiB [==================================================] 6s
Copying blob sha256:ae620432889d2553535199dbdd8ba5a264ce85fcdcd5a430974d81fc27c02b45
15.50 KiB / 15.50 KiB [====================================================] 0s
Copying blob sha256:c561538251751e3685c7c6e7479d488745455ad7f84e842019dcb452c7b6fecc
14.50 KiB / 14.50 KiB [====================================================] 0s
Copying blob sha256:f96e6b25195f1b36ad02598b5d4381e41997c93ce6170cab1b81d9c68c514db0
5.50 KiB / 5.50 KiB [======================================================] 0s
Copying blob sha256:7f7a065d245a6501a782bf674f4d7e9d0a62fa6bd212edbf1f17bad0d5cd0bfc
3.00 KiB / 3.00 KiB [======================================================] 0s
Copying blob sha256:70ca7d49f8e9c44705431e3dade0636a2156300ae646ff4f09c904c138728839
116.56 MiB / 116.56 MiB [==================================================] 6s
Copying config sha256:73d5b1025fbfa138f2cacf45bbf3f61f7de891559fa25b28ab365c7d9c3cbd82
3.33 KiB / 3.33 KiB [======================================================] 0s
Writing manifest to image destination
Storing signatures
INFO: Creating SIF file...
INFO: Build complete: lolcow_tar_def.sif
through build results in the SIF file lolcow_tar_def.sif. In other words, this is the definition-file counterpart to the command-line invocation provided above .
Optional Header Keywords¶
In the two-previous examples, the From keyword specifies both the user and repo-name in making use of Docker Hub. Optional use of Namespace permits the more-granular split across two keywords:
Bootstrap: docker
Namespace: godlovedc
From: lolcow
Note
In their documentation, “Docker ID namespace” and user are employed as synonyms in the text and examples, respectively.
Note
The default value for the optional keyword Namespace is library.
Private Images and Registries¶
Thus far, use of Docker Hub has been assumed. To make use of a different repository of Docker images the optional Registry keyword can be added to the Singularity definition file. For example, to make use of a Docker image from the NVIDIA GPU Cloud (NGC) corresponding definition file is:
Bootstrap: docker
From: nvidia/pytorch:18.11-py3
Registry: nvcr.io
This def file ngc_pytorch.def can be passed as a specification to build as follows:
$ sudo singularity build --docker-login mypytorch.sif ngc_pytorch.def
Enter Docker Username: $oauthtoken
Enter Docker Password: <obscured>
INFO: Starting build...
Getting image source signatures
Copying blob sha256:18d680d616571900d78ee1c8fff0310f2a2afe39c6ed0ba2651ff667af406c3e
41.34 MiB / 41.34 MiB [====================================================] 2s
<blob copying details deleted>
Copying config sha256:b77551af8073c85588088ab2a39007d04bc830831ba1eef4127b2d39aaf3a6b1
21.28 KiB / 21.28 KiB [====================================================] 0s
Writing manifest to image destination
Storing signatures
INFO: Creating SIF file...
INFO: Build complete: mypytorch.sif
After successful authentication via interactive use of the --docker-login option, output as the SIF container mypytorch.sif is (ultimately) produced. As above, use of environment variables is another option available for authenticating private Docker type repositories such as NGC; once set, the build command is as above save for the absence of the --docker-login option.
Directing Execution¶
The Dockerfile corresponding to godlovedc/lolcow (and available here) is as follows:
FROM ubuntu:16.04
RUN apt-get update && apt-get install -y fortune cowsay lolcat
ENV PATH /usr/games:${PATH}
ENV LC_ALL=C
ENTRYPOINT fortune | cowsay | lolcat
The execution-specific part of this Dockerfile is the ENTRYPOINT - “… an optional definition for the first part of the command to be run …” according to the available documentation. After conversion to SIF, execution of fortune | cowsay | lolcat within the container produces the output:
$ ./mylolcow.sif
______________________________________
/ Q: How did you get into artificial \
| intelligence? A: Seemed logical -- I |
\ didn't have any real intelligence. /
--------------------------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
In addition, CMD allows an arbitrary string to be appended to the ENTRYPOINT. Thus, multiple commands or flags can be passed together through combined use.
Suppose now that a Singularity %runscript section is added to the definition file as follows:
Bootstrap: docker
Namespace: godlovedc
From: lolcow
%runscript
fortune
After conversion to SIF via the Singularity build command, exection of the resulting container produces the output:
$ ./lolcow.sif
This was the most unkindest cut of all.
-- William Shakespeare, "Julius Caesar"
In other words, introduction of a %runscript section into the Singularity definition file causes the ENTRYPOINT of the Dockerfile to be bypassed. The presence of the %runscript section would also bypass a CMD entry in the Dockerfile.
To preserve use of ENTRYPOINT and/or CMD as defined in the Dockerfile, the %runscript section must be absent from the Singularity definition. In this case, and to favor execution of CMD over ENTRYPOINT, a non-empty assignment of the optional IncludeCmd should be included in the header section of the Singularity definition file as follows:
Bootstrap: docker
Namespace: godlovedc
From: lolcow
IncludeCmd: yes
Note
Because only a non-empty IncludeCmd is required, either yes (as above) or no results in execution of CMD over ENTRYPOINT.
To summarize execution precedence:
If present, the
%runscriptsection of the Singularity definition file is executedIf
IncludeCmdis a non-empty keyword entry in the header of the Singularity definition file, thenCMDfrom theDockerfileis executedIf present in the
Dockerfile,ENTRYPOINTappended byCMD(if present) are executed in sequenceExecution of the
bashshell is defaulted to
Container Metadata¶
Singularity’s inspect command displays container metadata - data about data that is encapsulated within a SIF file. Default output (assumed via the --labels option) from the command was illustrated above. inspect, however, provides a number of options that are detailed elsewhere; in the remainder of this section, Docker-specific use to establish execution precedence is emphasized.
As stated above (i.e., the first case of execution precedence), the very existence of a %runscript section in a Singularity definition file takes precedence over commands that might exist in the Dockerfile.
When the %runscript section is removed from the Singularity definition file, the result is (once again):
$ singularity inspect --deffile lolcow.sif
from: lolcow
bootstrap: docker
namespace: godlovedc
The runscript ‘inherited’ from the Dockerfile is:
$ singularity inspect --runscript lolcow.sif
#!/bin/sh
OCI_ENTRYPOINT='"/bin/sh" "-c" "fortune | cowsay | lolcat"'
OCI_CMD=''
# ENTRYPOINT only - run entrypoint plus args
if [ -z "$OCI_CMD" ] && [ -n "$OCI_ENTRYPOINT" ]; then
SINGULARITY_OCI_RUN="${OCI_ENTRYPOINT} $@"
fi
# CMD only - run CMD or override with args
if [ -n "$OCI_CMD" ] && [ -z "$OCI_ENTRYPOINT" ]; then
if [ $# -gt 0 ]; then
SINGULARITY_OCI_RUN="$@"
else
SINGULARITY_OCI_RUN="${OCI_CMD}"
fi
fi
# ENTRYPOINT and CMD - run ENTRYPOINT with CMD as default args
# override with user provided args
if [ $# -gt 0 ]; then
SINGULARITY_OCI_RUN="${OCI_ENTRYPOINT} $@"
else
SINGULARITY_OCI_RUN="${OCI_ENTRYPOINT} ${OCI_CMD}"
fi
eval ${SINGULARITY_OCI_RUN}
From this Bourne shell script, it is evident that only an ENTRYPOINT is detailed in the Dockerfile; thus the ENTRYPOINT only - run entrypoint plus args conditional block is executed. In this case then, the third case of execution precedence has been illustrated.
The above Bourne shell script also illustrates how the following scenarios will be handled:
A
CMDonly entry in theDockerfileBoth
ENTRYPOINTandCMDentries in theDockerfile
From this level of detail, use of ENTRYPOINT and/or CMD in a Dockerfile has been made explicit. These remain examples within the third case of execution precedence.
OCI Image Support¶
Overview¶
OCI is an acronym for the Open Containers Initiative - an independent organization whose mandate is to develop open standards relating to containerization. To date, standardization efforts have focused on container formats and runtimes; it is the former that is emphasized here. Stated simply, an OCI blob is content that can be addressed; in other words, each layer of a Docker image is rendered as an OCI blob as illustrated in the (revisited) pull example below.
Note
To facilitate interoperation with Docker Hub, the Singularity core makes use of the containers/image library - “… a set of Go libraries aimed at working in various way[s] with containers’ images and container image registries.”
Image Pulls Revisited¶
After describing various action commands that could be applied to images hosted remotely via Docker Hub, the notion of having a local copy in Singularity’s native format for containerization (SIF) was introduced:
$ singularity pull docker://godlovedc/lolcow
INFO: Starting build...
Getting image source signatures
Copying blob sha256:9fb6c798fa41e509b58bccc5c29654c3ff4648b608f5daa67c1aab6a7d02c118
45.33 MiB / 45.33 MiB [====================================================] 1s
Copying blob sha256:3b61febd4aefe982e0cb9c696d415137384d1a01052b50a85aae46439e15e49a
848 B / 848 B [============================================================] 0s
Copying blob sha256:9d99b9777eb02b8943c0e72d7a7baec5c782f8fd976825c9d3fb48b3101aacc2
621 B / 621 B [============================================================] 0s
Copying blob sha256:d010c8cf75d7eb5d2504d5ffa0d19696e8d745a457dd8d28ec6dd41d3763617e
853 B / 853 B [============================================================] 0s
Copying blob sha256:7fac07fb303e0589b9c23e6f49d5dc1ff9d6f3c8c88cabe768b430bdb47f03a9
169 B / 169 B [============================================================] 0s
Copying blob sha256:8e860504ff1ee5dc7953672d128ce1e4aa4d8e3716eb39fe710b849c64b20945
53.75 MiB / 53.75 MiB [====================================================] 2s
Copying config sha256:73d5b1025fbfa138f2cacf45bbf3f61f7de891559fa25b28ab365c7d9c3cbd82
3.33 KiB / 3.33 KiB [======================================================] 0s
Writing manifest to image destination
Storing signatures
INFO: Creating SIF file...
INFO: Build complete: lolcow_latest.sif
Thus use of Singularity’s pull command results in the local file copy in SIF, namely lolcow_latest.sif. Layers of the image from Docker Hub are copied locally as OCI blobs.
Image Caching in Singularity¶
If the same pull command is issued a second time, the output is different:
$ singularity pull docker://godlovedc/lolcow
INFO: Starting build...
Getting image source signatures
Skipping fetch of repeat blob sha256:9fb6c798fa41e509b58bccc5c29654c3ff4648b608f5daa67c1aab6a7d02c118
Skipping fetch of repeat blob sha256:3b61febd4aefe982e0cb9c696d415137384d1a01052b50a85aae46439e15e49a
Skipping fetch of repeat blob sha256:9d99b9777eb02b8943c0e72d7a7baec5c782f8fd976825c9d3fb48b3101aacc2
Skipping fetch of repeat blob sha256:d010c8cf75d7eb5d2504d5ffa0d19696e8d745a457dd8d28ec6dd41d3763617e
Skipping fetch of repeat blob sha256:7fac07fb303e0589b9c23e6f49d5dc1ff9d6f3c8c88cabe768b430bdb47f03a9
Skipping fetch of repeat blob sha256:8e860504ff1ee5dc7953672d128ce1e4aa4d8e3716eb39fe710b849c64b20945
Copying config sha256:73d5b1025fbfa138f2cacf45bbf3f61f7de891559fa25b28ab365c7d9c3cbd82
3.33 KiB / 3.33 KiB [======================================================] 0s
Writing manifest to image destination
Storing signatures
INFO: Creating SIF file...
INFO: Build complete: lolcow_latest.sif
As the copy operation has clearly been skipped, it is evident that a copy of all OCI blobs must be cached locally. Indeed, Singularity has made an entry in its local cache as follows:
$ tree .singularity/
.singularity/
└── cache
└── oci
├── blobs
│ └── sha256
│ ├── 3b61febd4aefe982e0cb9c696d415137384d1a01052b50a85aae46439e15e49a
│ ├── 73d5b1025fbfa138f2cacf45bbf3f61f7de891559fa25b28ab365c7d9c3cbd82
│ ├── 7fac07fb303e0589b9c23e6f49d5dc1ff9d6f3c8c88cabe768b430bdb47f03a9
│ ├── 8e860504ff1ee5dc7953672d128ce1e4aa4d8e3716eb39fe710b849c64b20945
│ ├── 9d99b9777eb02b8943c0e72d7a7baec5c782f8fd976825c9d3fb48b3101aacc2
│ ├── 9fb6c798fa41e509b58bccc5c29654c3ff4648b608f5daa67c1aab6a7d02c118
│ ├── d010c8cf75d7eb5d2504d5ffa0d19696e8d745a457dd8d28ec6dd41d3763617e
│ └── f2a852991b0a36a9f3d6b2a33b98a461e9ede8393482f0deb5287afcbae2ce10
├── index.json
└── oci-layout
4 directories, 10 files
Compliance with the OCI Image Layout Specification¶
From the perspective of the directory $HOME/.singularity/cache/oci, this cache implementation in Singularity complies with the OCI Image Layout Specification:
blobsdirectory - contains content addressable data, that is otherwise considered opaque
oci-layoutfile - a mandatory JSON object file containing both mandatory and optional content
index.jsonfile - a mandatory JSON object file containing an index of the images
Because one or more images is ‘bundled’ here, the directory $HOME/.singularity/cache/oci is referred to as the $OCI_BUNDLE_DIR.
For additional details regarding this specification, consult the OCI Image Format Specification.
OCI Compliance and the Singularity Cache¶
As required by the layout specification, OCI blobs are uniquely named by their contents:
$ shasum -a 256 ./blobs/sha256/9fb6c798fa41e509b58bccc5c29654c3ff4648b608f5daa67c1aab6a7d02c118
9fb6c798fa41e509b58bccc5c29654c3ff4648b608f5daa67c1aab6a7d02c118 ./blobs/sha256/9fb6c798fa41e509b58bccc5c29654c3ff4648b608f5daa67c1aab6a7d02c118
They are also otherwise opaque:
$ file ./blobs/sha256/9fb6c798fa41e509b58bccc5c29654c3ff4648b608f5daa67c1aab6a7d02c118 ./blobs/sha256/9fb6c798fa41e509b58bccc5c29654c3ff4648b608f5daa67c1aab6a7d02c118: gzip compressed data
The content of the oci-layout file in this example is:
$ cat oci-layout | jq
{
"imageLayoutVersion": "1.0.0"
}
This is as required for compliance with the layout standard.
Note
In rendering the above JSON object files, use has been made of jq - the command-line JSON processor.
The index of images in this case is:
$ cat index.json | jq
{
"schemaVersion": 2,
"manifests": [
{
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"digest": "sha256:f2a852991b0a36a9f3d6b2a33b98a461e9ede8393482f0deb5287afcbae2ce10",
"size": 1125,
"annotations": {
"org.opencontainers.image.ref.name": "a692b57abc43035b197b10390ea2c12855d21649f2ea2cc28094d18b93360eeb"
},
"platform": {
"architecture": "amd64",
"os": "linux"
}
}
]
}
The digest blob in this index file includes the details for all of the blobs that collectively comprise the godlovedc/lolcow image:
$ cat ./blobs/sha256/f2a852991b0a36a9f3d6b2a33b98a461e9ede8393482f0deb5287afcbae2ce10 | jq
{
"schemaVersion": 2,
"config": {
"mediaType": "application/vnd.oci.image.config.v1+json",
"digest": "sha256:73d5b1025fbfa138f2cacf45bbf3f61f7de891559fa25b28ab365c7d9c3cbd82",
"size": 3410
},
"layers": [
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:9fb6c798fa41e509b58bccc5c29654c3ff4648b608f5daa67c1aab6a7d02c118",
"size": 47536248
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:3b61febd4aefe982e0cb9c696d415137384d1a01052b50a85aae46439e15e49a",
"size": 848
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:9d99b9777eb02b8943c0e72d7a7baec5c782f8fd976825c9d3fb48b3101aacc2",
"size": 621
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:d010c8cf75d7eb5d2504d5ffa0d19696e8d745a457dd8d28ec6dd41d3763617e",
"size": 853
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:7fac07fb303e0589b9c23e6f49d5dc1ff9d6f3c8c88cabe768b430bdb47f03a9",
"size": 169
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:8e860504ff1ee5dc7953672d128ce1e4aa4d8e3716eb39fe710b849c64b20945",
"size": 56355961
}
]
}
The digest blob referenced in the index.json file references the following configuration file:
$ cat ./blobs/sha256/73d5b1025fbfa138f2cacf45bbf3f61f7de891559fa25b28ab365c7d9c3cbd82 | jq
{
"created": "2017-09-21T18:37:47.278336798Z",
"architecture": "amd64",
"os": "linux",
"config": {
"Env": [
"PATH=/usr/games:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"LC_ALL=C"
],
"Entrypoint": [
"/bin/sh",
"-c",
"fortune | cowsay | lolcat"
]
},
"rootfs": {
"type": "layers",
"diff_ids": [
"sha256:a2022691bf950a72f9d2d84d557183cb9eee07c065a76485f1695784855c5193",
"sha256:ae620432889d2553535199dbdd8ba5a264ce85fcdcd5a430974d81fc27c02b45",
"sha256:c561538251751e3685c7c6e7479d488745455ad7f84e842019dcb452c7b6fecc",
"sha256:f96e6b25195f1b36ad02598b5d4381e41997c93ce6170cab1b81d9c68c514db0",
"sha256:7f7a065d245a6501a782bf674f4d7e9d0a62fa6bd212edbf1f17bad0d5cd0bfc",
"sha256:70ca7d49f8e9c44705431e3dade0636a2156300ae646ff4f09c904c138728839"
]
},
"history": [
{
"created": "2017-09-18T23:31:37.453092323Z",
"created_by": "/bin/sh -c #(nop) ADD file:5ed435208da6621b45db657dd6549ee132cde58c4b6763920030794c2f31fbc0 in / "
},
{
"created": "2017-09-18T23:31:38.196268404Z",
"created_by": "/bin/sh -c set -xe \t\t&& echo '#!/bin/sh' > /usr/sbin/policy-rc.d \t&& echo 'exit 101' >> /usr/sbin/policy-rc.d \t&& chmod +x /usr/sbin/policy-rc.d \t\t&& dpkg-divert --local --rename --add /sbin/initctl \t&& cp -a /usr/sbin/policy-rc.d /sbin/initctl \t&& sed -i 's/^exit.*/exit 0/' /sbin/initctl \t\t&& echo 'force-unsafe-io' > /etc/dpkg/dpkg.cfg.d/docker-apt-speedup \t\t&& echo 'DPkg::Post-Invoke { \"rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true\"; };' > /etc/apt/apt.conf.d/docker-clean \t&& echo 'APT::Update::Post-Invoke { \"rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true\"; };' >> /etc/apt/apt.conf.d/docker-clean \t&& echo 'Dir::Cache::pkgcache \"\"; Dir::Cache::srcpkgcache \"\";' >> /etc/apt/apt.conf.d/docker-clean \t\t&& echo 'Acquire::Languages \"none\";' > /etc/apt/apt.conf.d/docker-no-languages \t\t&& echo 'Acquire::GzipIndexes \"true\"; Acquire::CompressionTypes::Order:: \"gz\";' > /etc/apt/apt.conf.d/docker-gzip-indexes \t\t&& echo 'Apt::AutoRemove::SuggestsImportant \"false\";' > /etc/apt/apt.conf.d/docker-autoremove-suggests"
},
{
"created": "2017-09-18T23:31:38.788043199Z",
"created_by": "/bin/sh -c rm -rf /var/lib/apt/lists/*"
},
{
"created": "2017-09-18T23:31:39.411670721Z",
"created_by": "/bin/sh -c sed -i 's/^#\\s*\\(deb.*universe\\)$/\\1/g' /etc/apt/sources.list"
},
{
"created": "2017-09-18T23:31:40.055188541Z",
"created_by": "/bin/sh -c mkdir -p /run/systemd && echo 'docker' > /run/systemd/container"
},
{
"created": "2017-09-18T23:31:40.215057796Z",
"created_by": "/bin/sh -c #(nop) CMD [\"/bin/bash\"]",
"empty_layer": true
},
{
"created": "2017-09-21T18:37:46.483638061Z",
"created_by": "/bin/sh -c apt-get update && apt-get install -y fortune cowsay lolcat"
},
{
"created": "2017-09-21T18:37:47.041333952Z",
"created_by": "/bin/sh -c #(nop) ENV PATH=/usr/games:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"empty_layer": true
},
{
"created": "2017-09-21T18:37:47.170535967Z",
"created_by": "/bin/sh -c #(nop) ENV LC_ALL=C",
"empty_layer": true
},
{
"created": "2017-09-21T18:37:47.278336798Z",
"created_by": "/bin/sh -c #(nop) ENTRYPOINT [\"/bin/sh\" \"-c\" \"fortune | cowsay | lolcat\"]",
"empty_layer": true
}
]
}
Even when all OCI blobs are already in Singularity’s local cache, repeated image pulls cause both these last-two JSON object files, as well as the oci-layout and index.json files, to be updated.
Building Containers for Singularity from OCI Images¶
Working Locally from the Singularity Command Line: oci Boostrap Agent¶
The example detailed in the previous section can be used to illustrate how a SIF file for use by Singularity can be created from the local cache - an albeit contrived example, that works because the Singularity cache is compliant with the OCI Image Layout Specification.
Note
Of course, the oci bootstrap agent can be applied to any bundle that is compliant with the OCI Image Layout Specification - not just the Singularity cache, as created by executing a Singularity pull command.
In this local case, the build command of Singularity makes use of the oci boostrap agent as follows:
$ singularity build ~/lolcow_oci_cache.sif oci://$HOME/.singularity/cache/oci:a692b57abc43035b197b10390ea2c12855d21649f2ea2cc28094d18b93360eeb
INFO: Starting build...
Getting image source signatures
Skipping fetch of repeat blob sha256:9fb6c798fa41e509b58bccc5c29654c3ff4648b608f5daa67c1aab6a7d02c118
Skipping fetch of repeat blob sha256:3b61febd4aefe982e0cb9c696d415137384d1a01052b50a85aae46439e15e49a
Skipping fetch of repeat blob sha256:9d99b9777eb02b8943c0e72d7a7baec5c782f8fd976825c9d3fb48b3101aacc2
Skipping fetch of repeat blob sha256:d010c8cf75d7eb5d2504d5ffa0d19696e8d745a457dd8d28ec6dd41d3763617e
Skipping fetch of repeat blob sha256:7fac07fb303e0589b9c23e6f49d5dc1ff9d6f3c8c88cabe768b430bdb47f03a9
Skipping fetch of repeat blob sha256:8e860504ff1ee5dc7953672d128ce1e4aa4d8e3716eb39fe710b849c64b20945
Copying config sha256:73d5b1025fbfa138f2cacf45bbf3f61f7de891559fa25b28ab365c7d9c3cbd82
3.33 KiB / 3.33 KiB [======================================================] 0s
Writing manifest to image destination
Storing signatures
INFO: Creating SIF file...
INFO: Build complete: /home/vagrant/lolcow_oci_cache.sif
As can be seen, this results in the SIF file lolcow_oci_cache.sif in the user’s home directory.
The syntax for the oci boostrap agent requires some elaboration, however. In this case, and as illustrated above, $HOME/.singularity/cache/oci has content:
$ ls
blobs index.json oci-layout
In other words, it is the $OCI_BUNDLE_DIR containing the data and metadata that collectively comprise the image layed out in accordance with the OCI Image Layout Specification as discussed previously - the same data and metadata that are assembled into a single SIF file through the build process. However,
$ singularity build ~/lolcow_oci_cache.sif oci://$HOME/.singularity/cache/oci
INFO: Starting build...
FATAL: While performing build: conveyor failed to get: more than one image in oci, choose an image
does not uniquely specify an image from which to bootstrap the build process. In other words, there are multiple images referenced via org.opencontainers.image.ref.name in the index.json file. By appending :a692b57abc43035b197b10390ea2c12855d21649f2ea2cc28094d18b93360eeb to oci in this example, the image is uniquely specified, and the container created in SIF (as illustrated previously).
Note
Executing the Singularity pull command multiple times on the same image produces multiple org.opencontainers.image.ref.name entries in the index.json file. Appending the value of the unique org.opencontainers.image.ref.name allows for use of the oci boostrap agent.
Working Locally from the Singularity Command Line: oci-archive Boostrap Agent¶
OCI archives, i.e., tar files obeying the OCI Image Layout Specification as discussed previously, can seed creation of a container for Singularity. In this case, use is made of the oci-archive bootstrap agent.
To illustrate this agent, it is convenient to build the archive from the Singularity cache. After a single pull of the godlovedc/lolcow image from Docker Hub, a tar format archive can be generated from the $HOME/.singularity/cache/oci directory as follows:
$ tar cvf $HOME/godlovedc_lolcow.tar *
blobs/
blobs/sha256/
blobs/sha256/73d5b1025fbfa138f2cacf45bbf3f61f7de891559fa25b28ab365c7d9c3cbd82
blobs/sha256/8e860504ff1ee5dc7953672d128ce1e4aa4d8e3716eb39fe710b849c64b20945
blobs/sha256/9d99b9777eb02b8943c0e72d7a7baec5c782f8fd976825c9d3fb48b3101aacc2
blobs/sha256/3b61febd4aefe982e0cb9c696d415137384d1a01052b50a85aae46439e15e49a
blobs/sha256/9fb6c798fa41e509b58bccc5c29654c3ff4648b608f5daa67c1aab6a7d02c118
blobs/sha256/d010c8cf75d7eb5d2504d5ffa0d19696e8d745a457dd8d28ec6dd41d3763617e
blobs/sha256/f2a852991b0a36a9f3d6b2a33b98a461e9ede8393482f0deb5287afcbae2ce10
blobs/sha256/7fac07fb303e0589b9c23e6f49d5dc1ff9d6f3c8c88cabe768b430bdb47f03a9
index.json
oci-layout
The native container lolcow_oci_tarfile.sif for use by Singularity can be created by issuing the build command as follows:
$ singularity build lolcow_oci_tarfile.sif oci-archive://godlovedc_lolcow.tar
Build target already exists. Do you want to overwrite? [N/y] y
INFO: Starting build...
Getting image source signatures
Skipping fetch of repeat blob sha256:9fb6c798fa41e509b58bccc5c29654c3ff4648b608f5daa67c1aab6a7d02c118
Skipping fetch of repeat blob sha256:3b61febd4aefe982e0cb9c696d415137384d1a01052b50a85aae46439e15e49a
Skipping fetch of repeat blob sha256:9d99b9777eb02b8943c0e72d7a7baec5c782f8fd976825c9d3fb48b3101aacc2
Skipping fetch of repeat blob sha256:d010c8cf75d7eb5d2504d5ffa0d19696e8d745a457dd8d28ec6dd41d3763617e
Skipping fetch of repeat blob sha256:7fac07fb303e0589b9c23e6f49d5dc1ff9d6f3c8c88cabe768b430bdb47f03a9
Skipping fetch of repeat blob sha256:8e860504ff1ee5dc7953672d128ce1e4aa4d8e3716eb39fe710b849c64b20945
Copying config sha256:73d5b1025fbfa138f2cacf45bbf3f61f7de891559fa25b28ab365c7d9c3cbd82
3.33 KiB / 3.33 KiB [======================================================] 0s
Writing manifest to image destination
Storing signatures
INFO: Creating SIF file...
INFO: Build complete: lolcow_oci_tarfile.sif
This assumes that the tar file exists in the current working directory.
Note
Cache maintenance is a manual process at the current time. In other words, the cache can be cleared by carefully issuing the command rm -rf $HOME/.singularity/cache. Of course, this will clear the local cache of all downloaded images.
Note
Because the layers of a Docker image as well as the blobs of an OCI image are already gzip compressed, there is a minimal advantage to having compressed archives representing OCI images. For this reason, the build detailed above boostraps a SIF file for use by Singularity from only a tar file, and not a tar.gz file.
Working from the Singularity Command Line with Remotely Hosted Images¶
In the previous section, an OCI archive was created from locally available OCI blobs and metadata; the resulting tar file served to bootstrap the creation of a container for Singularity in SIF via the oci-archive agent. Typically, however, OCI archives of interest are remotely hosted. Consider, for example, an Alpine Linux OCI archive stored in Amazon S3 storage. Because such an archive can be retrieved via secure HTTP, the following pull command results in a local copy as follows:
$ singularity pull https://s3.amazonaws.com/singularity-ci-public/alpine-oci-archive.tar
1.98 MiB / 1.98 MiB [==================================================================================] 100.00% 7.48 MiB/s 0s
Thus https (and http) are additional bootstrap agents available to seed development of containers for Singularity.
It is worth noting that the OCI image specfication compliant contents of this archive are:
$ tar tvf alpine-oci-archive.tar
drwxr-xr-x 1000/1000 0 2018-06-25 14:45 blobs/
drwxr-xr-x 1000/1000 0 2018-06-25 14:45 blobs/sha256/
-rw-r--r-- 1000/1000 585 2018-06-25 14:45 blobs/sha256/b1a7f144ece0194921befe57ab30ed1fd98c5950db7996719429020986092058
-rw-r--r-- 1000/1000 348 2018-06-25 14:45 blobs/sha256/d0ff39a54244ba25ac7447f19941765bee97b05f37ceb438a72e80c9ed39854a
-rw-r--r-- 1000/1000 2065537 2018-06-25 14:45 blobs/sha256/ff3a5c916c92643ff77519ffa742d3ec61b7f591b6b7504599d95a4a41134e28
-rw-r--r-- 1000/1000 296 2018-06-25 14:45 index.json
-rw-r--r-- 1000/1000 31 2018-06-25 14:45 oci-layout
Proceeding as before, for a (now) locally available OCI archive, a SIF file can be produced by executing:
$ singularity build alpine_oci_archive.sif oci-archive://alpine-oci-archive.tar
INFO: Starting build...
Getting image source signatures
Copying blob sha256:ff3a5c916c92643ff77519ffa742d3ec61b7f591b6b7504599d95a4a41134e28
1.97 MiB / 1.97 MiB [======================================================] 0s
Copying config sha256:b1a7f144ece0194921befe57ab30ed1fd98c5950db7996719429020986092058
585 B / 585 B [============================================================] 0s
Writing manifest to image destination
Storing signatures
INFO: Creating SIF file...
INFO: Build complete: alpine_oci_archive.sif
The resulting SIF file can be validated as follows, for example:
$ ./alpine_oci_archive.sif
Singularity> cat /etc/os-release
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.7.0
PRETTY_NAME="Alpine Linux v3.7"
HOME_URL="http://alpinelinux.org"
BUG_REPORT_URL="http://bugs.alpinelinux.org"
Singularity>
$
Note
The http and https bootstrap agents can only be used to pull OCI archives from where they are hosted.
In working with remotely hosted OCI image archives then, a two-step workflow is required to produce SIF files for native use by Singularity:
Transfer of the image to local storage via the
https(orhttp) bootstrap agent. The Singularitypullcommand achieves this.Creation of a SIF file via the
oci-archivebootstrap agent. The Singularitybuildcommand achieves this.
Note
Though a frequently asked question, the distribution of OCI images remains out of scope. In other words, there is no OCI endorsed distribution method or registry.
Established with nothing more than a Web server then, any individual, group or organization, could host OCI archives. This might be particularly appealing, for example, for organizations having security requirements that preclude access to public registries such as Docker Hub. Other that having a very basic hosting capability, OCI archives need only comply to the OCI Image Layout Specification as discussed previously.
Working with Definition Files: Mandatory Header Keywords¶
Three, new bootstrap agents have been introduced as a consequence of compliance with the OCI Image Specification - assuming http and https are considered together. In addition to bootstrapping images for Singularity completely from the command line, definition files can be employed.
As above, the OCI image layout compliant Singularity cache can be employed to create SIF containers; the definition file, lolcow-oci.def, equivalent is:
Bootstrap: oci
From: .singularity/cache/oci:a692b57abc43035b197b10390ea2c12855d21649f2ea2cc28094d18b93360eeb
Recall that the colon-appended string in this file uniquely specifies the org.opencontainers.image.ref.name of the desired image, as more than one possibility exists in the index.json file. The corresponding build command is:
$ sudo singularity build ~/lolcow_oci_cache.sif lolcow-oci.def
WARNING: Authentication token file not found : Only pulls of public images will succeed
Build target already exists. Do you want to overwrite? [N/y] y
INFO: Starting build...
Getting image source signatures
Copying blob sha256:9fb6c798fa41e509b58bccc5c29654c3ff4648b608f5daa67c1aab6a7d02c118
45.33 MiB / 45.33 MiB [====================================================] 0s
Copying blob sha256:3b61febd4aefe982e0cb9c696d415137384d1a01052b50a85aae46439e15e49a
848 B / 848 B [============================================================] 0s
Copying blob sha256:9d99b9777eb02b8943c0e72d7a7baec5c782f8fd976825c9d3fb48b3101aacc2
621 B / 621 B [============================================================] 0s
Copying blob sha256:d010c8cf75d7eb5d2504d5ffa0d19696e8d745a457dd8d28ec6dd41d3763617e
853 B / 853 B [============================================================] 0s
Copying blob sha256:7fac07fb303e0589b9c23e6f49d5dc1ff9d6f3c8c88cabe768b430bdb47f03a9
169 B / 169 B [============================================================] 0s
Copying blob sha256:8e860504ff1ee5dc7953672d128ce1e4aa4d8e3716eb39fe710b849c64b20945
53.75 MiB / 53.75 MiB [====================================================] 0s
Copying config sha256:73d5b1025fbfa138f2cacf45bbf3f61f7de891559fa25b28ab365c7d9c3cbd82
3.33 KiB / 3.33 KiB [======================================================] 0s
Writing manifest to image destination
Storing signatures
INFO: Creating SIF file...
INFO: Build complete: /home/vagrant/lolcow_oci_cache.sif
Required use of sudo allows Singularity to build the SIF container lolcow_oci_cache.sif.
When it comes to OCI archives, the definition file, lolcow-ocia.def corresponding to the command-line invocation above is:
Bootstrap: oci-archive
From: godlovedc_lolcow.tar
Applying build as follows
$ sudo singularity build lolcow_oci_tarfile.sif lolcow-ocia.def
WARNING: Authentication token file not found : Only pulls of public images will succeed
INFO: Starting build...
Getting image source signatures
Skipping fetch of repeat blob sha256:9fb6c798fa41e509b58bccc5c29654c3ff4648b608f5daa67c1aab6a7d02c118
Skipping fetch of repeat blob sha256:3b61febd4aefe982e0cb9c696d415137384d1a01052b50a85aae46439e15e49a
Skipping fetch of repeat blob sha256:9d99b9777eb02b8943c0e72d7a7baec5c782f8fd976825c9d3fb48b3101aacc2
Skipping fetch of repeat blob sha256:d010c8cf75d7eb5d2504d5ffa0d19696e8d745a457dd8d28ec6dd41d3763617e
Skipping fetch of repeat blob sha256:7fac07fb303e0589b9c23e6f49d5dc1ff9d6f3c8c88cabe768b430bdb47f03a9
Skipping fetch of repeat blob sha256:8e860504ff1ee5dc7953672d128ce1e4aa4d8e3716eb39fe710b849c64b20945
Copying config sha256:73d5b1025fbfa138f2cacf45bbf3f61f7de891559fa25b28ab365c7d9c3cbd82
3.33 KiB / 3.33 KiB [======================================================] 0s
Writing manifest to image destination
Storing signatures
INFO: Creating SIF file...
INFO: Build complete: lolcow_oci_tarfile.sif
results in the SIF container lolcow_oci_tarfile.sif.
Working with Definition Files: Additonal Considerations¶
In working with definition files, the following additional considerations arise:
In addition to the mandatory header keywords documented above, optional header keywords are possible additions to OCI bundle and/or archive bootstrap definition files.
As distribution of OCI bundles and/or archives is out of the Initiative’s scope, so is the authentication required to access private images and/or registries.
The direction of execution follows along the same lines as described above. Of course, the SIF container’s metadata will make clear the
runscriptthrough application of theinspectcommand as described previously.Container metadata will also reveal whether or not a given SIF file was bootstrapped from an OCI bundle or archive; for example, below it is evident that an OCI archive was employed to bootstrap creation of the SIF file:
$ singularity inspect --labels lolcow_oci_tarfile.sif | jq
{
"org.label-schema.build-date": "Sunday_27_January_2019_0:5:29_UTC",
"org.label-schema.schema-version": "1.0",
"org.label-schema.usage.singularity.deffile.bootstrap": "oci-archive",
"org.label-schema.usage.singularity.deffile.from": "godlovedc_lolcow.tar",
"org.label-schema.usage.singularity.version": "3.0.3-1"
}
Best Practices¶
Singularity can make use of most Docker and OCI images without complication. However, there exist known cases where complications can arise. Thus a brief compilation of best practices follows below.
Accounting for trust
Docker containers allow for privilege escalation. In a
Dockerfile, for example, theUSERinstruction allows for user and/or group settings to be made in the Linux operating environment. The trust model in Singularity is completely different: Singularity allows untrusted users to run untrusted containers in a trusted way. Because Singularity containers embodied as SIF files execute in user space, there is no possibility for privilege escalation. In other words, those familiar with Docker, should not expect access to elevated user permissions; and as a corollary, use of theUSERinstruction must be avoided.Singularity does, however, allow for fine-grained control over the permissions that containers require for execution. Given that Singularilty executes in user space, it is not surprising that permissions need to be externally established for the container through use of the
capabilitycommand. Detailed elsewhere in this documentation, Singularity allows users and/or groups to be granted/revoked authorized capabilties. Owing to Singularity’s trust model, this fundamental best practice can be stated as follows:“Employ
singularity capabilityto manage execution privileges for containers”
Maintaining containers built from Docker and OCI images
SIF files created by boostrapping from Docker or OCI images are, of course, only as current as the most recent Singularity
pull. Subsequent retrievals may result in containers that are built and/or behave differently, owing to changes in the correspondingDockerfile. A prudent practice then, for maintaining containers of value, is based upon use of Singularity definition files. Styled and implemented after aDockerfileretrieved at some point in time, use ofdiffon subsequent versions of this same file, can be employed to inform maintenance of the corresponding Singularity definition file. Understanding build specifications at this level of detail places container creators in a much more sensible position prior to signing with an encrypted key. Thus the best practice is:“Maintain detailed build specifications for containers, rather than opaque runtimes”
Working with environment variables
In a
Dockerfile, environment variables are declared as key-value pairs through use of theENVinstruction. Declaration in the build specification for a container is advised, rather than relying upon user (e.g.,.bashrc,.profile) or system-wide configuration files for interactive shells. Should aDockerfilebe converted into a definition file for Singularity, as suggested in the container-maintenance best practice above, environment variables can be explicitly represented asENVinstructions that have been converted into entries in the%environmentsection, respectively. This best practice can be stated as follows:“Define environment variables in container specifications, not interactive shells”
Installation to
/rootDocker and OCI container’s are typically run as the
rootuser; therefore,/root(this user’s$HOMEdirectory) will be the installation target when$HOMEis specified. Installation to/rootmay prove workable in some circumstances - e.g., while the container is executing, or if read-only access is required to this directory after installation. In general, however, because this is therootdirectory conventional wisdom suggests this practice be avoided. Thus the best practice is:“Avoid installations that make use of
/root.”
Read-only
/filesystemSingularity mounts a container’s
/filesystem in read-only mode. To ensure a Docker container meets Singularity’s requirements, it may prove useful to executedocker run --read-only --tmpfs /run --tmpfs /tmp godlovedc/lolcow. The best practioce here is:“Ensure Docker containers meet Singularity’s read-only
/filesystem requirement”
Installation to
$HOMEor$TMPIn making use of Singularity, it is common practice for
$USERto be automatically mounted on$HOME, and for$TMPalso to be mounted. To avoid the side effects (e.g., ‘missing’ or conflicting files) that might arise as a consequence of executingmountcommands then, the best practice is:“Avoid placing container ‘valuables’ in
$HOMEor$TMP.”A detailed review of the container’s build specification (e.g., its
Dockerfile) may be required to ensure this best practice is adhered to.
Current library caches
Irrespective of containers, a common runtime error stems from failing to locate shared libraries required for execution. Suppose now there exists a requirement for symbolically linked libraries within a Singularity container. If the builld process that creates the container fails to update the cache, then it is quite likely that (read-only) execution of this container will result in the common error of missing libraries. Upon investigation, it is likely revealed that the library exists, just not the required symbolic links. Thus the best practice is:
“Ensure calls to
ldconfigare executed towards the end ofbuildspecifications (e.g.,Dockerfile), so that the library cache is updated when the container is created.”
Use of plain-text passwords for authentication
For obvious reasons, it is desireable to completely avoid use of plain-text passwords. Therefore, for interactive sessions requiring authentication, use of the
--docker-loginoption for Singularity’spullandbuildcommands is recommended. At the present time, the only option available for non-interactive use is to embed plain-text passwords into environment variables. Because the Sylabs Cloud Singularity Library employs time-limited API tokens for authentication, use of SIF containers hosted through this service provides a more secure means for both interactive and non-interactive use. This best practice is:“Avoid use of plain-text passwords”
Execution ambiguity
Short of converting an entire
Dockerfileinto a Singularity definition file, informed specification of the%runscriptentry in the def file removes any ambiguity associated withENTRYPOINTversusCMDand ultimately execution precedence. Thus the best practice is:“Employ Singularity’s
%runscriptby default to avoid execution ambiguity”Note that the
ENTRYPOINTcan be bypassed completely, e.g.,docker run -i -t --entrypoint /bin/bash godlovedc/lolcow. This allows for an interactive session within the container, that may prove useful in validating the built runtime.
Best practices emerge from experience. Contributions that allow additional experiences to be shared as best practices are always encouraged. Please refer to Contributing for additional details.
Troubleshooting¶
In making use of Docker and OCI images through Singularity the need to troubleshoot may arise. A brief compilation of issues and their resolution is provided here.
Authentication issues
Authentication is required to make use of Docker-style private images and/or private registries. Examples involving private images hosted by the public Docker Hub were provided above, whereas the NVIDIA GPU Cloud was used to illustrate access to a private registry. Even if the intended use of containers is non-interactive, issues in authenticating with these image-hosting services are most easily addressed through use of the
--docker-loginoption that can be appended to a Singularitypullrequest. As soon as image signatures and blobs start being received, authentication credentials have been validated, and the imagepullcan be cancelled.
Execution mismatches
Execution intentions are detailed through specification files - i.e., the
Dockerfilein the case of Docker images. However, intentions and precedence aside, the reality of executing a container may not align with expectations. To alleviate this mismatch, use ofsingularity inspect --runscript <somecontainer>.sifdetails the effective runscript - i.e., the one that is actually being executed. Of course, the ultimate solution to this issue is to develop and maintain Singularity definition files for containers of interest.
More than one image in the OCI bundle directory
As illustrated above, and with respect to the bootstrap agent
oci://$OCI_BUNDLE_DIR, a fatal error is generated when more than one image is referenced in the$OCI_BUNDLE_DIR/index.jsonfile. The workaround shared previously was to append the bootstrap directive with the unique reference name for the image of interest - i.e.,oci://$OCI_BUNDLE_DIR:org.opencontainers.image.ref.name. Because it may take some effort to locate the reference name for an image of interest, an even simpler solution is to ensure that each$OCI_BUNDLE_DIRcontains at most a single image.
Cache maintenance
Maintenance of the Singularity cache (i.e.,
$HOME/.singularity/cache) requires manual intervention at this time. By carefully issuing the commandrm -rf $HOME/.singularity/cache, its local cache will be cleared of all downloaded images.
The
httpandhttpsarepullonly boostrap agents
httpandhttpsare the only examples ofpullonly boostrap agents. In other words, when used with Singularity’spullcommand, the result is a local copy of, for example, an OCI archive image. This means that a subsequent step is necessary to actually create a SIF container for use by Singularity - a step involving theoci-archivebootstrap agent in the case of an OCI image archive.
Like best practices, troubleshooting scenarios and solutions emerge from experience. Contributions that allow additional experiences to be shared are always encouraged. Please refer to Contributing for additional details.