Encrypted Containers

Users can build a secure, confidential container environment by encrypting the root file system.

Overview

Apptainer provides a feature to build and run encrypted containers to allow users to encrypt the file system image within a SIF. This encryption can be performed using either a passphrase or asymmetrically via an RSA key pair in Privacy Enhanced Mail (PEM/PKCS1) format. The container is encrypted in transit, at rest, and even while running. In other words, there is no intermediate, decrypted version of the container on disk. Container decryption occurs at runtime in memory.

Note

This feature utilizes the Linux dm-crypt library and cryptsetup utility and requires cryptsetup version of >= 2.0.0. This version should be standard with recent Linux versions, but users of older Linux versions may have to update.

Note

We use Privileged Encryption to represent the case of an encrypted image built by the root user and decrypted with a suid installation. The partition type in a SIF image in this case wil be shown as Encrypted squashfs.

We use Unprivileged Encryption to represent the case of an encrypted image built by normal unprivileged users and decrypted either without a suid installion or with the --userns option. The partition type in a SIF image in this case will be shown as Gocryptfs squashfs. This case requires unprivileged user namespaces.

Encrypting a container

A container can be encrypted either by supplying a plaintext passphrase or a PEM file containing an asymmetric RSA public key. Of these two methods the PEM file is more secure and is therefore recommended for production use.

An -e|--encrypt flag to apptainer build is used to indicate that the container needs to be encrypted.

A passphrase or a key-file used to perform the encryption is supplied at build time via an environment variable or a command line option.

Encryption Method

Environment Variable

Commandline Option

Passphrase

APPTAINER_ENCRYPTION_PASSPHRASE

--passphrase

Asymmetric Key (PEM)

APPTAINER_ENCRYPTION_PEM_PATH

--pem-path

The -e|--encrypt flag is implicitly set when the --passphrase or --pem-path flags are passed with the build command. If multiple encryption related flags and/or environment variables are set, the following precedence is respected.

  1. --pem-path

  2. --passphrase

  3. APPTAINER_ENCRYPTION_PEM_PATH

  4. APPTAINER_ENCRYPTION_PASSPHRASE

Passphrase Encryption

Note

Passphrase encryption is less secure than encrypting containers using an RSA key pair (detailed below). Passphrase encryption is provided as a convenience, and as a way for users to familiarize themselves with the encrypted container workflow, but users running encrypted containers in production are encouraged to use asymmetric keys.

In case of plaintext passphrase encryption, a passphrase is supplied by one of the following methods.

Encrypting with a passphrase interactively (Privileged Encryption)

$ sudo apptainer build --passphrase encrypted.sif encrypted.def
Enter encryption passphrase: <secret>
INFO:    Starting build...

Using an environment variable (Privileged Encryption)

$ sudo APPTAINER_ENCRYPTION_PASSPHRASE=<secret> apptainer build --encrypt encrypted.sif encrypted.def
Starting build...

In this case it is necessary to use the --encrypt flag since the presence of an environment variable alone will not trigger the encrypted build workflow.

While this example shows how an environment variable can be used to set a passphrase, you should set the environment variable in a way that will not record your passphrase on the command line. For instance, you could save a plain text passphrase in a file (e.g. secret.txt) and use it like so.

$ export APPTAINER_ENCRYPTION_PASSPHRASE=$(cat secret.txt)

$ sudo -E apptainer build --encrypt encrypted.sif encrypted.def
Starting build...

Image info verification (Privileged Encryption)

$ apptainer sif list encrypted.sif

Example:

------------------------------------------------------------------------------
ID   |GROUP   |LINK    |SIF POSITION (start-end)  |TYPE
------------------------------------------------------------------------------
1    |1       |NONE    |32176-32223               |Def.FILE
2    |1       |NONE    |32223-34202               |JSON.Generic
3    |1       |NONE    |34202-34292               |JSON.Generic
4    |1       |NONE    |36864-17608704            |FS (Encrypted squashfs/*System/arm64)
5    |1       |4       |17608704-17609449         |Cryptographic Message (PEM/RSA-OAEP)

Note that partition 4 is type Encrypted squashfs.

Encrypting with a passphrase interactively (Unprivileged Encryption)

$ apptainer build --passphrase encrypted.sif encrypted.def
Enter encryption passphrase: <secret>
INFO:    Starting build...

Using an environment variable (Unprivileged Encryption)

$ APPTAINER_ENCRYPTION_PASSPHRASE=<secret> apptainer build encrypted.sif encrypted.def
Starting build...
$ export APPTAINER_ENCRYPTION_PASSPHRASE=$(cat secret.txt)

$ apptainer build encrypted.sif encrypted.def
Starting build...

Image info verification (Unprivileged Encryption)

$ apptainer sif list encrypted.sif

Example:

------------------------------------------------------------------------------
ID   |GROUP   |LINK    |SIF POSITION (start-end)  |TYPE
------------------------------------------------------------------------------
1    |1       |NONE    |32176-32215               |Def.FILE
2    |1       |NONE    |32215-36269               |JSON.Generic
3    |1       |NONE    |36269-36465               |JSON.Generic
4    |1       |NONE    |36864-26386432            |FS (Gocryptfs squashfs/*System/arm64)

Note that partition 4 is type Gocryptfs squashfs.

PEM File Encryption

Apptainer currently supports RSA encryption using a public/private key-pair. Keys are supplied in PEM format. The public key is used to encrypt containers that can be decrypted on a host that has access to the secret private key.

You can create a pair of RSA keys suitable for encrypting your container with the ssh-keygen command, and then create a PEM file with a few specific flags like so:

# Generate a key pair
$ ssh-keygen -t rsa -b 4096 -m pem -N ''
Generating public/private rsa key pair.
Enter file in which to save the key (/home/vagrant/.ssh/id_rsa): rsa
[snip...]

# Convert the public key to PEM PKCS1 format
$ ssh-keygen -f ./rsa.pub -e -m pem >rsa_pub.pem

# Rename the private key (already PEM PKCS1) to a nice name
$ mv rsa rsa_pri.pem

You would use the rsa_pub.pem file to encrypt your container and the rsa_pri.pem file to run it. Be sure to keep the private key safe and private, because it is not encrypted itself.

Encrypting with a command line option (Privileged Encryption)

$ sudo apptainer build --pem-path=rsa_pub.pem encrypted.sif encrypted.def
Starting build...

Encrypting with an environment variable (Privileged Encryption)

$ sudo APPTAINER_ENCRYPTION_PEM_PATH=rsa_pub.pem apptainer build --encrypt encrypted.sif encrypted.def
Starting build...

In this case it is necessary to use the --encrypt flag since the presence of an environment variable alone will not trigger the encrypted build workflow.

$ apptainer sif list encrypted.sif

Example:

------------------------------------------------------------------------------
ID   |GROUP   |LINK    |SIF POSITION (start-end)  |TYPE
------------------------------------------------------------------------------
1    |1       |NONE    |32176-32215               |Def.FILE
2    |1       |NONE    |32215-36269               |JSON.Generic
3    |1       |NONE    |36269-36465               |JSON.Generic
4    |1       |NONE    |36864-42954752            |FS (Encrypted squashfs/*System/arm64)
5    |1       |4       |42954752-42955497         |Cryptographic Message (PEM/RSA-OAEP)

Note that partition 4 is type Encrypted squashfs.

Encrypting with a command line option (Unprivileged Encryption)

$ apptainer build --pem-path=rsa_pub.pem encrypted.sif encrypted.def
Starting build...

Encrypting with an environment variable (Unprivileged Encryption)

$ APPTAINER_ENCRYPTION_PEM_PATH=rsa_pub.pem apptainer build --encrypt encrypted.sif encrypted.def
Starting build...

In this case it is necessary to use the --encrypt flag since the presence of an environment variable alone will not trigger the encrypted build workflow.

$ apptainer sif list encrypted.sif

Example:

------------------------------------------------------------------------------
ID   |GROUP   |LINK    |SIF POSITION (start-end)  |TYPE
------------------------------------------------------------------------------
1    |1       |NONE    |32176-32215               |Def.FILE
2    |1       |NONE    |32215-36269               |JSON.Generic
3    |1       |NONE    |36269-36465               |JSON.Generic
4    |1       |NONE    |36864-26386432            |FS (Gocryptfs squashfs/*System/arm64)
5    |1       |4       |26386432-26387177         |Cryptographic Message (PEM/RSA-OAEP)

Note that partition 4 is type Gocryptfs squashfs.

Running an encrypted container

To run, shell, or exec an encrypted image, credentials to decrypt the image need to be supplied at runtime either in a key-file or a plaintext passphrase.

Running a container encrypted with a passphrase

A passphrase can be supplied at runtime by either of the ways listed in the sections above.

Running with a passphrase interactively

$ apptainer run --passphrase encrypted.sif
Enter passphrase for encrypted container: <secret>

Running with a passphrase in an environment variable

$ APPTAINER_ENCRYPTION_PASSPHRASE="secret" apptainer run encrypted.sif

While this example shows how an environment variable can be used to set a passphrase, you should set the environment variable in a way that will not record your passphrase on the command line. For instance, you could save a plain text passphrase in a file (e.g. secret.txt) and use it like so.

$ export APPTAINER_ENCRYPTION_PASSPHRASE=$(cat secret.txt)

$ apptainer run encrypted.sif

Running a container encrypted with a PEM file

A private key is supplied using either of the methods listed in the Encryption section above.

Running using a command line option

$ apptainer run --pem-path=rsa_pri.pem encrypted.sif

Running using an environment variable

$ APPTAINER_ENCRYPTION_PEM_PATH=rsa_pri.pem apptainer run encrypted.sif

When executing an Unprivileged Encryption encrypted image with an Apptainer suid installation, --userns is required.

$ apptainer run --pem-path=rsa_pri.pem encrypted.sif

The following error will be shown

FATAL:   container creation failed: mount hook function failure: mount /proc/self/fd/3->/usr/local/var/apptainer/mnt/session/rootfs error: while mounting image /proc/self/fd/3: gocryptfs requires user namespace, please add `--userns` option

Add --userns option

$ apptainer run --pem-path=rsa_pri.pem --userns encrypted.sif