'\" t
.\" Man page generated from reStructuredText.
.
.
.nr rst2man-indent-level 0
.
.de1 rstReportMargin
\\$1 \\n[an-margin]
level \\n[rst2man-indent-level]
level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
-
\\n[rst2man-indent0]
\\n[rst2man-indent1]
\\n[rst2man-indent2]
..
.de1 INDENT
.\" .rstReportMargin pre:
. RS \\$1
. nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin]
. nr rst2man-indent-level +1
.\" .rstReportMargin post:
..
.de UNINDENT
. RE
.\" indent \\n[an-margin]
.\" old: \\n[rst2man-indent\\n[rst2man-indent-level]]
.nr rst2man-indent-level -1
.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
..
.TH "CH-RUN" "1" "2024-03-16 10:47 UTC" "0.37" "Charliecloud"
.SH NAME
ch-run \- Run a command in a Charliecloud container
.SH SYNOPSIS
.INDENT 0.0
.INDENT 3.5
.sp
.EX
$ ch\-run [OPTION...] IMAGE \-\- COMMAND [ARG...]
.EE
.UNINDENT
.UNINDENT
.SH DESCRIPTION
.sp
Run command \fBCOMMAND\fP in a fully unprivileged Charliecloud container using
the image specified by \fBIMAGE\fP, which can be: (1)\ a path to a directory,
(2)\ the name of an image in \fBch\-image\fP storage (e.g.
\fBexample.com:5050/foo\fP) or, if the proper support is enabled, a SquashFS
archive. \fBch\-run\fP does not use any setuid or setcap helpers, even for
mounting SquashFS images with FUSE.
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.TP
.B \fB\-b\fP, \fB\-\-bind=SRC[:DST]\fP
Bind\-mount \fBSRC\fP at guest \fBDST\fP\&. The default destination if
not specified is to use the same path as the host; i.e., the default is
\fB\-\-bind=SRC:SRC\fP\&. Can be repeated.
.sp
With a read\-only image (the default), \fBDST\fP must exist. However, if
\fB\-\-write\fP or \fB\-\-write\-fake\fP are given, \fBDST\fP will be
created as an empty directory (possibly with the tmpfs overmount trick
described in \fI\%\-\-bind creates mount points within un\-writeable directories!\fP). In this case, \fBDST\fP must be
entirely within the image itself, i.e., \fBDST\fP cannot enter a
previous bind mount. For example, \fB\-\-bind /foo:/tmp/foo\fP will fail
because \fB/tmp\fP is shared with the host via bind\-mount (unless
\fB$TMPDIR\fP is set to something else or \fB\-\-private\-tmp\fP is
given).
.sp
Most images have ten directories \fB/mnt/[0\-9]\fP already available as
mount points.
.sp
Symlinks in \fBDST\fP are followed, and absolute links can have
surprising behavior. Bind\-mounting happens after namespace setup but
before pivoting into the container image, so absolute links use the host
root. For example, suppose the image has a symlink \fB/foo \-> /mnt\fP\&.
Then, \fB\-\-bind=/bar:/foo\fP will bind\-mount on the \fIhost’s\fP
\fB/mnt\fP, which is inaccessible on the host because namespaces are
already set up and \fIalso\fP inaccessible in the container because of the
subsequent pivot into the image. Currently, this problem is only detected
when \fBDST\fP needs to be created: \fBch\-run\fP will refuse to follow
absolute symlinks in this case, to avoid directory creation surprises.
.TP
.B \fB\-c\fP, \fB\-\-cd=DIR\fP
Initial working directory in container.
.TP
.B \fB\-\-env\-no\-expand\fP
Don’t expand variables when using \fB\-\-set\-env\fP\&.
.TP
.B \fB\-\-feature=FEAT\fP
If feature \fBFEAT\fP is enabled, exit with success. Valid values of
\fBFEAT\fP are \fBextglob\fP for extended globs, \fBseccomp\fP for
\fBseccomp(2)\fP, and \fBsquash\fP for squashfs archives.
.TP
.B \fB\-g\fP, \fB\-\-gid=GID\fP
Run as group \fBGID\fP within container.
.TP
.B \fB\-\-home\fP
Bind\-mount your host home directory (i.e., \fB$HOME\fP) at guest
\fB/home/$USER\fP, hiding any existing image content at that path.
Implies \fB\-\-write\-fake\fP so the mount point can be created if needed.
.TP
.B \fB\-j\fP, \fB\-\-join\fP
Use the same container (namespaces) as peer \fBch\-run\fP invocations.
.TP
.B \fB\-\-join\-pid=PID\fP
Join the namespaces of an existing process.
.TP
.B \fB\-\-join\-ct=N\fP
Number of \fBch\-run\fP peers (implies \fB\-\-join\fP; default: see
below).
.TP
.B \fB\-\-join\-tag=TAG\fP
Label for \fBch\-run\fP peer group (implies \fB\-\-join\fP; default: see
below).
.TP
.B \fB\-m\fP, \fB\-\-mount=DIR\fP
Use \fBDIR\fP for the SquashFS mount point, which must already exist. If
not specified, the default is \fB/var/tmp/$USER.ch/mnt\fP, which \fIwill\fP
be created if needed.
.TP
.B \fB\-\-no\-passwd\fP
By default, temporary \fB/etc/passwd\fP and \fB/etc/group\fP files are
created according to the UID and GID maps for the container and
bind\-mounted into it. If this is specified, no such temporary files are
created and the image’s files are exposed.
.TP
.B \fB\-q\fP, \fB\-\-quiet\fP
Be quieter; can be repeated. Incompatible with \fB\-v\fP\&. See the
\fI\%How can I control Charliecloud’s quietness or verbosity?\fP for details.
.TP
.B \fB\-s\fP, \fB\-\-storage DIR\fP
Set the storage directory. Equivalent to the same option for
\fBch\-image(1)\fP\&.
.TP
.B \fB\-\-seccomp\fP
Using seccomp, intercept some system calls that would fail due to lack of
privilege, do nothing, and return fake success to the calling program.
This is intended for use by \fBch\-image(1)\fP when building images; see
that man page for a detailed discussion.
.TP
.B \fB\-t\fP, \fB\-\-private\-tmp\fP
By default, the host’s \fB/tmp\fP (or \fB$TMPDIR\fP if set) is
bind\-mounted at container \fB/tmp\fP\&. If this is specified, a new
\fBtmpfs\fP is mounted on the container’s \fB/tmp\fP instead.
.TP
.B \fB\-\-set\-env\fP, \fB\-\-set\-env=FILE\fP, \fB\-\-set\-env=VAR=VALUE\fP
Set environment variables with newline\-separated file
(\fB/ch/environment\fP within the image if not specified) or on the
command line. See below for details.
.TP
.B \fB\-\-set\-env0\fP, \fB\-\-set\-env0=FILE\fP, \fB\-\-set\-env0=VAR=VALUE\fP
Like \fB\-\-set\-env\fP, but file is null\-byte separated.
.TP
.B \fB\-u\fP, \fB\-\-uid=UID\fP
Run as user \fBUID\fP within container.
.TP
.B \fB\-\-unsafe\fP
Enable various unsafe behavior. For internal use only. Seriously, stay
away from this option.
.TP
.B \fB\-\-unset\-env=GLOB\fP
Unset environment variables whose names match \fBGLOB\fP\&.
.TP
.B \fB\-v\fP, \fB\-\-verbose\fP
Print extra chatter; can be repeated. See the \fI\%FAQ entry on verbosity\fP for details.
.TP
.B \fB\-w\fP, \fB\-\-write\fP
Mount image read\-write. By default, the image is mounted read\-only. \fIThis
option should be avoided for most use cases,\fP because (1)\ changing images
live (as opposed to prescriptively with a Dockerfile) destroys their
provenance and (2)\ SquashFS images, which is the best\-practice format on
parallel filesystems, must be read\-only. It is better to use
\fB\-\-write\-fake\fP (for disposable data) or bind\-mount host directories
(for retained data).
.TP
.B \fB\-W\fP, \fB\-\-write\-fake[=SIZE]\fP
Overlay a writeable tmpfs on top of the image. This makes the image
\fIappear\fP read\-write, but it actually remains read\-only and unchanged. All
data “written” to the image are discarded when the container exits.
.sp
The size of the writeable filesystem \fBSIZE\fP is any size
specification acceptable to \fBtmpfs\fP, e.g. \fB4m\fP for 4MiB or
\fB50%\fP for half of physical memory. If this option is specified
without \fBSIZE\fP, the default is \fB12%\fP\&. Note (1)\ this limit is a
maximum — only actually stored files consume virtual memory — and
(2)\ \fBSIZE\fP larger than memory can be requested without error (the
failure happens later if the actual contents become too large).
.sp
This requires kernel support and there are some caveats. See section
“\fI\%Writeable overlay with \-\-write\-fake\fP” below for details.
.TP
.B \fB\-?\fP, \fB\-\-help\fP
Print help and exit.
.TP
.B \fB\-\-usage\fP
Print a short usage message and exit.
.TP
.B \fB\-V\fP, \fB\-\-version\fP
Print version and exit.
.UNINDENT
.UNINDENT
.UNINDENT
.sp
\fBNote:\fP Because \fBch\-run\fP is fully unprivileged, it is not possible to
change UIDs and GIDs within the container (the relevant system calls fail). In
particular, setuid, setgid, and setcap executables do not work. As a
precaution, \fBch\-run\fP calls \fBprctl(PR_SET_NO_NEW_PRIVS, 1)\fP to
\fI\%disable these executables\fP within the
container. This does not reduce functionality but is a “belt and suspenders”
precaution to reduce the attack surface should bugs in these system calls or
elsewhere arise.
.SH IMAGE FORMAT
.sp
\fBch\-run\fP supports two different image formats.
.sp
The first is a simple directory that contains a Linux filesystem tree. This
can be accomplished by:
.INDENT 0.0
.IP \(bu 2
\fBch\-convert\fP directly from \fBch\-image\fP or another builder to a
directory.
.IP \(bu 2
Charliecloud’s tarball workflow: build or pull the image, \fBch\-convert\fP
it to a tarball, transfer the tarball to the target system, then
\fBch\-convert\fP the tarball to a directory.
.IP \(bu 2
Manually mount a SquashFS image, e.g. with \fBsquashfuse(1)\fP and then
un\-mount it after run with \fBfusermount \-u\fP\&.
.IP \(bu 2
Any other workflow that produces an appropriate directory tree.
.UNINDENT
.sp
The second is a SquashFS image archive mounted internally by \fBch\-run\fP,
available if it’s linked with the optional \fBlibsquashfuse_ll\fP shared
library. \fBch\-run\fP mounts the image filesystem, services all FUSE
requests, and unmounts it, all within \fBch\-run\fP\&. See \fB\-\-mount\fP
above to set the mount point location.
.sp
Like other FUSE implementations, Charliecloud calls the \fBfusermount3(1)\fP
utility to mount the SquashFS filesystem. However, \fBthis executable does not
need to be installed setuid root\fP, and in fact \fBch\-run\fP actively
suppresses its setuid bit if set (using \fBprctl(2)\fP).
.sp
Prior versions of Charliecloud provided wrappers for the \fBsquashfuse\fP
and \fBsquashfuse_ll\fP SquashFS mount commands and \fBfusermount \-u\fP
unmount command. We removed these because we concluded they had minimal
value\-add over the standard, unwrapped commands.
.sp
\fBWARNING:\fP
.INDENT 0.0
.INDENT 3.5
Currently, Charliecloud unmounts the SquashFS filesystem when user command
\fBCOMMAND\fP’s process exits. It does not monitor any of its child
processes. Therefore, if the user command spawns child processes and then
exits before them (e.g., some daemons), those children will have the image
unmounted from underneath them. In this case, the workaround is to
mount/unmount using external tools. We expect to remove this limitation in a
future version.
.UNINDENT
.UNINDENT
.SH HOST FILES AND DIRECTORIES AVAILABLE IN CONTAINER VIA BIND MOUNTS
.sp
In addition to any directories specified by the user with \fB\-\-bind\fP,
\fBch\-run\fP has standard host files and directories that are bind\-mounted
in as well.
.sp
The following host files and directories are bind\-mounted at the same location
in the container. These give access to the host’s devices and various kernel
facilities. (Recall that Charliecloud provides minimal isolation and
containerized processes are mostly normal unprivileged processes.) They cannot
be disabled and are required; i.e., they must exist both on host and within
the image.
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.IP \(bu 2
\fB/dev\fP
.IP \(bu 2
\fB/proc\fP
.IP \(bu 2
\fB/sys\fP
.UNINDENT
.UNINDENT
.UNINDENT
.sp
Optional; bind\-mounted only if path exists on both host and within the image,
without error or warning if not.
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.IP \(bu 2
\fB/etc/hosts\fP and \fB/etc/resolv.conf\fP\&. Because Charliecloud
containers share the host network namespace, they need the same hostname
resolution configuration.
.IP \(bu 2
\fB/etc/machine\-id\fP\&. Provides a unique ID for the OS installation;
matching the host works for most situations. Needed to support D\-Bus, some
software licensing situations, and likely other use cases. See also \fI\%issue
#1050\fP\&.
.IP \(bu 2
\fB/var/lib/hugetlbfs\fP at guest \fB/var/opt/cray/hugetlbfs\fP, and
\fB/var/opt/cray/alps/spool\fP\&. These support Cray MPI.
.UNINDENT
.UNINDENT
.UNINDENT
.sp
Additional bind mounts done by default but can be disabled; see the options
above.
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.IP \(bu 2
\fB$HOME\fP at \fB/home/$USER\fP (and image \fB/home\fP is hidden).
Makes user data and init files available.
.IP \(bu 2
\fB/tmp\fP (or \fB$TMPDIR\fP if set) at guest \fB/tmp\fP\&. Provides a
temporary directory that persists between container runs and is shared
with non\-containerized application components.
.IP \(bu 2
temporary files at \fB/etc/passwd\fP and \fB/etc/group\fP\&. Usernames
and group names need to be customized for each container run.
.UNINDENT
.UNINDENT
.UNINDENT
.SH MULTIPLE PROCESSES IN THE SAME CONTAINER WITH --JOIN
.sp
By default, different \fBch\-run\fP invocations use different user and mount
namespaces (i.e., different containers). While this has no impact on sharing
most resources between invocations, there are a few important exceptions.
These include:
.INDENT 0.0
.IP 1. 3
\fBptrace(2)\fP, used by debuggers and related tools. One can attach a
debugger to processes in descendant namespaces, but not sibling namespaces.
The practical effect of this is that (without \fB\-\-join\fP), you can’t
run a command with \fBch\-run\fP and then attach to it with a debugger
also run with \fBch\-run\fP\&.
.IP 2. 3
\fICross\-memory attach\fP (CMA) is used by cooperating processes to communicate
by simply reading and writing one another’s memory. This is also not
permitted between sibling namespaces. This affects various MPI
implementations that use CMA to pass messages between ranks on the same
node, because it’s faster than traditional shared memory.
.UNINDENT
.sp
\fB\-\-join\fP is designed to address this by placing related \fBch\-run\fP
commands (the “peer group”) in the same container. This is done by one of the
peers creating the namespaces with \fBunshare(2)\fP and the others joining
with \fBsetns(2)\fP\&.
.sp
To do so, we need to know the number of peers and a name for the group. These
are specified by additional arguments that can (hopefully) be left at default
values in most cases:
.INDENT 0.0
.IP \(bu 2
\fB\-\-join\-ct\fP sets the number of peers. The default is the value of the
first of the following environment variables that is defined:
\fBOMPI_COMM_WORLD_LOCAL_SIZE\fP, \fBSLURM_STEP_TASKS_PER_NODE\fP,
\fBSLURM_CPUS_ON_NODE\fP\&.
.IP \(bu 2
\fB\-\-join\-tag\fP sets the tag that names the peer group. The default is
environment variable \fBSLURM_STEP_ID\fP, if defined; otherwise, the PID
of \fBch\-run\fP’s parent. Tags can be re\-used for peer groups that start
at different times, i.e., once all peer \fBch\-run\fP have replaced
themselves with the user command, the tag can be re\-used.
.UNINDENT
.sp
Caveats:
.INDENT 0.0
.IP \(bu 2
One cannot currently add peers after the fact, for example, if one decides
to start a debugger after the fact. (This is only required for code with
bugs and is thus an unusual use case.)
.IP \(bu 2
\fBch\-run\fP instances race. The winner of this race sets up the
namespaces, and the other peers use the winner to find the namespaces to
join. Therefore, if the user command of the winner exits, any remaining
peers will not be able to join the namespaces, even if they are still
active. There is currently no general way to specify which \fBch\-run\fP
should be the winner.
.IP \(bu 2
If \fB\-\-join\-ct\fP is too high, the winning \fBch\-run\fP’s user command
exits before all peers join, or \fBch\-run\fP itself crashes, IPC resources
such as semaphores and shared memory segments will be leaked. These appear
as files in \fB/dev/shm/\fP and can be removed with \fBrm(1)\fP\&.
.IP \(bu 2
Many of the arguments given to the race losers, such as the image path and
\fB\-\-bind\fP, will be ignored in favor of what was given to the winner.
.UNINDENT
.SH WRITEABLE OVERLAY WITH --WRITE-FAKE
.sp
If you need the image to stay read\-only but appear writeable, you may be able
to use \fB\-\-write\-fake\fP to overlay a writeable tmpfs atop the image. This
requires kernel support. Specifically:
.INDENT 0.0
.IP 1. 3
To use the feature at all, you need unprivileged overlayfs support. This is
available in \fI\%upstream 5.11\fP
(February 2021), but distributions vary considerably. If you don’t have
this, the container will fail to start with error “operation not
permitted”.
.IP 2. 3
For a fully functional overlay, you need a tmpfs that supports xattrs in
the \fBuser\fP namespace. This is available in \fI\%upstream 6.6\fP (October 2023). If you don’t
have this, most things will work fine, but some operations will fail with
“I/O error”, for example creating a directory with the same path as a
previously deleted directory. There will also be syslog noise about xattr
problems.
.sp
(overlayfs can also use xattrs in the \fBtrusted\fP namespace, but this
requires \fBCAP_SYS_ADMIN\fP \fI\%on the host\fP
and thus is not helpful for unprivileged containers.)
.UNINDENT
.SH ENVIRONMENT VARIABLES
.sp
\fBch\-run\fP leaves environment variables unchanged, i.e. the host
environment is passed through unaltered, except:
.INDENT 0.0
.IP \(bu 2
by default (\fB\-\-home\fP not specified), \fBHOME\fP is set to
\fB/root\fP, if it exists, and \fB/\fP otherwise.
.IP \(bu 2
limited tweaks to avoid significant guest breakage;
.IP \(bu 2
user\-set variables via \fB\-\-set\-env\fP;
.IP \(bu 2
user\-unset variables via \fB\-\-unset\-env\fP; and
.IP \(bu 2
set \fBCH_RUNNING\fP\&.
.UNINDENT
.sp
This section describes these features.
.sp
The default tweaks happen first, then \fB\-\-set\-env\fP and
\fB\-\-unset\-env\fP in the order specified on the command line, and then
\fBCH_RUNNING\fP\&. The two options can be repeated arbitrarily many times,
e.g. to add/remove multiple variable sets or add only some variables in a
file.
.SS Default behavior
.sp
By default, \fBch\-run\fP makes the following environment variable changes:
.INDENT 0.0
.TP
.B \fB$CH_RUNNING\fP
Set to \fBWeird Al Yankovic\fP\&. While a process can figure out that it’s
in an unprivileged container and what namespaces are active without this
hint, that can be messy, and there is no way to tell that it’s a
\fICharliecloud\fP container specifically. This variable makes such a test
simple and well\-defined. (\fBNote:\fP This variable is unaffected by
\fB\-\-unset\-env\fP\&.)
.TP
.B \fB$HOME\fP
If \fB\-\-home\fP is specified, then your home directory is bind\-mounted
into the guest at \fB/home/$USER\fP\&. If you also have a different home
directory path on the host, an inherited \fB$HOME\fP will be incorrect
inside the guest, which confuses lots of software, notably Spack. Thus, with
\fB\-\-home\fP, \fB$HOME\fP is set to \fB/home/$USER\fP (by default, it
is unchanged.)
.TP
.B \fB$PATH\fP
Newer Linux distributions replace some root\-level directories, such as
\fB/bin\fP, with symlinks to their counterparts in \fB/usr\fP\&.
.sp
Some of these distributions (e.g., Fedora 24) have also dropped \fB/bin\fP
from the default \fB$PATH\fP\&. This is a problem when the guest OS does
\fInot\fP have a merged \fB/usr\fP (e.g., Debian 8 “Jessie”). Thus, we add
\fB/bin\fP to \fB$PATH\fP if it’s not already present.
.sp
Further reading:
.INDENT 7.0
.INDENT 3.5
.INDENT 0.0
.IP \(bu 2
\fI\%The case for the /usr Merge\fP
.IP \(bu 2
\fI\%Fedora\fP
.IP \(bu 2
\fI\%Debian\fP
.UNINDENT
.UNINDENT
.UNINDENT
.TP
.B \fB$TMPDIR\fP
Unset, because this is almost certainly a host path, and that host path is
made available in the guest at \fB/tmp\fP unless \fB\-\-private\-tmp\fP is
given.
.UNINDENT
.SS Setting variables with \fB\-\-set\-env\fP or \fB\-\-set\-env0\fP
.sp
The purpose of these two options is to set environment variables within the
container. Values given replace any already in the environment (i.e.,
inherited from the host shell) or set by earlier uses of the options. These
flags take an optional argument with two possible forms:
.INDENT 0.0
.IP 1. 3
\fBIf the argument contains an equals sign\fP (\fB=\fP, ASCII 61), that
sets an environment variable directly. For example, to set \fBFOO\fP to
the string value \fBbar\fP:
.INDENT 3.0
.INDENT 3.5
.sp
.EX
$ ch\-run \-\-set\-env=FOO=bar ...
.EE
.UNINDENT
.UNINDENT
.sp
Single straight quotes around the value (\fB\(aq\fP, ASCII 39) are stripped,
though be aware that both single and double quotes are also interpreted by
the shell. For example, this example is similar to the prior one; the
double quotes are removed by the shell and the single quotes are removed by
\fBch\-run\fP:
.INDENT 3.0
.INDENT 3.5
.sp
.EX
$ ch\-run \-\-set\-env=\(dq\(aqBAZ=qux\(aq\(dq ...
.EE
.UNINDENT
.UNINDENT
.IP 2. 3
\fBIf the argument does not contain an equals sign\fP, it is a host path to a
file containing zero or more variables using the same syntax as above
(except with no prior shell processing).
.sp
With \fB\-\-set\-env\fP, this file contains a sequence of assignments
separated by newline (\fBn\fP or ASCII 10); with \fB\-\-set\-env0\fP, the
assignments are separated by the null byte (i.e., \fB0\fP or ASCII 0).
Empty assignments are ignored, and no comments are interpreted. (This
syntax is designed to accept the output of \fBprintenv\fP and be easily
produced by other simple mechanisms.) The file need not be seekable.
.sp
For example:
.INDENT 3.0
.INDENT 3.5
.sp
.EX
$ cat /tmp/env.txt
FOO=bar
BAZ=\(aqqux\(aq
$ ch\-run \-\-set\-env=/tmp/env.txt ...
.EE
.UNINDENT
.UNINDENT
.sp
For directory images only (because the file is read before containerizing),
guest paths can be given by prepending the image path.
.IP 3. 3
\fBIf there is no argument\fP, the file \fB/ch/environment\fP within the
image is used. This file is commonly populated by \fBENV\fP instructions
in the Dockerfile. For example, equivalently to form\ 2:
.INDENT 3.0
.INDENT 3.5
.sp
.EX
$ cat Dockerfile
[...]
ENV FOO=bar
ENV BAZ=qux
[...]
$ ch\-image build \-t foo .
$ ch\-convert foo /var/tmp/foo.sqfs
$ ch\-run \-\-set\-env /var/tmp/foo.sqfs \-\- ...
.EE
.UNINDENT
.UNINDENT
.sp
(Note the image path is interpreted correctly, not as the \fB\-\-set\-env\fP
argument.)
.sp
At present, there is no way to use files other than \fB/ch/environment\fP
within SquashFS images.
.UNINDENT
.sp
Environment variables are expanded for values that look like search paths,
unless \fB\-\-env\-no\-expand\fP is given prior to \fB\-\-set\-env\fP\&. In this
case, the value is a sequence of zero or more possibly\-empty items separated
by colon (\fB:\fP, ASCII 58). If an item begins with dollar sign (\fB$\fP,
ASCII 36), then the rest of the item is the name of an environment variable.
If this variable is set to a non\-empty value, that value is substituted for
the item; otherwise (i.e., the variable is unset or the empty string), the
item is deleted, including a delimiter colon. The purpose of omitting empty
expansions is to avoid surprising behavior such as an empty element in
\fB$PATH\fP meaning \fI\%the current directory\fP\&.
.sp
For example, to set \fBHOSTPATH\fP to the search path in the current shell
(this is expanded by \fBch\-run\fP, though letting the shell do it happens to
be equivalent):
.INDENT 0.0
.INDENT 3.5
.sp
.EX
$ ch\-run \-\-set\-env=\(aqHOSTPATH=$PATH\(aq ...
.EE
.UNINDENT
.UNINDENT
.sp
To prepend \fB/opt/bin\fP to this current search path:
.INDENT 0.0
.INDENT 3.5
.sp
.EX
$ ch\-run \-\-set\-env=\(aqPATH=/opt/bin:$PATH\(aq ...
.EE
.UNINDENT
.UNINDENT
.sp
To prepend \fB/opt/bin\fP to the search path set by the Dockerfile, as
retrieved from guest file \fB/ch/environment\fP (here we really cannot let
the shell expand \fB$PATH\fP):
.INDENT 0.0
.INDENT 3.5
.sp
.EX
$ ch\-run \-\-set\-env \-\-set\-env=\(aqPATH=/opt/bin:$PATH\(aq ...
.EE
.UNINDENT
.UNINDENT
.sp
Examples of valid assignment, assuming that environment variable \fBBAR\fP
is set to \fBbar\fP and \fBUNSET\fP is unset or set to the empty string:
.TS
center;
|l|l|l|.
_
T{
Assignment
T}	T{
Name
T}	T{
Value
T}
_
T{
\fBFOO=bar\fP
T}	T{
\fBFOO\fP
T}	T{
\fBbar\fP
T}
_
T{
\fBFOO=bar=baz\fP
T}	T{
\fBFOO\fP
T}	T{
\fBbar=baz\fP
T}
_
T{
\fBFLAGS=\-march=foo \-mtune=bar\fP
T}	T{
\fBFLAGS\fP
T}	T{
\fB\-march=foo \-mtune=bar\fP
T}
_
T{
\fBFLAGS=\(aq\-march=foo \-mtune=bar\(aq\fP
T}	T{
\fBFLAGS\fP
T}	T{
\fB\-march=foo \-mtune=bar\fP
T}
_
T{
\fBFOO=$BAR\fP
T}	T{
\fBFOO\fP
T}	T{
\fBbar\fP
T}
_
T{
\fBFOO=$BAR:baz\fP
T}	T{
\fBFOO\fP
T}	T{
\fBbar:baz\fP
T}
_
T{
\fBFOO=\fP
T}	T{
\fBFOO\fP
T}	T{
empty string
T}
_
T{
\fBFOO=$UNSET\fP
T}	T{
\fBFOO\fP
T}	T{
empty string
T}
_
T{
\fBFOO=baz:$UNSET:qux\fP
T}	T{
\fBFOO\fP
T}	T{
\fBbaz:qux\fP (not \fBbaz::qux\fP)
T}
_
T{
\fBFOO=:bar:baz::\fP
T}	T{
\fBFOO\fP
T}	T{
\fB:bar:baz::\fP
T}
_
T{
\fBFOO=\(aq\(aq\fP
T}	T{
\fBFOO\fP
T}	T{
empty string
T}
_
T{
\fBFOO=\(aq\(aq\(aq\(aq\fP
T}	T{
\fBFOO\fP
T}	T{
\fB\(aq\(aq\fP (two single quotes)
T}
_
.TE
.sp
Example invalid assignments:
.TS
center;
|l|l|.
_
T{
Assignment
T}	T{
Problem
T}
_
T{
\fBFOO bar\fP
T}	T{
no equals separator
T}
_
T{
\fB=bar\fP
T}	T{
name cannot be empty
T}
_
.TE
.sp
Example valid assignments that are probably not what you want:
.TS
center;
|l|l|l|l|.
_
T{
Assignment
T}	T{
Name
T}	T{
Value
T}	T{
Problem
T}
_
T{
\fBFOO=\(dqbar\(dq\fP
T}	T{
\fBFOO\fP
T}	T{
\fB\(dqbar\(dq\fP
T}	T{
double quotes aren’t stripped
T}
_
T{
\fBFOO=bar # baz\fP
T}	T{
\fBFOO\fP
T}	T{
\fBbar # baz\fP
T}	T{
comments not supported
T}
_
T{
\fBFOO=bartbaz\fP
T}	T{
\fBFOO\fP
T}	T{
\fBbartbaz\fP
T}	T{
backslashes are not special
T}
_
T{
\fB​ FOO=bar\fP
T}	T{
\fB​ FOO\fP
T}	T{
\fBbar\fP
T}	T{
leading space in key
T}
_
T{
\fBFOO= bar\fP
T}	T{
\fBFOO\fP
T}	T{
\fB​ bar\fP
T}	T{
leading space in value
T}
_
T{
\fB$FOO=bar\fP
T}	T{
\fB$FOO\fP
T}	T{
\fBbar\fP
T}	T{
variables not expanded in key
T}
_
T{
\fBFOO=$BAR baz:qux\fP
T}	T{
\fBFOO\fP
T}	T{
\fBqux\fP
T}	T{
variable \fBBAR baz\fP not set
T}
_
.TE
.SS Removing variables with \fB\-\-unset\-env\fP
.sp
The purpose of \fB\-\-unset\-env=GLOB\fP is to remove unwanted environment
variables. The argument \fBGLOB\fP is a glob pattern (\fI\%dialect\fP \fBfnmatch(3)\fP
with the \fBFNM_EXTMATCH\fP flag where supported); all variables with
matching names are removed from the environment.
.sp
\fBWARNING:\fP
.INDENT 0.0
.INDENT 3.5
Because the shell also interprets glob patterns, if any wildcard characters
are in \fBGLOB\fP, it is important to put it in single quotes to avoid
surprises.
.UNINDENT
.UNINDENT
.sp
\fBGLOB\fP must be a non\-empty string.
.sp
Example 1: Remove the single environment variable \fBFOO\fP:
.INDENT 0.0
.INDENT 3.5
.sp
.EX
$ export FOO=bar
$ env | fgrep FOO
FOO=bar
$ ch\-run \-\-unset\-env=FOO $CH_TEST_IMGDIR/chtest \-\- env | fgrep FOO
$
.EE
.UNINDENT
.UNINDENT
.sp
Example 2: Hide from a container the fact that it’s running in a Slurm
allocation, by removing all variables beginning with \fBSLURM\fP\&. You might
want to do this to test an MPI program with one rank and no launcher:
.INDENT 0.0
.INDENT 3.5
.sp
.EX
$ salloc \-N1
$ env | egrep \(aq^SLURM\(aq | wc
   44      44    1092
$ ch\-run $CH_TEST_IMGDIR/mpihello\-openmpi \-\- /hello/hello
[... long error message ...]
$ ch\-run \-\-unset\-env=\(aqSLURM*\(aq $CH_TEST_IMGDIR/mpihello\-openmpi \-\- /hello/hello
0: MPI version:
Open MPI v3.1.3, package: Open MPI root@c897a83f6f92 Distribution, ident: 3.1.3, repo rev: v3.1.3, Oct 29, 2018
0: init ok cn001.localdomain, 1 ranks, userns 4026532530
0: send/receive ok
0: finalize ok
.EE
.UNINDENT
.UNINDENT
.sp
Example 3: Clear the environment completely (remove all variables):
.INDENT 0.0
.INDENT 3.5
.sp
.EX
$ ch\-run \-\-unset\-env=\(aq*\(aq $CH_TEST_IMGDIR/chtest \-\- env
$
.EE
.UNINDENT
.UNINDENT
.sp
Example 4: Remove all environment variables \fIexcept\fP for those prefixed with
either \fBWANTED_\fP or \fBALSO_WANTED_\fP:
.INDENT 0.0
.INDENT 3.5
.sp
.EX
$ export WANTED_1=yes
$ export ALSO_WANTED_2=yes
$ export NOT_WANTED_1=no
$ ch\-run \-\-unset\-env=\(aq!(WANTED_*|ALSO_WANTED_*)\(aq $CH_TEST_IMGDIR/chtest \-\- env
WANTED_1=yes
ALSO_WANTED_2=yes
$
.EE
.UNINDENT
.UNINDENT
.sp
Note that some programs, such as shells, set some environment variables even
if started with no init files:
.INDENT 0.0
.INDENT 3.5
.sp
.EX
$ ch\-run \-\-unset\-env=\(aq*\(aq $CH_TEST_IMGDIR/debian_9ch \-\- bash \-\-noprofile \-\-norc \-c env
SHLVL=1
PWD=/
_=/usr/bin/env
$
.EE
.UNINDENT
.UNINDENT
.SH EXAMPLES
.sp
Run the command \fBecho hello\fP inside a Charliecloud container using the
unpacked image at \fB/data/foo\fP:
.INDENT 0.0
.INDENT 3.5
.sp
.EX
$ ch\-run /data/foo \-\- echo hello
hello
.EE
.UNINDENT
.UNINDENT
.sp
Run an MPI job that can use CMA to communicate:
.INDENT 0.0
.INDENT 3.5
.sp
.EX
$ srun ch\-run \-\-join /data/foo \-\- bar
.EE
.UNINDENT
.UNINDENT
.SH SYSLOG
.sp
By default, \fBch\-run\fP logs its command line to \fI\%syslog\fP\&. (This can be disabled by configuring
with \fB\-\-disable\-syslog\fP\&.) This includes: (1) the invoking real UID, (2)
the number of command line arguments, and (3) the arguments, separated by
spaces. For example:
.INDENT 0.0
.INDENT 3.5
.sp
.EX
Dec 10 18:19:08 mybox ch\-run: uid=1000 args=7: ch\-run \-v /var/tmp/00_tiny \-\- echo hello \(dqwor l}\e$d\(dq
.EE
.UNINDENT
.UNINDENT
.sp
Logging is one of the first things done during program initialization, even
before command line parsing. That is, almost all command lines are logged,
even if erroneous, and there is no logging of program success or failure.
.sp
Arguments are serialized with the following procedure. The purpose is to
provide a human\-readable reconstruction of the command line while also
allowing each argument to be recovered byte\-for\-byte.
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.IP \(bu 2
If an argument contains only printable ASCII bytes that are not
whitespace, shell metacharacters, double quote (\fB\(dq\fP, ASCII 34
decimal), or backslash (\fB​\fP, ASCII 92), then log it unchanged.
.IP \(bu 2
Otherwise, (a)\ enclose the argument in double quotes and
(b)\ backslash\-escape double quotes, backslashes, and characters
interpreted by Bash (including POSIX shells) within double quotes.
.UNINDENT
.UNINDENT
.UNINDENT
.sp
The verbatim command line typed in the shell cannot be recovered, because not
enough information is provided to UNIX programs. For example,
\fBecho\ \ \(aqfoo\(aq\fP is given to programs as a sequence of two arguments,
\fBecho\fP and \fBfoo\fP; the two spaces and single quotes are removed by
the shell. The zero byte, ASCII NUL, cannot appear in arguments because it
would terminate the string.
.SH EXIT STATUS
.sp
If there is an error during containerization, \fBch\-run\fP exits with status
non\-zero. If the user command is started successfully, the exit status is that
of the user command, with one exception: if the image is an internally mounted
SquashFS filesystem and the user command is killed by a signal, the exit
status is 1 regardless of the signal value.
.SH REPORTING BUGS
.sp
If Charliecloud was obtained from your Linux distribution, use your
distribution’s bug reporting procedures.
.sp
Otherwise, report bugs to: \fI\%https://github.com/hpc/charliecloud/issues\fP
.SH SEE ALSO
.sp
charliecloud(7)
.sp
Full documentation at: <\fI\%https://hpc.github.io/charliecloud\fP>
.SH COPYRIGHT
2014–2023, Triad National Security, LLC and others
.\" Generated by docutils manpage writer.
.