.\" Man page generated from reStructuredText.
.
.TH "CH-RUN" "1" "2019-09-08 18:23 Coordinated Universal Time" "" "Charliecloud"
.SH NAME
ch-run \- Run a command in a Charliecloud container
.
.nr rst2man-indent-level 0
.
.de1 rstReportMargin
\\$1 \\n[an-margin]
level \\n[rst2man-indent-level]
level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
-
\\n[rst2man-indent0]
\\n[rst2man-indent1]
\\n[rst2man-indent2]
..
.de1 INDENT
.\" .rstReportMargin pre:
. RS \\$1
. nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin]
. nr rst2man-indent-level +1
.\" .rstReportMargin post:
..
.de UNINDENT
. RE
.\" indent \\n[an-margin]
.\" old: \\n[rst2man-indent\\n[rst2man-indent-level]]
.nr rst2man-indent-level -1
.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
..
.SH SYNOPSIS
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
$ ch\-run [OPTION...] NEWROOT CMD [ARG...]
.ft P
.fi
.UNINDENT
.UNINDENT
.SH DESCRIPTION
.sp
Run command \fBCMD\fP in a Charliecloud container using the flattened and
unpacked image directory located at \fBNEWROOT\fP\&.
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.TP
.B \fB\-b\fP, \fB\-\-bind=SRC[:DST]\fP
mount \fBSRC\fP at guest \fBDST\fP (default \fB/mnt/0\fP,
\fB/mnt/1\fP, etc.)
.TP
.B \fB\-c\fP, \fB\-\-cd=DIR\fP
initial working directory in container
.TP
.B \fB\-\-ch\-ssh\fP
bind \fBch\-ssh(1)\fP into container at \fB/usr/bin/ch\-ssh\fP
.TP
.B \fB\-g\fP, \fB\-\-gid=GID\fP
run as group \fBGID\fP within container
.TP
.B \fB\-j\fP, \fB\-\-join\fP
use the same container (namespaces) as peer \fBch\-run\fP invocations
.TP
.B \fB\-\-join\-pid=PID\fP
join the namespaces of an existing process
.TP
.B \fB\-\-join\-ct=N\fP
number of \fBch\-run\fP peers (implies \fB\-\-join\fP; default: see below)
.TP
.B \fB\-\-join\-tag=TAG\fP
label for \fBch\-run\fP peer group (implies \fB\-\-join\fP; default: see
below)
.TP
.B \fB\-\-no\-home\fP
do not bind\-mount your home directory (by default, your home directory is
mounted at \fB/home/$USER\fP in the container)
.TP
.B \fB\-t\fP, \fB\-\-private\-tmp\fP
use container\-private \fB/tmp\fP (by default, \fB/tmp\fP is shared with
the host)
.TP
.B \fB\-\-set\-env=FILE\fP
set environment variables as specified in host path \fBFILE\fP
.TP
.B \fB\-u\fP, \fB\-\-uid=UID\fP
run as user \fBUID\fP within container
.TP
.B \fB\-\-unset\-env=GLOB\fP
unset environment variables whose names match \fBGLOB\fP
.TP
.B \fB\-v\fP, \fB\-\-verbose\fP
be more verbose (debug if repeated)
.TP
.B \fB\-w\fP, \fB\-\-write\fP
mount image read\-write (by default, the image is mounted read\-only)
.TP
.B \fB\-?\fP, \fB\-\-help\fP
print help and exit
.TP
.B \fB\-\-usage\fP
print a short usage message and exit
.TP
.B \fB\-V\fP, \fB\-\-version\fP
print version and exit
.UNINDENT
.UNINDENT
.UNINDENT
.SH HOST FILES AND DIRECTORIES AVAILABLE IN CONTAINER VIA BIND MOUNTS
.sp
In addition to any directories specified by the user with \fB\-\-bind\fP,
\fBch\-run\fP has standard host files and directories that are bind\-mounted
in as well.
.sp
The following host files and directories are bind\-mounted at the same location
in the container. These cannot be disabled.
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.IP \(bu 2
\fB/dev\fP
.IP \(bu 2
\fB/etc/passwd\fP
.IP \(bu 2
\fB/etc/group\fP
.IP \(bu 2
\fB/etc/hosts\fP
.IP \(bu 2
\fB/etc/resolv.conf\fP
.IP \(bu 2
\fB/proc\fP
.IP \(bu 2
\fB/sys\fP
.UNINDENT
.UNINDENT
.UNINDENT
.sp
Three additional bind mounts can be disabled by the user:
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.IP \(bu 2
Your home directory (i.e., \fB$HOME\fP) is mounted at guest
\fB/home/$USER\fP by default. This is accomplished by mounting a new
\fBtmpfs\fP at \fB/home\fP, which hides any image content under that
path. If \fB\-\-no\-home\fP is specified, neither of these things happens
and the image’s \fB/home\fP is exposed unaltered.
.IP \(bu 2
\fB/tmp\fP is shared with the host by default. If \fB\-\-private\-tmp\fP
is specified, a new \fBtmpfs\fP is mounted on the guest’s \fB/tmp\fP
instead.
.IP \(bu 2
If file \fB/usr/bin/ch\-ssh\fP is present in the image, it is
over\-mounted with the \fBch\-ssh\fP binary in the same directory as
\fBch\-run\fP\&.
.UNINDENT
.UNINDENT
.UNINDENT
.SH MULTIPLE PROCESSES IN THE SAME CONTAINER WITH --JOIN
.sp
By default, different \fBch\-run\fP invocations use different user and mount
namespaces (i.e., different containers). While this has no impact on sharing
most resources between invocations, there are a few important exceptions.
These include:
.INDENT 0.0
.IP 1. 3
\fBptrace(2)\fP, used by debuggers and related tools. One can attach a
debugger to processes in descendant namespaces, but not sibling namespaces.
The practical effect of this is that (without \fB\-\-join\fP), you can’t
run a command with \fBch\-run\fP and then attach to it with a debugger
also run with \fBch\-run\fP\&.
.IP 2. 3
\fICross\-memory attach\fP (CMA) is used by cooperating processes to communicate
by simply reading and writing one another’s memory. This is also not
permitted between sibling namespaces. This affects various MPI
implementations that use CMA to pass messages between ranks on the same
node, because it’s faster than traditional shared memory.
.UNINDENT
.sp
\fB\-\-join\fP is designed to address this by placing related \fBch\-run\fP
commands (the “peer group”) in the same container. This is done by one of the
peers creating the namespaces with \fBunshare(2)\fP and the others joining
with \fBsetns(2)\fP\&.
.sp
To do so, we need to know the number of peers and a name for the group. These
are specified by additional arguments that can (hopefully) be left at default
values in most cases:
.INDENT 0.0
.IP \(bu 2
\fB\-\-join\-ct\fP sets the number of peers. The default is the value of the
first of the following environment variables that is defined:
\fBOMPI_COMM_WORLD_LOCAL_SIZE\fP, \fBSLURM_STEP_TASKS_PER_NODE\fP,
\fBSLURM_CPUS_ON_NODE\fP\&.
.IP \(bu 2
\fB\-\-join\-tag\fP sets the tag that names the peer group. The default is
environment variable \fBSLURM_STEP_ID\fP, if defined; otherwise, the PID
of \fBch\-run\fP’s parent. Tags can be re\-used for peer groups that start
at different times, i.e., once all peer \fBch\-run\fP have replaced
themselves with the user command, the tag can be re\-used.
.UNINDENT
.sp
Caveats:
.INDENT 0.0
.IP \(bu 2
One cannot currently add peers after the fact, for example, if one decides
to start a debugger after the fact. (This is only required for code with
bugs and is thus an unusual use case.)
.IP \(bu 2
\fBch\-run\fP instances race. The winner of this race sets up the
namespaces, and the other peers use the winner to find the namespaces to
join. Therefore, if the user command of the winner exits, any remaining
peers will not be able to join the namespaces, even if they are still
active. There is currently no general way to specify which \fBch\-run\fP
should be the winner.
.IP \(bu 2
If \fB\-\-join\-ct\fP is too high, the winning \fBch\-run\fP’s user command
exits before all peers join, or \fBch\-run\fP itself crashes, IPC resources
such as semaphores and shared memory segments will be leaked. These appear
as files in \fB/dev/shm/\fP and can be removed with \fBrm(1)\fP\&.
.IP \(bu 2
Many of the arguments given to the race losers, such as the image path and
\fB\-\-bind\fP, will be ignored in favor of what was given to the winner.
.UNINDENT
.SH ENVIRONMENT VARIABLES
.sp
\fBch\-run\fP leaves environment variables unchanged, i.e. the host
environment is passed through unaltered, except:
.INDENT 0.0
.IP \(bu 2
limited tweaks to avoid significant guest breakage;
.IP \(bu 2
user\-set variables via \fB\-\-set\-env\fP; and
.IP \(bu 2
user\-unset variables via \fB\-\-unset\-env\fP\&.
.UNINDENT
.sp
This section describes these features.
.sp
The default tweaks happen first, and then \fB\-\-set\-env\fP and
\fB\-\-unset\-env\fP in the order specified on the command line. The latter two
can be repeated arbitrarily many times, e.g. to add/remove multiple variable
sets or add only some variables in a file.
.SS Default behavior
.sp
By default, \fBch\-run\fP makes the following environment variable changes:
.INDENT 0.0
.IP \(bu 2
\fB$HOME\fP: If the path to your home directory is not \fB/home/$USER\fP
on the host, then an inherited \fB$HOME\fP will be incorrect inside the
guest. This confuses some software, such as Spack.
.sp
Thus, we change \fB$HOME\fP to \fB/home/$USER\fP, unless
\fB\-\-no\-home\fP is specified, in which case it is left unchanged.
.IP \(bu 2
\fB$PATH\fP: Newer Linux distributions replace some root\-level
directories, such as \fB/bin\fP, with symlinks to their counterparts in
\fB/usr\fP\&.
.sp
Some of these distributions (e.g., Fedora 24) have also dropped \fB/bin\fP
from the default \fB$PATH\fP\&. This is a problem when the guest OS does
\fInot\fP have a merged \fB/usr\fP (e.g., Debian 8 “Jessie”). Thus, we add
\fB/bin\fP to \fB$PATH\fP if it’s not already present.
.sp
Further reading:
.INDENT 2.0
.INDENT 3.5
.INDENT 0.0
.IP \(bu 2
\fI\%The case for the /usr Merge\fP
.IP \(bu 2
\fI\%Fedora\fP
.IP \(bu 2
\fI\%Debian\fP
.UNINDENT
.UNINDENT
.UNINDENT
.UNINDENT
.SS Setting variables with \fB\-\-set\-env\fP
.sp
The purpose of \fB\-\-set\-env=FILE\fP is to set environment variables that
cannot be inherited from the host shell, e.g. Dockerfile \fBENV\fP
directives or other build\-time configuration. \fBFILE\fP is a host path to
provide the greatest flexibility; guest paths can be specified by prepending
the image path.
.sp
\fBch\-docker2tar(1)\fP lists variables specified at build time in
Dockerfiles in the image in file \fB/environment\fP\&. To set these variables:
\fB\-\-set\-env=$IMG/environment\fP\&.
.sp
Variable values in \fBFILE\fP replace any already set. If a variable is
repeated, the last value wins.
.sp
The syntax of \fBFILE\fP is key\-value pairs separated by the first equals
character (\fB=\fP, ASCII 61), one per line, with optional single straight
quotes (\fB\(aq\fP, ASCII 39) around the value. Empty lines are ignored.
Newlines (ASCII 10) are not permitted in either key or value. No variable
expansion, comments, etc. are provided. The value may be empty, but not the
key. (This syntax is designed to accept the output of \fBprintenv\fP and be
easily produced by other simple mechanisms.) Examples of valid lines:
.TS
center;
|l|l|l|.
_
T{
Line
T}	T{
Key
T}	T{
Value
T}
_
T{
\fBFOO=bar\fP
T}	T{
\fBFOO\fP
T}	T{
\fBbar\fP
T}
_
T{
\fBFOO=bar=baz\fP
T}	T{
\fBFOO\fP
T}	T{
\fBbar=baz\fP
T}
_
T{
\fBFLAGS=\-march=foo \-mtune=bar\fP
T}	T{
\fBFLAGS\fP
T}	T{
\fB\-march=foo \-mtune=bar\fP
T}
_
T{
\fBFLAGS=\(aq\-march=foo \-mtune=bar\(aq\fP
T}	T{
\fBFLAGS\fP
T}	T{
\fB\-march=foo \-mtune=bar\fP
T}
_
T{
\fBFOO=\fP
T}	T{
\fBFOO\fP
T}	T{
(empty string)
T}
_
T{
\fBFOO=\(aq\(aq\fP
T}	T{
\fBFOO\fP
T}	T{
(empty string)
T}
_
T{
\fBFOO=\(aq\(aq\(aq\(aq\fP
T}	T{
\fBFOO\fP
T}	T{
\fB\(aq\(aq\fP (two single quotes)
T}
_
.TE
.sp
Example invalid lines:
.TS
center;
|l|l|.
_
T{
Line
T}	T{
Problem
T}
_
T{
\fBFOO bar\fP
T}	T{
no separator
T}
_
T{
\fB=bar\fP
T}	T{
key cannot be empty
T}
_
.TE
.sp
Example valid lines that are probably not what you want:
.TS
center;
|l|l|l|l|.
_
T{
Line
T}	T{
Key
T}	T{
Value
T}	T{
Problem
T}
_
T{
\fBFOO="bar"\fP
T}	T{
\fBFOO\fP
T}	T{
\fB"bar"\fP
T}	T{
double quotes aren’t stripped
T}
_
T{
\fBFOO=bar # baz\fP
T}	T{
\fBFOO\fP
T}	T{
\fBbar # baz\fP
T}	T{
comments not supported
T}
_
T{
\fBPATH=$PATH:/opt/bin\fP
T}	T{
\fBPATH\fP
T}	T{
\fB$PATH:/opt/bin\fP
T}	T{
variables not expanded
T}
_
T{
\fB​ FOO=bar\fP
T}	T{
\fB​ FOO\fP
T}	T{
\fBbar\fP
T}	T{
leading space in key
T}
_
T{
\fBFOO= bar\fP
T}	T{
\fBFOO\fP
T}	T{
\fB​ bar\fP
T}	T{
leading space in value
T}
_
.TE
.SS Removing variables with \fB\-\-unset\-env\fP
.sp
The purpose of \fB\-\-unset\-env=GLOB\fP is to remove unwanted environment
variables. The argument \fBGLOB\fP is a glob pattern (\fI\%dialect\fP \fBfnmatch(3)\fP
with no flags); all variables with matching names are removed from the
environment.
.sp
\fBWARNING:\fP
.INDENT 0.0
.INDENT 3.5
Because the shell also interprets glob patterns, if any wildcard characters
are in \fBGLOB\fP, it is important to put it in single quotes to avoid
surprises.
.UNINDENT
.UNINDENT
.sp
\fBGLOB\fP must be a non\-empty string.
.sp
Example 1: Remove the single environment variable \fBFOO\fP:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
$ export FOO=bar
$ env | fgrep FOO
FOO=bar
$ ch\-run \-\-unset\-env=FOO $CH_TEST_IMGDIR/chtest \-\- env | fgrep FOO
$
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Example 2: Hide from a container the fact that it’s running in a Slurm
allocation, by removing all variables beginning with \fBSLURM\fP\&. You might
want to do this to test an MPI program with one rank and no launcher:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
$ salloc \-N1
$ env | egrep \(aq^SLURM\(aq | wc
   44      44    1092
$ ch\-run $CH_TEST_IMGDIR/mpihello\-openmpi \-\- /hello/hello
[... long error message ...]
$ ch\-run \-\-unset\-env=\(aqSLURM*\(aq $CH_TEST_IMGDIR/mpihello\-openmpi \-\- /hello/hello
0: MPI version:
Open MPI v3.1.3, package: Open MPI root@c897a83f6f92 Distribution, ident: 3.1.3, repo rev: v3.1.3, Oct 29, 2018
0: init ok cn001.localdomain, 1 ranks, userns 4026532530
0: send/receive ok
0: finalize ok
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Example 3: Clear the environment completely (remove all variables):
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
$ ch\-run \-\-unset\-env=\(aq*\(aq $CH_TEST_IMGDIR/chtest \-\- env
$
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Note that some programs, such as shells, set some environment variables even
if started with no init files:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
$ ch\-run \-\-unset\-env=\(aq*\(aq $CH_TEST_IMGDIR/debian9 \-\- bash \-\-noprofile \-\-norc \-c env
SHLVL=1
PWD=/
_=/usr/bin/env
$
.ft P
.fi
.UNINDENT
.UNINDENT
.SH EXAMPLES
.sp
Run the command \fBecho hello\fP inside a Charliecloud container using the
unpacked image at \fB/data/foo\fP:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
$ ch\-run /data/foo \-\- echo hello
hello
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Run an MPI job that can use CMA to communicate:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
$ srun ch\-run \-\-join /data/foo \-\- bar
.ft P
.fi
.UNINDENT
.UNINDENT
.SH REPORTING BUGS
.sp
If Charliecloud was obtained from your Linux distribution, use your
distribution’s bug reporting procedures.
.sp
Otherwise, report bugs to: <\fI\%https://github.com/hpc/charliecloud/issues\fP>
.SH SEE ALSO
.sp
charliecloud(1)
.sp
Full documentation at: <\fI\%https://hpc.github.io/charliecloud\fP>
.SH AUTHOR
Reid Priedhorsky, Tim Randles, and others
.SH COPYRIGHT
2014–2018, Los Alamos National Security, LLC
.\" Generated by docutils manpage writer.
.