NAME¶
Ladder - creates migration repositories for software release sets
Description¶
Ladder creates a SecureApt repository to migrate production devices from one
release milestone to the next. The repository contains all binary packages
which would be installed to upgrade the target package of the specified
release, including base packages. Source packages are not included as this
would make the final tarball much larger than necessary. Sources should remain
available via the main repositories.
For the purposes of "ladder", the bare installation / rootfs should be
considered to always precede the first software release. Subsequent steps can
then be based on the tarball of the previous milestone.
Note that if using "multistrap" or a foreign architecture
"debootstrap", ensure that the rootfs inside the tarball is
configured and repacked before being used with "ladder". i.e.
use the production tarball rather than the build system tarball.
Ladder checks the installed package list from the production tarball for that
release, calculates the packages needed to migrate to the specified milestone
and prepares a repository containing those packages, including all
dependencies.
If the specified package list and the specified milestone are
NOT
contiguous, errors can result if some of the contained packages need to
migrate between data formats. For most cases, create a ladder step for each
software release and upgrade devices in the same sequence. "ladder"
steps can be chained by modifying the update scripts.
Config files¶
Ladder configuration files live in
/etc/ladder.d/ and need to be named
after the release described. e.g.
/etc/ladder.d/internal.conf.
A minimal file to upgrade to Debian sid could look like:
[sid]
suite=unstable
location=http://ftp.uk.debian.org/debian
targetpackage=apt
A more comprehensive config file could look like:
[internal]
suite=interim
codename=milestone
branch=software_release_4
key=0xDEADBEEF
keyringdir=keys
location=copy:///srv/repo
rootpackage=libfoo3
targetpackage=metapackage
extrapackages=bar baz other
updatescript=true
mountpoint=/media/ladder
(It is possible to list more than one package, as a space separated list. Commas
or other markers will not be parsed by apt.)
The section name (e.g. internal in the example above) is used as the milestone
name, which can differ from the suite name and the branch name.
For more information on the key and keyringdir options, see the section on
SecureApt below.
Requirements¶
The rootfs is expected to carry some existing apt sources, the location
specified in the config file should be the one additional source which
provides the updated packages and the expectation is that this will have a
different suite name to the suite configured in the rootfs. If the location
and suite are the same, "apt" will print messages about duplicate
source lists but these messages can be ignored.
In order for apt to calculate the packages needed for the update,
all
repositories which are enabled in the rootfs tarball
including the
location specified in the config file
must be accessible on the machine
running "ladder".
Deployment of ladder tarballs¶
The final tarball contains an example apt source showing the syntax which would
be suitable for use with the packaged repository. The full path will need to
be specified in the final sources list file. e.g.
deb copy:///milestone suite main
May need to be modified to:
deb copy:///media/usb0/milestone suite main
The example source is packaged as
ladder.list in the tarball.
The key should normally already be part of a keyring package and installed on
the devices. If not, an exported copy of the public key is also included in
the tarball which can be included into the device keyring using
"apt-key" (which needs to be run as root):
apt-key add /path/milestone/ladder.gpg
Some scripting / programming support will be needed to make this process
seamless on-device, in particular to provide the knowledge of the actual
sequence of milestone names, but this is beyond the scope of
"ladder", if only because the ladder tarball needs to be unpacked
first.
If the system is set with some standard apt sources already, the upgrade will
need to only allow "apt-get" to see the ladder repository (because
the normal network connection isn't available, so the update would fail). To
do this, use apt command line options to reset the location of the SourceList
and SourceParts:
apt-get -o Dir::Etc::SourceList=ladder.list -o Dir::Etc::SourceParts=./dir/ update
(
./dir/ should be an empty directory - or a directory containing empty
.list files and nothing else.)
The only requirements to use the ladder tarball are to create the relevant
source list file, ensure the key is available and then call apt-get update;
apt-get upgrade. There is no need for perl, reprepro or anything else used by
"ladder" itself.
Example update script¶
If the configuration file includes the "updatescript" option an
example script will be included, listing the value of the
"rootpackage" option to be removed. If the "mountpoint"
option is set, the
DIR variable will be set in the example script as
well. (You may need to invest time in a "udev" rule as part of your
rootfs to get a known mount point but such rules are beyond the scope of this
documentation.)
#!/bin/sh
set -e
DIR=/media/ladder
ROOTPKG=
CONFIG=-y -o Dir::Etc::SourceList=${DIR}/ladder.list -o Dir::Etc::SourceParts=${DIR}/list.d/
apt-key add ${DIR}/pubkey.asc
apt-get ${CONFIG} update
if [ -n "$ROOTPKG" ]; then
apt-get ${CONFIG} --purge autoremove $ROOTPKG
fi
apt-get ${CONFIG} dist-upgrade
apt-get ${CONFIG} autoclean
SecureApt¶
Signing a ladder step repository requires that the secret key is usable without
a passphrase and that the secret key is accessible to the root user, either
directly or via sudo.
As with anything related to GnuPG, protecting the secret key is the sole
responsibility of the key owner. It is recommended that ladder steps are only
created in a secure environment comparable with that used to generate the
keys. The same requirements apply to the machines which use the secret key to
sign the internal milestone repositories, so it may be appropriate to create
ladder steps on those machines.
- Specifying a keyring directory and key ID
- If "keyringdir" is used, the specified directory
must contain the public and secret keyrings which contain the specified
"key". "ladder" will then make both the secret key and
public key accessible to the root user using a temporary keyring in
/var/lib/ladder/keys. Only the key available in
/var/lib/ladder/keys will be available to the repository signing
process. "ladder" only needs to be able to read the secret and
public keyrings of the keyringdir specified. Ensure that the secret
key is available - without a passphrase - or the repository will
not be signed.
- Using just a key ID
- If "keyringdir" is not used, the user must
ensure that the key is available to the root user as ladder requires
sudo/root to be able to use apt. Ensure that the specified secret key is
available - without a passphrase - to the root user or the
repository will not be signed.
sudo gpg --list-secret-key KEYID
Using "keyringdir" is generally the easiest option.
If the key is not available, the repository simply won't be signed and devices
would need to pass the AllowUnauthenticated option to "apt-get" when
using the ladder repository.
ladder does not add the unauthenticated
option to generated upgrade scripts! You can tell a SecureApt repository
by the presence of the
Release.gpg file.
It is possible to auto-generate GnuPG keys but "ladder" does not
support this currently. The main problem is entropy - generating a new GnuPG
(or SSH) key requires a lot of entropy, especially as default key lengths
increase. It is a lot easier to ensure high entropy when the key generation
process is interactive.
Keyring packages are recommended¶
With careful planning, the security of the step upgrades can be much improved by
modifying the update scripts to
not add the signing key using
"apt-key add" but instead to provide a keyring package in the rootfs
itself which contains the public key which will be used to sign the next
milestone. This is how Debian arranges keys - the release of milestone A is
not made until the key which will be used to sign milestone B has the
corresponding public key already included in the keyring package in milestone
A.
Such keyring packages themselves need to be in the milestone repository because
then the keyring package itself is protected by SecureApt.
Note that keyring packages will make it harder to use the downgrade solution
explained below, hence the need for planning.
Key expiries¶
Key expiry dates will complicate "ladder" usage, especially if
downgrades are to be available. If a device was released on milestone D and
needs to be downgraded to milestone B, you will have problems if the key used
to sign milestone B has since expired. Equally, repairing or servicing a
device running milestone B becomes problematic if the key for milestone B has
expired whilst the device was in use.
Avoid using expiry dates on keys unless you are very, very confident that a
particular milestone will not be in use after a certain date.
Key management¶
Keys can be revoked but this relies on the devices which need to verify that key
being able to download the revocation certificate and then to still have a
usable key available for the upgrade. Consider revoking the key for milestone
B in the version of the keyring package released with milestone D (milestone C
still needs it to be able to upgrade). This allows keys to be revoked
on-device but still be usable should it become necessary to repair, service or
downgrade.
If a key is compromised, then unless the keyring package in any one milestone
still includes a usable key, there may be no way of securely upgrading devices
without manually adding a replacement key. Take care of your secret keys.
Steps and milestones¶
Ladder - as with Debian - only works forwards. Downgrades are not supported. If
the rootfs tarball contains an existing apt source which contains packages
NEWER than the requested milestone, then the packages downloaded will
be for the existing apt source, not the milestone. Check the output with the
"-n|--dry-run" option.
However, judicious use of the "rootpackage" option can assist with
limited downgrades - especially when the software being downgraded is under
your own control. The generated
updatescript can use "apt-get
--purge autoremove" on the root package. Specifying a core library or
special platform dependency package here can allow the rootfs to be returned
to a pristine state. The required milestone can then be installed as if from a
clean base. This is not quite the same as an explicit downgrade but is a much
more reliable mechanism as it provides the equivalent rootfs to when the
original milestone was created.
For these reasons,
always keep a copy of the original clean rootfs which
has no complicating apt sources.
If your root package is a shared library, you can specify multiple root packages
in the config file so that all released SONAME versions are removed. Use only
spaces to separate packages in the config file.
If you are using keyring packages, ensure that a suitable keyring package is
available to the ladder step which purges the root package. To be able to
upgrade to the end milestone from a purged rootfs, the keyring package first
needs to be upgraded to include the key used to sign the end milestone
(although the upgraded keyring package is free to include revoked copies of
intermediary keys, if appropriate).
Output¶
Ladder works in the
/var/lib/ladder directory, unpacking the tarball into
./rootfs and creating the repository in a directory named after the
milestone.
Results will be
/var/lib/ladder/ladder-$name.tgz
Support¶
"ladder" was written with a specific purpose in mind but is available
in Debian in the hope it will be useful for other situations as well. If there
are specific situations where "ladder" could be extended to be more
useful for others, let me know using the Debian bug tracking system:
bugs.debian.org/ladder.
Note that "reprepro" already has
snapshot support which is not
the same as a "ladder" of milestones. Snapshots include full sources
and ancillary packages which are not needed on-device and are intended for
build systems and developer use - ladder milestones are intended to provide a
small repository which can be used on machines after production.