Scroll to navigation

MIRRORRIB(1) Debian, the Universal Operating System MIRRORRIB(1)

NAME

mirrorrib - locally mirror a Debian release, including backports

SYNOPSIS

mirrorrib [OPTIONS] DEBIAN-VER ARGUMENTS...
mirrorrib [-ls?-] [-p FILE] {10.N|11.N|12.N} CD-REV ARCH OLD TARGET ISO MIRROR [LANGS...]

DESCRIPTION

Debian releases a major revision of its operating system about every two years and a minor revision approximately quarterly, but these revisions exclude Debian backports. Debian releases backports only on a rolling basis like sid.

mirrorrib is for Debian users who want an approximately quarterly, stable revision of backports to accompany the approximately quarterly, stable revision of the rest of the operating system—with both revisions dated as of the same date.

mirrorrib reproducibly assembles a stable backports revision and release to accompany a stable regular revision and release. It downloads the matched pair of releases with all their packages and associated files, mirroring the pair together to your hard drive. After running mirrorrib and configuring /etc/apt/sources.list to access the local repository mirrorrib has assembled, one no longer needs a live network connection to update or reinstall one's system to Debian stable—not even if the update or reinstallation requires access to backports.

In addition to the matched pair of releases, mirrorrib also mirrors the standard, bootable system-installation image, netinst.iso, from the Debian-CD archive.

mirrorrib's name stands for “MIRROR Release Including Backports.”

ARGUMENTS

The first argument to mirrorrib, the DEBIAN-VER, specifies the stable version of Debian whose release mirrorrib is to fetch. The arrangement and number of the other arguments depend on the value of the first argument. (At present, only Debian 10.N, 11.N and 12.N are supported. mirrorrib arranges its command line similarly whether for Debian 10.N, for Debian 11.N or for Debian 12.N. When future support for Debian 13.N arrives, though, mirrorrib might arrange its command line for that Debian version differently. Check this page once future support arrives.)

The number of the stable version of Debian whose release mirrorrib is to fetch (for example 11.1 to fetch the .1 release of Debian 11 bullseye).
The Debian-CD revision number (usually 0, for example for Debian-CD revision 11.1.0).
The computer architecture (amd64, for example) for which the release is to be fetched.
To avoid unnecessary downloading, an old Debian repository already on your hard drive (/home/jsmith/debian-11.0, for example), from which mirrorrib can try to copy or hard-link files already downloaded. Enter /dev/null if no such repository is available, but beware: /dev/null results in a huge download, possibly of hundreds of GiB, which is presumably not what you wanted if you had run mirrorrib before and, thus, already possessed most of the needed files! (Note that /dev/null does not cause files to be downloaded to /dev/null, but to be copied locally from /dev/null, which is to say, not copied locally at all.)
The location (/home/jsmith/debian-11.1, for example), which must be a directory that does not yet exist, at which mirrorrib is to install the new, matched pair of Debian releases—with all their packages and associated files—as a proper Debian repository. You may find it useful to put this on a different hard-disk partition (as /other-partition/debian-11.1, for example) than the one onto which the operating system is installed, for doing so lets the new repository be used during a future wipe-and-reinstallation of the machine.
The location (/home/jsmith/debian-cd-11.1.0 or /other-partition/debian-cd-11.1.0, for example), which must be a directory that does not yet exist, at which mirrorrib is to install the new Debian-CD repository. (Note that only your architecture's netinst.iso along with its associated checksums are fetched to the new Debian-CD repository, for the Debian Project's debian-cd service specifies no particular release suite and netinst.iso is the minimal, standard ISO image one needs to boot the installation of a fresh Debian system—during which fresh installation one can rely on the new, local repository to supply packages and associated files instead of relying on a network connection. Alternate ISO images and various other nonessential Debian-CD files exist, but because a canonical, reproducible selection among those is impossible, none of those other images or files is fetched.)
The regular Debian mirror from which mirrorrib is to download (mirror.example.org, for example). Choose from the list at https://www.debian.org/mirror/ a single mirror that serves both debian/ and debian-cd/ for your architecture via the rsync protocol. (If unsure about rsync, then just pick a mirror and try. Most Debian mirrors support rsync and if the one you have chosen does not, then mirrorrib will detect that and abort before starting the big download. If you have issued the -s option, then the mirror does not matter, so you can just enter localhost in that case.)
A space-separated list of the languages whose package-description Translation files are to be fetched (for example, en fr de de_DE for English, French, German and the German of Germany; normally, you should at least list en here). Enter @ to fetch all available languages.

The examples use absolute pathnames but of course relative pathnames (debian-11.1 or foo/debian-11.1, for example) work just as well.

OPTIONS

Save local disk space by hard-linking pool files from OLD to TARGET where practicable. (Recommended. Note however that a hard-linked file is one, single file, accessible in two or more different locations. If for some reason mirrorrib found it necessary to update a hard-linked file—most likely by correcting the file's timestamp—then it would update the file equally in OLD as well as in TARGET.)
Change nothing, but print checksums as listed in the release's control files (such as Packages, Sources and SHA256SUMS files). This option does not care about the OLD, ISO and MIRROR parameters, so you can enter /dev/null for each of the first two and localhost for the last if you wish. However, it does require a correct LANGS list.
-?
Print a help message. Do nothing else.
--
Disable further options processing.
Set FILE's modification timestamp to indicate the date and time of the snapshot upon which mirrorrib has relied. (The FILE is not a file mirrorrib downloads but is any file you name, usually a nonexistent or empty file. Naming such a file is useful if you wish to identify at https://snapshot.debian.org/archive/debian/ the specific snapshot mirrorrib has referenced. If the file you name exists, then mirrorrib timestamps it but does not otherwise alter it. If the file does not exist, then mirrorrib creates and timestamps it.)

Besides these, mirrorrib also accepts --help or --version, provided that the option immediately follows the command. These do what you expect. The --help option does the same as the -? option does.

EXIT STATUS

0
Successful program execution.
Failure. [mirrorrib itself always generates the exit status 1 upon failure, but the various other programs mirrorrib calls during its execution, such as rsync(1), can issue other integers. In most cases, an error message will be printed to stderr to indicate the failure's cause.]

ENVIRONMENT

Summary: except for LC_MESSAGES, neither the environment nor the umask matters much to mirrorrib. Details follow.

The locale, as fr_FR.UTF-8 (for the French language of France) or de_AT.UTF-8 (for the German language of Austria) for example, to which mirrorrib's diagnostic messages are translated.
Alternatives to LC_MESSAGES that have exactly the same effect on mirrorrib LC_MESSAGES has. As locale(7) explains, LC_ALL supersedes LC_MESSAGES unless LC_ALL is null, whereas LANG serves as a fallback in case both of the others are null. If you need localization, then your LANG is probably already set to the right default, so you won't need to worry about the other variables; but if your LANG is null or wrong and you are unsure of what to do, then just set LC_MESSAGES and leave the other two alone. The most straightforward way to do this is to set the variable by prefixing the command line. For example, LC_MESSAGES=fr_FR.UTF-8 mirrorrib.... (Note that the last works only if you have generated the fr_FR.UTF-8 locale on your system. If you haven't and you wish to, then issue dpkg-reconfigure locales as root.)
No effect. mirrorrib uses UTC, regardless of the time zone to which TZ is set. (The matter of times zones is mentioned because the matter confuses some inexperienced Debian users and because timestamps are significant to mirrorrib. Internally, the Debian operating system keeps time solely in UTC, regardless of the time zone to which TZ is set. Files are stamped in UTC, not only by mirrorrib but by other programs, as well. The TZ affects how time is displayed by certain tools like date(1), but not how time is stored. The point is that, if you transferred a repository assembled by mirrorrib in one time zone to a second machine in another time zone, then the second machine could not—by examining the transferred repository's timestamps—infer the time zone in which the repository had been assembled. The second machine would find only timestamps in correct UTC. In other words, results are reliable. You need not worry about it.)
No effect. Internally, mirrorrib launches several other programs but, except for the aforementioned LC_MESSAGES, mirrorrib does not pass the user's environment variables to them.

Though the user's umask setting is not part of the environment as such (for a new Linux process inherits its umask, along with its working directory, directly without regard to the environment via the kernel's process-launching mechanism), the umask is mentioned here because mirrorrib respects it—momentarily. After downloading, mirrorrib resets the modes of downloaded files and symlinks to 0644 and of downloaded directories to 0755, anyway, so in the end the umask hardly matters in normal use cases. As far as the maintainer is aware, the umask affects no persistent production of mirrorrib except the -p option's timestamp. Nevertheless, the umask's usual default value on a Debian system is 0022, so if you wish to set a specific value before launching mirrorrib (or almost any other program), then you can issue umask 0022 first.

But it probably isn't necessary.

FILES

/etc/mirrorrib.conf
Systemwide configuration of mirrorrib. (You probably won't need to touch it.)
/etc/apt/sources.list
List of locations from which apt-get(8) is to fetch packages.

CONFORMING TO

  • Debian 10 buster
  • Debian 11 bullseye
  • Debian 12 bookworm (tentative)

USAGE WITH APT

Debian's rolling backports releases are each marked to expire a week after issue, whereas a mirrorrib repository expects to remain in service about three months, until the next stable release arrives. APT will object if asked to update according to an expired backports Release file. After mirrorrib has finished its run and you have suitably configured /etc/apt/sources.list, you can overcome APT's objection by issuing the command

apt-get -o Acquire::Check-Valid-Until=false update
    

as root.

If backports use with APT is generally unfamiliar to you, then refer to the -t option on the manual page of apt-get(8).

As far as /etc/apt/sources.list goes, if you are already as familiar with that file as many Debian users are, then you can configure the file however you like. However, if you are unsure, then, after backing up the existing file, you might try letting the file consist of several lines resembling these:

deb     file:///home/jsmith/debian-11.1/ bullseye main contrib non-free

deb-src file:///home/jsmith/debian-11.1/ bullseye main contrib non-free
deb     file:///home/jsmith/debian-11.1/ bullseye-backports main contrib non-free
deb-src file:///home/jsmith/debian-11.1/ bullseye-backports main contrib non-free

Notice that the repository mirrorrib has installed makes it unnecessary to list an online mirror, though if you require rolling security updates then, of course, you still must list the Debian Project's security mirror for that. (In the latter case, consult https://www.debian.org/security/ for further instructions.)

RELIANCE ON THE DEBIAN PROJECT'S SNAPSHOT SERVICE

mirrorrib relies on the Debian Project's snapshot service at https://snapshot.debian.org/ for several purposes, mainly to serve backports' dists/ and, in pool/, to serve specific backports packages that have expired from the regular mirrors.

The snapshot service is not a mirror. Therefore, it is important that the user not abuse it. Use is logged. Abusers can be banned, but if like most Debian users your habit is to be a good netizen, then the prospect of bans has little to do with you. Therefore, to moderate the load (and indeed to unclog your own Internet connection), please consider specifying an OLD repository when invoking mirrorrib if feasible.

(Kindly do not contact the maintainer regarding bans. The maintainer does not issue bans, cannot revoke them, and does not know you well enough to advocate on your behalf. If the maintainer understands, there was one instance in which the persistent actions of a single abuser forced the snapshot service to ban the entirety of a major Internet infrastructure company whose name would be familiar to you if the name were mentioned here, that the snapshot service could remain functional for the rest of us. So don't do that.)

For information, rather than serving files via rsync as the regular mirror does, snapshot serves files only via http(s), which leaves mirrorrib locally to treat several complications regarding symlinks, directories and timestamps. mirrorrib is programmed to treat these complications automatically, so the details need not concern the user; except that, to achieve reproducibility, mirrorrib is forced locally to stamp a standardized date and time on certain symlinks and directories (mainly regarding backports) rather than duplicating all timestamps off a regular mirror as one otherwise would do.

THE REPOSITORY'S SIZE

Even for a single stable release for a single architecture, the local repository mirrorrib assembles is big. The manual page you are reading cannot tell you in advance how big, but, at this writing, the last time the author ran mirrorrib, himself, for Debian bullseye 11.0 amd64, the repository proved to be 165 GiB in size. The size of backports' contribution fluctuates but, by the time you read this manual page, the combined repository including backports will probably have grown even larger than the 165 GiB.

Fortunately, during your second and subsequent invocations of mirrorrib, or even during your first invocation if you already happen to have a local repository on your hard drive by another tool, the -l option helps to conserve drive space.

THE @ LANGUAGE

When the author runs mirrorrib, himself, he lists on the command line only en for English plus the standard abbreviations for one or two other languages he is able to read. However, this is a matter of preference. Compared to the size of the overall repository, the size of Debian's Translation files is not very significant, so if you prefer to order @ for all available languages, go ahead and do it.

WIRELESS FIRMWARE TROUBLES

Refer to /usr/share/doc/mirrorrib/NON-FREE-FIRMWARE.

REPRODUCIBLE RESULTS

mirrorrib is designed to deliver strictly reproducible results, even as regards file-, symlink- and directory-modification timestamps. (If you should notice a difference between results obtained on one date or machine and results obtained on another date or machine, if you are running the latest mirrorrib, and if you can show how to reproduce the behavior, then kindly report the bug via Debian's Bug Tracking System with normal priority.)

TENTATIVE SUPPORT FOR DEBIAN 12

When the present version of mirrorrib was prepared, Debian 12 was still in testing. At that time, the present version of mirrorrib seemed ready to handle Debian 12 correctly. Nevertheless, mirrorrib's maintainer cannot guarantee that, when Debian 12 stable arrives, the present version of mirrorrib will still handle it correctly. Indeed, depending on the finalized details of Debian 12 (and on the soundness of mirrorrib's maintainer's forward-looking grasp of those details), there is a significant probability that mirrorrib will not handle it correctly. If it does not seem to, then look for an updated mirrorrib package in Debian's backports archive online.

PATIENCE ON THE DAY OF A NEW RELEASE

mirrorrib will probably not do what you want during the first 48 hours or so following the Debian Project's approximately quarterly announcement of a new stable release. The reason is that time is needed for Debian's Images Team to build and upload ISO CD installation images, for hundreds or thousands of pool and dists files to propagate to the mirrors, and for sufficient numbers of pre- and post-release snapshots to appear on snapshot.debian.org to afford mirrorrib enough information to deduce with confidence which is the optimal snapshot for mirrorrib to reference. If mirrorrib fails and the release is new, then try again tomorrow. Forty-eight hours probably suffice. The author has never noticed the overall delay to take longer than a week.

EXAMPLES

If an old repository is locally present and English, French, German and the German of Germany are wanted,

mirrorrib -lp /home/jsmith/snapshot.stamp 11.1 0 amd64 \

/home/jsmith/debian-11.0 \

/home/jsmith/debian-11.1 \
/home/jsmith/debian-cd-11.1.0 \
mirror.example.org en fr de de_DE

Alternately, if no old repository is locally present and all available languages are wanted (caution: because no old repository, a huge download will ensue!),

mirrorrib -p /home/jsmith/snapshot.stamp 11.1 0 amd64 \

/dev/null \

/home/jsmith/debian-11.1 \
/home/jsmith/debian-cd-11.1.0 \
mirror.example.org @

Subsequently, user jsmith, who has run the program, can examine the snapshot timestamp by

TZ=UTC stat -c%y /home/jsmith/snapshot.stamp
    

If user jsmith has sufficient hard-drive space available, then he or she can pack up pristine copies of the new repositories by

cd /home/jsmith

tar --owner=0 --group=0 -cf debian-11.1{.tar,}
tar --owner=0 --group=0 -cf debian-cd-11.1.0{.tar,}

(Note that it is probably unnecessary and undesirable for user jsmith to compress the .tar files as .tar.xz or the like, for most of the content is already compressed. Note also that user jsmith's language selection influences only which Translation files are retrieved. All packages belonging to the release for the selected architecture, including culture-specific packages, are retrieved in any case.)

Before running mirrorrib, if user jsmith has old files in two different old repositories, then he or she can issue

for A in $(find old2 -type f -printf '%P\n'); do

[ -e "old1/$A" ] || {

mkdir -pv "old1/$(dirname "$A")" \

&& ln -v "$(realpath -e old2)/$A" \

"$(realpath -e old1)/$A"
}
done

to merge the files of old2/ into old1/.

After running mirrorrib, if user jsmith wishes to verify, against the Release files mirrorrib has fetched, the rest of the files it has fetched, then user jsmith can issue

cd /home/jsmith/debian-11.1

sha256sum -c --quiet <(mirrorrib -s 11.1 0 amd64 \
/dev/null . /dev/null localhost @)

cd /home/jsmith/debian-cd-11.1.0/11.1.0/amd64/iso-cd

sha256sum -c --ignore-missing SHA256SUMS

Of the pair of sha256sum(1) commands, the first checks the main repository, takes perhaps half an hour, and is quiet if it finds no problems. The second checks the ISO CD repository's netinst.iso image, is much quicker, and prints a single line to verify the image. Observe that mirrorrib's -s option requires a correct LANGS list, so the example's @ is correct only if @ was also issued earlier, during mirrorrib's main run.

AUTHOR

Thaddeus H. Black

SEE ALSO

aptly(1), chmod(1), date(1), debmirror(1), dirname(1), df(1), du(1), find(1), ln(1), mkdir(1), realpath(1), rsync(1), sha256sum(1), stat(1), tar(1), wget(1), null(4), apt-conf(5), sources.list(5), locale(7), apt-get(8), dpkg-reconfigure(8)

2021-09-04 0.14.4.2