.\" Automatically generated by Pod::Man 2.23 (Pod::Simple 3.14) .\" .\" Standard preamble: .\" ======================================================================== .de Sp \" Vertical space (when we can't use .PP) .if t .sp .5v .if n .sp .. .de Vb \" Begin verbatim text .ft CW .nf .ne \\$1 .. .de Ve \" End verbatim text .ft R .fi .. .\" Set up some character translations and predefined strings. \*(-- will .\" give an unbreakable dash, \*(PI will give pi, \*(L" will give a left .\" double quote, and \*(R" will give a right double quote. \*(C+ will .\" give a nicer C++. Capital omega is used to do unbreakable dashes and .\" therefore won't be available. \*(C` and \*(C' expand to `' in nroff, .\" nothing in troff, for use with C<>. .tr \(*W- .ds C+ C\v'-.1v'\h'-1p'\s-2+\h'-1p'+\s0\v'.1v'\h'-1p' .ie n \{\ . ds -- \(*W- . ds PI pi . if (\n(.H=4u)&(1m=24u) .ds -- \(*W\h'-12u'\(*W\h'-12u'-\" diablo 10 pitch . if (\n(.H=4u)&(1m=20u) .ds -- \(*W\h'-12u'\(*W\h'-8u'-\" diablo 12 pitch . ds L" "" . ds R" "" . ds C` "" . ds C' "" 'br\} .el\{\ . ds -- \|\(em\| . ds PI \(*p . ds L" `` . ds R" '' 'br\} .\" .\" Escape single quotes in literal strings from groff's Unicode transform. .ie \n(.g .ds Aq \(aq .el .ds Aq ' .\" .\" If the F register is turned on, we'll generate index entries on stderr for .\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index .\" entries marked with X<> in POD. Of course, you'll have to process the .\" output yourself in some meaningful fashion. .ie \nF \{\ . de IX . tm Index:\\$1\t\\n%\t"\\$2" .. . nr % 0 . rr F .\} .el \{\ . de IX .. .\} .\" .\" Accent mark definitions (@(#)ms.acc 1.5 88/02/08 SMI; from UCB 4.2). .\" Fear. Run. Save yourself. No user-serviceable parts. . \" fudge factors for nroff and troff .if n \{\ . ds #H 0 . ds #V .8m . ds #F .3m . ds #[ \f1 . ds #] \fP .\} .if t \{\ . ds #H ((1u-(\\\\n(.fu%2u))*.13m) . ds #V .6m . ds #F 0 . ds #[ \& . ds #] \& .\} . \" simple accents for nroff and troff .if n \{\ . ds ' \& . ds ` \& . ds ^ \& . ds , \& . ds ~ ~ . ds / .\} .if t \{\ . ds ' \\k:\h'-(\\n(.wu*8/10-\*(#H)'\'\h"|\\n:u" . ds ` \\k:\h'-(\\n(.wu*8/10-\*(#H)'\`\h'|\\n:u' . ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'^\h'|\\n:u' . ds , \\k:\h'-(\\n(.wu*8/10)',\h'|\\n:u' . ds ~ \\k:\h'-(\\n(.wu-\*(#H-.1m)'~\h'|\\n:u' . ds / \\k:\h'-(\\n(.wu*8/10-\*(#H)'\z\(sl\h'|\\n:u' .\} . \" troff and (daisy-wheel) nroff accents .ds : \\k:\h'-(\\n(.wu*8/10-\*(#H+.1m+\*(#F)'\v'-\*(#V'\z.\h'.2m+\*(#F'.\h'|\\n:u'\v'\*(#V' .ds 8 \h'\*(#H'\(*b\h'-\*(#H' .ds o \\k:\h'-(\\n(.wu+\w'\(de'u-\*(#H)/2u'\v'-.3n'\*(#[\z\(de\v'.3n'\h'|\\n:u'\*(#] .ds d- \h'\*(#H'\(pd\h'-\w'~'u'\v'-.25m'\f2\(hy\fP\v'.25m'\h'-\*(#H' .ds D- D\\k:\h'-\w'D'u'\v'-.11m'\z\(hy\v'.11m'\h'|\\n:u' .ds th \*(#[\v'.3m'\s+1I\s-1\v'-.3m'\h'-(\w'I'u*2/3)'\s-1o\s+1\*(#] .ds Th \*(#[\s+2I\s-2\h'-\w'I'u*3/5'\v'-.3m'o\v'.3m'\*(#] .ds ae a\h'-(\w'a'u*4/10)'e .ds Ae A\h'-(\w'A'u*4/10)'E . \" corrections for vroff .if v .ds ~ \\k:\h'-(\\n(.wu*9/10-\*(#H)'\s-2\u~\d\s+2\h'|\\n:u' .if v .ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'\v'-.4m'^\v'.4m'\h'|\\n:u' . \" for low resolution devices (crt and lpr) .if \n(.H>23 .if \n(.V>19 \ \{\ . ds : e . ds 8 ss . ds o a . ds d- d\h'-1'\(ga . ds D- D\h'-1'\(hy . ds th \o'bp' . ds Th \o'LP' . ds ae ae . ds Ae AE .\} .rm #[ #] #H #V #F C .\" ======================================================================== .\" .IX Title "Emdebian::Grip 3" .TH Emdebian::Grip 3 "2011-03-27" "perl v5.12.3" "User Contributed Perl Documentation" .\" For nroff, turn off justification. Always turn off hyphenation; it makes .\" way too many mistakes in technical documents. .if n .ad l .nh .SH "NAME" Emdebian::Grip \- internal Emdebian module for Emdebian Grip repositories .SH "Description" .IX Header "Description" It is worthwhile being familiar with \f(CW\*(C`reprepro\*(C'\fR (1) when working with this module. .PP Emdebian::Grip provides repository management support for the \&\fIem_autogrip\fR scripting to manage a three-way repository hierarchy for Emdebian Grip, including support for migrating packages from unstable into testing. .PP Considering just \f(CW\*(C`unstable\*(C'\fR initially, the hierarchy is as follows: .PP Debian unstable is filtered using \fI${base}${filter_name}/conf/pkglist\fR which in turn is based on the output of dpkg \-\-get\-selections. Only packages that are actually installed on machines running Emdebian Grip get added to the pkglist. \f(CW\*(C`reprepro\*(C'\fR then updates \fI${filter_name}\fR from the chosen Debian mirror, downloading only the binary and source packages specified in pkglist, to create a partial local mirror for the list of supported architectures. .PP Grip unstable is built from the filtered mirror by a combined process of converting the source package and then identifying the missing binary packages from the pkglist. During the \f(CW\*(C`emgrip\*(C'\fR process, Emdebian TDebs are created which get included into the locale repository: .PP Locale is a very specialised repository that is designed for Debian source packages and Emdebian TDebs only, organised by locale root. e.g. en_GB support is in the \fIen\fR component. Whereas Debian has three components (\fImain\fR, \fIcontrib\fR and \fInon-free\fR), the \fBlocale repository has ninety seven components\fR (including main for the source packages). In the locale pool, packages are organised by component so whereas in the filter or grip repository, \f(CW\*(C`apt\*(C'\fR packages are under \&\fIpool/main/a/apt\fR in the locale repository the \fIen\fR TDebs for apt are in \fIpool/en/a/apt/\fR and the French under \fIpool/fr/a/apt/\fR etc. The list of locale roots supported by the locale repository is returned by \&\fI&get_locale_roots\fR, part of the \f(CW\*(C`Debian::Packages::Compare\*(C'\fR module. For more information on locale roots as components see: .PP As far as testing is concerned, the hierarchy works in a similar way to Debian, but with extra support for repositories that are started after packages have migrated into Debian testing and a newer version exists in Debian unstable. .PP Note that the testing repository data needs to be setup manually in many cases \- editing conf/distributions for each repository. The current Emdebian conf files will find their way onto the Debian Wiki in due course. .PP With a fully populated Debian unstable, Grip unstable and locale unstable, \fI&grip_britney\fR can then migrate packages into Grip testing using the criteria implemented in \fI&get_britney_list\fR from \&\f(CW\*(C`Debian::Packages::Compare\*(C'\fR and the \f(CW\*(C`reprepro copysrc\*(C'\fR command. As most, if not all, Grip repositories will begin at a time when some packages in Debian have already migrated into testing and had a newer version uploaded to unstable, \f(CW\*(C`Emdebian::Grip\*(C'\fR also includes \&\fIcatch-up\fR support via \fI&migrate_missing\fR which includes the relevant packages from Debian testing as if that version had been uploaded directly to Grip testing (which, in effect, is what is happening). .PP The \fIemgrip\fR process can take \fBa significant amount of time\fR \- think many hours, not minutes \- principally due to the overhead of generating the TDebs at one per source per locale per architecture. To create a full unstable and testing set is likely to take a day or more but is mostly automated. .PP The problems that remain are related to dependency issues. The \fI&edos\fR function in \f(CW\*(C`Emdebian::Grip\*(C'\fR does have problems trying to calculate the solution for dependency problems identified by \f(CW\*(C`edos\-debcheck\*(C'\fR and the problems become particularly acute if the repository itself is not in good shape at the start. Therefore, \fI&edos\fR is not run automatically by the current tools. The repository must be fully up to date and all relevant packages need to exist in the filter and in Grip before using \fI&edos\fR support. Initially, use \f(CW\*(C`edos\-debcheck\*(C'\fR directly to identify major problems: .PP .Vb 1 \& edos\-debcheck \-explain \-failures < ${base}${grip_name}/dists/sid/main/binary\-i386/Packages .Ve .PP Packages that are \fI\s-1NOT\s0 \s-1AVAILABLE\s0\fR need to be gripped as source or binary packages as appropriate but care is needed to correctly interpret the \f(CW\*(C`edos\-debcheck\*(C'\fR output to identify the correct link in the chain. .PP Two major problems exist with automating this process, see \&\f(CW\*(C`em_autogrip\*(C'\fR (1) for specific information, but suffice it to say here that there exist in Debian unstable, packages that will appear in the list of \fI\s-1NOT\s0 \s-1AVAILABLE\s0\fR packages where the correct action is to \&\fB\s-1REMOVE\s0\fR the package that depends on the unavailable package. These issues are being discussed within Debian. .SH "Copyright and Licence" .IX Header "Copyright and Licence" .Vb 1 \& Copyright (C) 2007\-2009 Neil Williams \& \& This package is free software; you can redistribute it and/or modify \& it under the terms of the GNU General Public License as published by \& the Free Software Foundation; either version 3 of the License, or \& (at your option) any later version. \& \& This program is distributed in the hope that it will be useful, \& but WITHOUT ANY WARRANTY; without even the implied warranty of \& MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the \& GNU General Public License for more details. \& \& You should have received a copy of the GNU General Public License \& along with this program. If not, see . .Ve .SH "Example" .IX Header "Example" .Vb 4 \& use strict; \& use warnings; \& use Emdebian::Grip; \& use Debian::Packages::Compare; \& \& use vars qw/ $filter_name $grip_name $suite $base $verbose \& $noskip @archlist @locroots @lines $line %pkg @filter $have \& %debianunstable %gripunstable %tdebunstable /; \& \& my $mirror=\*(Aqhttp://ftp.uk.debian.org/debian\*(Aq; # default \& $filter_name = \*(Aqfilter\*(Aq; \& $grip_name = \*(Aqgrip\*(Aq; \& $suite = "unstable"; # at first \& $base = \*(Aq/opt/reprepro/\*(Aq; \& $verbose = 0; \& \& &set_base($base); \& &set_repo_names ($filter_name, $grip_name); \& my $a = &get_archlist ($suite, $filter_name); \& @archlist = (not defined $a or not @$a) ? \& qw/i386 amd64 arm armel powerpc mips mipsel/ : @$a; \& my $l = &get_locale_roots ($suite, \*(Aqlocale\*(Aq); \& @locroots = (not defined $l or not @$l) ? qw/ af am ang ar as ast az be bg \& bn br bs ca cs cy da de dz el en eo es et eu fa fi fr ga gl gu he hi hr \& hu hy ia id io is it ja ka kn km ko ku ky lg li lt lv mai mg mi mk ml mn mr \& ms nb ne nl nn no ns nso oc or pa pl ps pt rm ro ru rw si sk sl sq sr sv \& ta te th tk tl tr tt ug uk ur uz vi wa wo xh yi zh zu / : @$l; \& \& &setup_repos if ( not \-f "${base}${filter_name}/conf/pkglist" ); \& \& my $debu = &read_packages (\*(Aqunstable\*(Aq, $filter_name); \& my $gripu = &read_packages (\*(Aqunstable\*(Aq, $grip_name); \& my $tdebu = &read_locale (\*(Aqunstable\*(Aq, \*(Aqlocale\*(Aq); \& %debianunstable = %$debu if (defined $debu); \& %gripunstable = %$gripu if (defined $gripu); \& %tdebunstable = %$tdebu if (defined $tdebu); \& &update_filter; \& &update_repo($verbose); \& \& # begin the work. .Ve .SH "set_repo_names" .IX Header "set_repo_names" Copies the default or user-specified filter and grip repository names into the internal module. .SH "set_noskip" .IX Header "set_noskip" Copies the user-specified \-\-noskipold option into the internal module for use when calling reprepro. .SH "set_dry_run" .IX Header "set_dry_run" Configures a dry-run where external commands are printed instead of executed. .SH "print_missing" .IX Header "print_missing" Outputs the list returned by \&\f(CW\*(C`Debian::Package::Compare::get_missing_sources\*(C'\fR in a simple format .SH "print_build_deps" .IX Header "print_build_deps" Outputs the list returned by \&\f(CW\*(C`Debian::Package::Compare::get_missing_builddeps\*(C'\fR in a simple format .SH "print_testing_status" .IX Header "print_testing_status" Provides a detailed output of the status of the testing repository for Grip, comparing Debian unstable (filter) against Debian testing (filter) and identifying which source packages in Emdebian Grip testing are behind Debian. .PP This list is further split into those where the source version in Debian unstable is precisely the same as the source version in Debian testing (packages that need to be migrated into Grip testing) and those where the version in testing differs. .SH "grip_britney" .IX Header "grip_britney" Grip version of the Britney script that migrates packages into Debian testing. See \f(CW\*(C`Debian::Packages::Compare\*(C'\fR (3) for details on \fIget_britney_list\fR criteria. .PP In addition, in order to allow new repositories to catch up with Debian, packages in Grip unstable that are already ahead of Debian testing need to be gripped directly from Debian testing. .PP This can also happen when packages are put directly into testing using testing-proposed-updates during a release freeze in Debian. .PP Note that grip_britney only migrates packages where the version in Debian unstable is the same as the version in Debian testing \- i.e. where a migration has already taken place in Debian. Packages that arrive in testing via testing-proposed-updates or migrated into testing before the Grip repository was started and which have a later version now in unstable, can be migrated using migrate_missing. .PP If you are running \fIgrip_britney\fR and \fImigrate_missing\fR together, \&\fBrun grip_britney first and reload all the data\fR using \fIread_packages\fR before attempting to run \fImigrate_missing\fR. See \fImigrate_missing\fR for more information. .SH "migrate_missing" .IX Header "migrate_missing" Corollary of grip_britney that completes the process by including into testing, packages that have a different version in Debian unstable. .PP Missing packages in testing are included using \fIgrip_source\fR and \&\fIgrip_binary\fR as if uploaded directly to testing (which, in effect, is what is happening). This means that \fImigrate_missing\fR will take just as long to process the package in testing as it did to process the later version that went into unstable. Depending on the number of packages affected, \fBmigrate_missing can take a lot longer than grip_britney\fR but it should be needed less often once the repository is complete. .SH "edos" .IX Header "edos" Only a basic framework at this stage, this function needs to eventually collate the results by architecture and repository and try to present a solution. .PP See \f(CW\*(C`em_autogrip\*(C'\fR (1) for problems with edos recursion. .PP In \f(CW$mode\fR eq 'edos', simply outputs the check data for each of the filter and grip repositories. .SH "clean_incoming" .IX Header "clean_incoming" Call intermittently during long runs. .SH "cleanup" .IX Header "cleanup" Only call before exit. .SH "update_filter" .IX Header "update_filter" Fill in the FilterList so that automatic updates can proceed \- without a filter, the entire archive is added. However, updating takes an appreciable amount of time, so call only when necessary. .SH "extend_filter" .IX Header "extend_filter" Adds package(s) to the filter list, a space separated list is acceptable. .SH "update_repo" .IX Header "update_repo" Once the pkglist filter file is correct, call reprepro to get updated packages. .SH "incoming_locale" .IX Header "incoming_locale" Checks the incoming directory for Emdebian TDebs and includes them into reprepro. .PP Until the .tdeb extension is supported, reprepro needs to be persuaded to allow such files using the \f(CW\*(C`\-\-ignore=extension\*(C'\fR option. .PP Amongst all the expected \fIerror\fR messages, some useful output is sent to \f(CW\*(C`STDERR\*(C'\fR by reprepro when handling TDebs so errors are not redirected to \fI/dev/null\fR. .SH "check_dirs" .IX Header "check_dirs" Check that critical directories in the three-way repository hierarchy exist and create any that are missing. .SH "setup_repos" .IX Header "setup_repos" Note: When setting up your own repository, remember that including the Contents: specifier in reprepro costs \fBa lot\fR of time during each and every operation on the repository. Contents is not enabled by \f(CW\*(C`setup_repos\*(C'\fR and it is strongly recommended that it is not re-enabled in later changes. .PP Need to decide whether this will cope with multiple suites or just enforce suite=unstable. .PP locale is shared with Crush. It's not particularly easy to add a new locale root as the scripts and the repos need to be changed at the same time. .PP necessary reprepro data (needs config support): these files need to be created for reprepro final repos will also need \f(CW\*(C`SignWith: 0x97BB3B58\*(C'\fR except stable which should be signed manually to bridge the security gap between Grip and Debian. .SH "convert_prefix" .IX Header "convert_prefix" Debian repositories use the index character of the source package name in the path to the package directory beneath the pool in order to reduce the number of listings per directory to a manageable level. The exception is source packages beginning with '\f(CW\*(C`lib\*(C'\fR', because there are so many that the '\f(CW\*(C`l\*(C'\fR' section would be too large. Whilst '\f(CW\*(C`l\*(C'\fR' remains, any source package that begins with '\f(CW\*(C`lib\*(C'\fR' is put under a separate set of directories using the first four letters instead of just '\f(CW\*(C`l\*(C'\fR', e.g. \f(CW\*(C`libaa\*(C'\fR will be found under \f(CW\*(C`pool/main/liba/libaa/\*(C'\fR and \f(CW\*(C`libfoo\*(C'\fR under \&\f(CW\*(C`pool/main/libf/libfoo/\*(C'\fR but \f(CW\*(C`limpet\*(C'\fR would be under \&\f(CW\*(C`pool/main/l/limpet/\*(C'\fR .PP \&\fIconvert_prefix\fR is a quick function to determine the correct prefix under these rules. .SH "grip_source" .IX Header "grip_source" Wrapper for emgrip that takes a single Debian source package, grips the binary package where the source package name matches the binary package name (due to a hitch in how \f(CW\*(C`reprepro\*(C'\fR works), generates any Emdebian TDebs, includes the unchanged source package into Grip, includes any gripped binary packages into Grip and includes any generated Emdebian TDebs into locale. .PP Once the source package has been processed, runs \fI&clean_incoming\fR. .SH "grip_binary" .IX Header "grip_binary" Wrapper for emgrip that takes a single Debian binary package, and grips it, generates any Emdebian TDebs, includes the gripped binary package into Grip and includes any generated Emdebian TDebs into locale. .PP Once the binary package has been processed, runs \fI&clean_incoming\fR. .SS "Components and Sections" .IX Subsection "Components and Sections" Each binary package belongs to a \f(CW\*(C`Section\*(C'\fR that broadly describes the type of functionality in the package. If a component exists in the repository with the same name as the \f(CW\*(C`Section\*(C'\fR, the package will be put into that component. Additionally, if the package name ends in \f(CW\*(C`\-dev\*(C'\fR or the \f(CW\*(C`Section\*(C'\fR is devel, the \f(CW\*(C`dev\*(C'\fR component is selected. If the package name ends in \f(CW\*(C`\-doc\*(C'\fR or the \f(CW\*(C`Section\*(C'\fR is \&\f(CW\*(C`doc\*(C'\fR, the \f(CW\*(C`doc\*(C'\fR component is selected. If there is no matching component configured for the Grip repository, \f(CW\*(C`main\*(C'\fR is used. Section names themselves are read from the filter repository \- i.e. set by the Debian ftp-master overrides. .SH "switch_component" .IX Header "switch_component" Single routine to work out which components should be used for which sections. .SH "AUTHOR" .IX Header "AUTHOR" Neil Williams, \f(CW\*(C`\*(C'\fR .SH "BUGS" .IX Header "BUGS" Please report any bugs or feature requests to the Debian Bug Tracking System using \f(CW\*(C`reportbug emdebian\-tools\*(C'\fR. .SH "Known bugs" .IX Header "Known bugs" (Otherwise known as a ToDo list.) .PP The status of TDebs in the locale repository needs to be checked before generating more but this means changes in emgrip too so that tdebs can be skipped with a command line option. However, in normal operation, the package is only processed because it is out of date so the TDebs and other files need to be replaced anyway. .PP \&\f(CW\*(C`Architecture: all\*(C'\fR are packages being continually reprocessed for several architectures. Originally this was because the package could contain translations which would need to be architecture-dependent in Emdebian. However, TDebs should be being generated only from source packages and this step probably needs to resolved. \f(CW\*(C`reprepro\*(C'\fR correctly handles such duplication, it is just a bit noisy when handling it and it adds to the time required by the process as a whole. .PP Also need a general overview function that summaries all three repositories.