.\" Automatically generated by Pod::Man 4.14 (Pod::Simple 3.40)
.\"
.\" Standard preamble:
.\" ========================================================================
.de Sp \" Vertical space (when we can't use .PP)
.if t .sp .5v
.if n .sp
..
.de Vb \" Begin verbatim text
.ft CW
.nf
.ne \\$1
..
.de Ve \" End verbatim text
.ft R
.fi
..
.\" Set up some character translations and predefined strings. \*(-- will
.\" give an unbreakable dash, \*(PI will give pi, \*(L" will give a left
.\" double quote, and \*(R" will give a right double quote. \*(C+ will
.\" give a nicer C++. Capital omega is used to do unbreakable dashes and
.\" therefore won't be available. \*(C` and \*(C' expand to `' in nroff,
.\" nothing in troff, for use with C<>.
.tr \(*W-
.ds C+ C\v'-.1v'\h'-1p'\s-2+\h'-1p'+\s0\v'.1v'\h'-1p'
.ie n \{\
. ds -- \(*W-
. ds PI pi
. if (\n(.H=4u)&(1m=24u) .ds -- \(*W\h'-12u'\(*W\h'-12u'-\" diablo 10 pitch
. if (\n(.H=4u)&(1m=20u) .ds -- \(*W\h'-12u'\(*W\h'-8u'-\" diablo 12 pitch
. ds L" ""
. ds R" ""
. ds C` ""
. ds C' ""
'br\}
.el\{\
. ds -- \|\(em\|
. ds PI \(*p
. ds L" ``
. ds R" ''
. ds C`
. ds C'
'br\}
.\"
.\" Escape single quotes in literal strings from groff's Unicode transform.
.ie \n(.g .ds Aq \(aq
.el .ds Aq '
.\"
.\" If the F register is >0, we'll generate index entries on stderr for
.\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index
.\" entries marked with X<> in POD. Of course, you'll have to process the
.\" output yourself in some meaningful fashion.
.\"
.\" Avoid warning from groff about undefined register 'F'.
.de IX
..
.nr rF 0
.if \n(.g .if rF .nr rF 1
.if (\n(rF:(\n(.g==0)) \{\
. if \nF \{\
. de IX
. tm Index:\\$1\t\\n%\t"\\$2"
..
. if !\nF==2 \{\
. nr % 0
. nr F 2
. \}
. \}
.\}
.rr rF
.\"
.\" Accent mark definitions (@(#)ms.acc 1.5 88/02/08 SMI; from UCB 4.2).
.\" Fear. Run. Save yourself. No user-serviceable parts.
. \" fudge factors for nroff and troff
.if n \{\
. ds #H 0
. ds #V .8m
. ds #F .3m
. ds #[ \f1
. ds #] \fP
.\}
.if t \{\
. ds #H ((1u-(\\\\n(.fu%2u))*.13m)
. ds #V .6m
. ds #F 0
. ds #[ \&
. ds #] \&
.\}
. \" simple accents for nroff and troff
.if n \{\
. ds ' \&
. ds ` \&
. ds ^ \&
. ds , \&
. ds ~ ~
. ds /
.\}
.if t \{\
. ds ' \\k:\h'-(\\n(.wu*8/10-\*(#H)'\'\h"|\\n:u"
. ds ` \\k:\h'-(\\n(.wu*8/10-\*(#H)'\`\h'|\\n:u'
. ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'^\h'|\\n:u'
. ds , \\k:\h'-(\\n(.wu*8/10)',\h'|\\n:u'
. ds ~ \\k:\h'-(\\n(.wu-\*(#H-.1m)'~\h'|\\n:u'
. ds / \\k:\h'-(\\n(.wu*8/10-\*(#H)'\z\(sl\h'|\\n:u'
.\}
. \" troff and (daisy-wheel) nroff accents
.ds : \\k:\h'-(\\n(.wu*8/10-\*(#H+.1m+\*(#F)'\v'-\*(#V'\z.\h'.2m+\*(#F'.\h'|\\n:u'\v'\*(#V'
.ds 8 \h'\*(#H'\(*b\h'-\*(#H'
.ds o \\k:\h'-(\\n(.wu+\w'\(de'u-\*(#H)/2u'\v'-.3n'\*(#[\z\(de\v'.3n'\h'|\\n:u'\*(#]
.ds d- \h'\*(#H'\(pd\h'-\w'~'u'\v'-.25m'\f2\(hy\fP\v'.25m'\h'-\*(#H'
.ds D- D\\k:\h'-\w'D'u'\v'-.11m'\z\(hy\v'.11m'\h'|\\n:u'
.ds th \*(#[\v'.3m'\s+1I\s-1\v'-.3m'\h'-(\w'I'u*2/3)'\s-1o\s+1\*(#]
.ds Th \*(#[\s+2I\s-2\h'-\w'I'u*3/5'\v'-.3m'o\v'.3m'\*(#]
.ds ae a\h'-(\w'a'u*4/10)'e
.ds Ae A\h'-(\w'A'u*4/10)'E
. \" corrections for vroff
.if v .ds ~ \\k:\h'-(\\n(.wu*9/10-\*(#H)'\s-2\u~\d\s+2\h'|\\n:u'
.if v .ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'\v'-.4m'^\v'.4m'\h'|\\n:u'
. \" for low resolution devices (crt and lpr)
.if \n(.H>23 .if \n(.V>19 \
\{\
. ds : e
. ds 8 ss
. ds o a
. ds d- d\h'-1'\(ga
. ds D- D\h'-1'\(hy
. ds th \o'bp'
. ds Th \o'LP'
. ds ae ae
. ds Ae AE
.\}
.rm #[ #] #H #V #F C
.\" ========================================================================
.\"
.IX Title "PUBLIC-INBOX-V2-FORMAT 5"
.TH PUBLIC-INBOX-V2-FORMAT 5 "1993-10-02" "public-inbox.git" "public-inbox user manual"
.\" For nroff, turn off justification. Always turn off hyphenation; it makes
.\" way too many mistakes in technical documents.
.if n .ad l
.nh
.SH "NAME"
public\-inbox\-v2\-format \- structure of public inbox v2 archives
.SH "DESCRIPTION"
.IX Header "DESCRIPTION"
The v2 format is designed primarily to address several
scalability problems of the original format described at
\&\fBpublic\-inbox\-v1\-format\fR\|(5). It also handles messages with
Message-IDs.
.SH "INBOX LAYOUT"
.IX Header "INBOX LAYOUT"
The key change in v2 is the inbox is no longer a bare git
repository, but a directory with two or more git repositories.
v2 divides git repositories by time \*(L"epochs\*(R" and Xapian
databases for parallelism by \*(L"shards\*(R".
.SS "\s-1INBOX OVERVIEW AND DEFINITIONS\s0"
.IX Subsection "INBOX OVERVIEW AND DEFINITIONS"
.Vb 3
\& $EPOCH \- Integer starting with 0 based on time
\& $SCHEMA_VERSION \- DB schema version (for Xapian)
\& $SHARD \- Integer starting with 0 based on parallelism
\&
\& foo/ # "foo" is the name of the inbox
\& \- inbox.lock # lock file to protect global state
\& \- git/$EPOCH.git # normal git repositories
\& \- all.git # empty, alternates to $EPOCH.git
\& \- xap$SCHEMA_VERSION/$SHARD # per\-shard Xapian DB
\& \- xap$SCHEMA_VERSION/over.sqlite3 # OVER\-view DB for NNTP, threading
\& \- msgmap.sqlite3 # same the v1 msgmap
.Ve
.PP
For blob lookups, the reader only needs to open the \*(L"all.git\*(R"
repository with \f(CW$GIT_DIR\fR/objects/info/alternates which references
every \f(CW$EPOCH\fR.git repo.
.PP
Individual \f(CW$EPOCH\fR.git repos \s-1DO NOT\s0 use alternates themselves as
git currently limits recursion of alternates nesting depth to 5.
.SS "\s-1GIT EPOCHS\s0"
.IX Subsection "GIT EPOCHS"
One of the inherent scalability problems with git itself is the
full history of a project must be stored and carried around to
all clients. To address this problem, the v2 format uses
multiple git repositories, stored as time-based \*(L"epochs\*(R".
.PP
We currently divide epochs into roughly one gigabyte segments;
but this size can be configurable (if needed) in the future.
.PP
A pleasant side-effect of this design is the git packs of older
epochs are stable, allowing them to be cloned without requiring
expensive pack generation. This also allows clients to clone
only the epochs they are interested in to save bandwidth and
storage.
.PP
To minimize changes to existing v1\-based code and simplify our
code, we use the \*(L"alternates\*(R" mechanism described in
\&\fBgitrepository\-layout\fR\|(5) to link all the epoch repositories
with a single read-only \*(L"all.git\*(R" endpoint.
.PP
Processes retrieve blobs via the \*(L"all.git\*(R" repository, while
writers write blobs directly to epochs.
.SS "\s-1GIT TREE LAYOUT\s0"
.IX Subsection "GIT TREE LAYOUT"
One key problem specific to v1 was large trees were frequently a
performance problem as name lookups are expensive and there were
limited deltafication opportunities with unpredictable file
names. As a result, all Xapian-enabled installations retrieve
blob object_ids directly in v1, bypassing tree lookups.
.PP
While dividing git repositories into epochs caps the growth of
trees, worst-case tree size was still unnecessary overhead and
worth eliminating.
.PP
So in contrast to the big trees of v1, the v2 git tree contains
only a single file at the top-level of the tree, either 'm' (for
\&'mail' or 'message') or 'd' (for deleted). A tree does not have
\&'m' and 'd' at the same time.
.PP
Mail is still stored in blobs (instead of inline with the commit
object) as we still need a stable reference in the indices in
case commit history is rewritten to comply with legal
requirements.
.PP
After-the-fact invocations of public-inbox-index will ignore
messages written to 'd' after they are written to 'm'.
.PP
Deltafication is not significantly improved over v1, but overall
storage for trees is made as as small as possible. Initial
statistics and benchmarks showing the benefits of this approach
are documented at:
.PP
.SS "\s-1XAPIAN SHARDS\s0"
.IX Subsection "XAPIAN SHARDS"
Another second scalability problem in v1 was the inability to
utilize multiple \s-1CPU\s0 cores for Xapian indexing. This is
addressed by using shards in Xapian to perform import
indexing in parallel.
.PP
As with git alternates, Xapian natively supports a read-only
interface which transparently abstracts away the knowledge of
multiple shards. This allows us to simplify our read-only
code paths.
.PP
The performance of the storage device is now the bottleneck on
larger multi-core systems. In our experience, performance is
improved with high-quality and high-quantity solid-state storage.
Issuing \s-1TRIM\s0 commands with \fBfstrim\fR\|(8) was necessary to maintain
consistent performance while developing this feature.
.PP
Rotational storage devices perform significantly worse than
solid state storage for indexing of large mail archives; but are
fine for backup and usable for small instances.
.PP
As of public-inbox 1.6.0, the \f(CW\*(C`publicInbox.indexSequentialShard\*(C'\fR
option of \fBpublic\-inbox\-index\fR\|(1) may be used with a high shard
count to ensure individual shards fit into page cache when the entire
Xapian \s-1DB\s0 cannot.
.PP
Our use of the \*(L"\s-1OVERVIEW DB\*(R"\s0 requires Xapian document IDs to
remain stable. Using \fBpublic\-inbox\-compact\fR\|(1) and
\&\fBpublic\-inbox\-xcpdb\fR\|(1) wrappers are recommended over tools
provided by Xapian.
.SS "\s-1OVERVIEW DB\s0"
.IX Subsection "OVERVIEW DB"
Towards the end of v2 development, it became apparent Xapian did
not perform well for sorting large result sets used to generate
the landing page in the \s-1PSGI UI\s0 (/$INBOX/) or many queries used
by the \s-1NNTP\s0 server. Thus, SQLite was employed and the Xapian
\&\*(L"skeleton\*(R" \s-1DB\s0 was renamed to the \*(L"overview\*(R" \s-1DB\s0 (after the \s-1NNTP
OVER/XOVER\s0 commands).
.PP
The overview \s-1DB\s0 maintains all the header information necessary
to implement the \s-1NNTP OVER/XOVER\s0 commands and non-search
endpoints of the \s-1PSGI UI.\s0
.PP
Xapian has become completely optional for v2 (as it is for v1), but
SQLite remains required for v2. SQLite turns out to be powerful
enough to maintain overview information. Most of the \s-1PSGI\s0 and all
of the \s-1NNTP\s0 functionality is possible with only SQLite in addition
to git.
.PP
The overview \s-1DB\s0 was an instrumental piece in maintaining near
constant-time read performance on a dataset 2\-3 times larger
than \s-1LKML\s0 history as of 2018.
.PP
\fI\s-1GHOST MESSAGES\s0\fR
.IX Subsection "GHOST MESSAGES"
.PP
The overview \s-1DB\s0 also includes references to \*(L"ghost\*(R" messages,
or messages which have replies but have not been seen by us.
Thus it is expected to have more rows than the \*(L"msgmap\*(R" \s-1DB\s0
described below.
.SS "msgmap.sqlite3"
.IX Subsection "msgmap.sqlite3"
The SQLite msgmap \s-1DB\s0 is unchanged from v1, but it is now at the
top-level of the directory.
.SH "OBJECT IDENTIFIERS"
.IX Header "OBJECT IDENTIFIERS"
There are three distinct type of identifiers. content_hash is the
new one for v2 and should make message removal and deduplication
easier. object_id and Message-ID are already known.
.IP "object_id" 4
.IX Item "object_id"
The blob identifier git uses (currently \s-1SHA\-1\s0). No need to
publicly expose this outside of normal git ops (cloning) and
there's no need to make this searchable. As with v1 of
public-inbox, this is stored as part of the Xapian document so
expensive name lookups can be avoided for document retrieval.
.IP "Message-ID" 4
.IX Item "Message-ID"
The email header; duplicates allowed for archival purposes.
This remains a searchable field in Xapian. Note: it's possible
for emails to have multiple Message-ID headers (and \fBgit\-send\-email\fR\|(1)
had that bug for a bit); so we take all of them into account.
In case of conflicts detected by content_hash below, we generate a new
Message-ID based on content_hash; if the generated Message-ID still
conflicts, a random one is generated.
.IP "content_hash" 4
.IX Item "content_hash"
A hash of relevant headers and raw body content for
purging of unwanted content. This is not stored anywhere,
but always calculated on-the-fly.
.Sp
For now, the relevant headers are:
.Sp
.Vb 1
\& Subject, From, Date, References, In\-Reply\-To, To, Cc
.Ve
.Sp
Received, List-Id, and similar headers are \s-1NOT\s0 part of content_hash as
they differ across lists and we will want removal to be able to cross
lists.
.Sp
The textual parts of the body are decoded, \s-1CRLF\s0 normalized to
\&\s-1LF,\s0 and trailing whitespace stripped. Notably, hashing the
raw body risks being broken by list signatures; but we can use
filters (e.g. PublicInbox::Filter::Vger) to clean the body for
imports.
.Sp
content_hash is \s-1SHA\-256\s0 for now; but can be changed at any time
without making \s-1DB\s0 changes.
.SH "LOCKING"
.IX Header "LOCKING"
\&\fBflock\fR\|(2) locking exclusively locks the empty inbox.lock file
for all non-atomic operations.
.SH "HEADERS"
.IX Header "HEADERS"
Same handling as with v1, except the Message-ID header will
be generated if not provided or conflicting. \*(L"Bytes\*(R", \*(L"Lines\*(R"
and \*(L"Content-Length\*(R" headers are stripped and not allowed, they
can interfere with further processing.
.PP
The \*(L"Status\*(R" mbox header is also stripped as that header makes
no sense in a public archive.
.SH "THANKS"
.IX Header "THANKS"
Thanks to the Linux Foundation for sponsoring the development
and testing of the v2 format.
.SH "COPYRIGHT"
.IX Header "COPYRIGHT"
Copyright 2018\-2021 all contributors
.PP
License: \s-1AGPL\-3.0+\s0
.SH "SEE ALSO"
.IX Header "SEE ALSO"
\&\fBgitrepository\-layout\fR\|(5), \fBpublic\-inbox\-v1\-format\fR\|(5)