.\" Automatically generated by Pod::Man 4.14 (Pod::Simple 3.40) .\" .\" Standard preamble: .\" ======================================================================== .de Sp \" Vertical space (when we can't use .PP) .if t .sp .5v .if n .sp .. .de Vb \" Begin verbatim text .ft CW .nf .ne \\$1 .. .de Ve \" End verbatim text .ft R .fi .. .\" Set up some character translations and predefined strings. \*(-- will .\" give an unbreakable dash, \*(PI will give pi, \*(L" will give a left .\" double quote, and \*(R" will give a right double quote. \*(C+ will .\" give a nicer C++. Capital omega is used to do unbreakable dashes and .\" therefore won't be available. \*(C` and \*(C' expand to `' in nroff, .\" nothing in troff, for use with C<>. .tr \(*W- .ds C+ C\v'-.1v'\h'-1p'\s-2+\h'-1p'+\s0\v'.1v'\h'-1p' .ie n \{\ . ds -- \(*W- . ds PI pi . if (\n(.H=4u)&(1m=24u) .ds -- \(*W\h'-12u'\(*W\h'-12u'-\" diablo 10 pitch . if (\n(.H=4u)&(1m=20u) .ds -- \(*W\h'-12u'\(*W\h'-8u'-\" diablo 12 pitch . ds L" "" . ds R" "" . ds C` "" . ds C' "" 'br\} .el\{\ . ds -- \|\(em\| . ds PI \(*p . ds L" `` . ds R" '' . ds C` . ds C' 'br\} .\" .\" Escape single quotes in literal strings from groff's Unicode transform. .ie \n(.g .ds Aq \(aq .el .ds Aq ' .\" .\" If the F register is >0, we'll generate index entries on stderr for .\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index .\" entries marked with X<> in POD. Of course, you'll have to process the .\" output yourself in some meaningful fashion. .\" .\" Avoid warning from groff about undefined register 'F'. .de IX .. .nr rF 0 .if \n(.g .if rF .nr rF 1 .if (\n(rF:(\n(.g==0)) \{\ . if \nF \{\ . de IX . tm Index:\\$1\t\\n%\t"\\$2" .. . if !\nF==2 \{\ . nr % 0 . nr F 2 . \} . \} .\} .rr rF .\" .\" Accent mark definitions (@(#)ms.acc 1.5 88/02/08 SMI; from UCB 4.2). .\" Fear. Run. Save yourself. No user-serviceable parts. . \" fudge factors for nroff and troff .if n \{\ . ds #H 0 . ds #V .8m . ds #F .3m . ds #[ \f1 . ds #] \fP .\} .if t \{\ . ds #H ((1u-(\\\\n(.fu%2u))*.13m) . ds #V .6m . ds #F 0 . ds #[ \& . ds #] \& .\} . \" simple accents for nroff and troff .if n \{\ . ds ' \& . ds ` \& . ds ^ \& . ds , \& . ds ~ ~ . ds / .\} .if t \{\ . ds ' \\k:\h'-(\\n(.wu*8/10-\*(#H)'\'\h"|\\n:u" . ds ` \\k:\h'-(\\n(.wu*8/10-\*(#H)'\`\h'|\\n:u' . ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'^\h'|\\n:u' . ds , \\k:\h'-(\\n(.wu*8/10)',\h'|\\n:u' . ds ~ \\k:\h'-(\\n(.wu-\*(#H-.1m)'~\h'|\\n:u' . ds / \\k:\h'-(\\n(.wu*8/10-\*(#H)'\z\(sl\h'|\\n:u' .\} . \" troff and (daisy-wheel) nroff accents .ds : \\k:\h'-(\\n(.wu*8/10-\*(#H+.1m+\*(#F)'\v'-\*(#V'\z.\h'.2m+\*(#F'.\h'|\\n:u'\v'\*(#V' .ds 8 \h'\*(#H'\(*b\h'-\*(#H' .ds o \\k:\h'-(\\n(.wu+\w'\(de'u-\*(#H)/2u'\v'-.3n'\*(#[\z\(de\v'.3n'\h'|\\n:u'\*(#] .ds d- \h'\*(#H'\(pd\h'-\w'~'u'\v'-.25m'\f2\(hy\fP\v'.25m'\h'-\*(#H' .ds D- D\\k:\h'-\w'D'u'\v'-.11m'\z\(hy\v'.11m'\h'|\\n:u' .ds th \*(#[\v'.3m'\s+1I\s-1\v'-.3m'\h'-(\w'I'u*2/3)'\s-1o\s+1\*(#] .ds Th \*(#[\s+2I\s-2\h'-\w'I'u*3/5'\v'-.3m'o\v'.3m'\*(#] .ds ae a\h'-(\w'a'u*4/10)'e .ds Ae A\h'-(\w'A'u*4/10)'E . \" corrections for vroff .if v .ds ~ \\k:\h'-(\\n(.wu*9/10-\*(#H)'\s-2\u~\d\s+2\h'|\\n:u' .if v .ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'\v'-.4m'^\v'.4m'\h'|\\n:u' . \" for low resolution devices (crt and lpr) .if \n(.H>23 .if \n(.V>19 \ \{\ . ds : e . ds 8 ss . ds o a . ds d- d\h'-1'\(ga . ds D- D\h'-1'\(hy . ds th \o'bp' . ds Th \o'LP' . ds ae ae . ds Ae AE .\} .rm #[ #] #H #V #F C .\" ======================================================================== .\" .IX Title "PUBLIC-INBOX-V1-FORMAT 5" .TH PUBLIC-INBOX-V1-FORMAT 5 "1993-10-02" "public-inbox.git" "public-inbox user manual" .\" For nroff, turn off justification. Always turn off hyphenation; it makes .\" way too many mistakes in technical documents. .if n .ad l .nh .SH "NAME" public\-inbox\-v1\-format \- git repository and tree description (aka "ssoma") .SH "DESCRIPTION" .IX Header "DESCRIPTION" \&\s-1WARNING:\s0 this does \s-1NOT\s0 describe the scalable v2 format used by public-inbox. Use of ssoma is not recommended for new installations due to scalability problems. .PP ssoma uses a git repository to store each email as a git blob. The tree filename of the blob is based on the \s-1SHA1\s0 hexdigest of the first Message-ID header. A commit is made for each message delivered. The commit \s-1SHA\-1\s0 identifier is used by ssoma clients to track synchronization state. .SH "PATHNAMES IN TREES" .IX Header "PATHNAMES IN TREES" A Message-ID may be extremely long and also contain slashes, so using them as a path name is challenging. Instead we use the \s-1SHA\-1\s0 hexdigest of the Message-ID (excluding the leading \*(L"<\*(R" and trailing \*(L">\*(R") to generate a path name. Leading and trailing white space in the Message-ID header is ignored for hashing. .PP A message with Message-ID of: <20131106023245.GA20224@dcvr.yhbt.net> .PP Would be stored as: f2/8c6cfd2b0a65f994c3e1be266105413b3d3f63 .PP Thus it is easy to look up the contents of a message matching a given a Message-ID. .SH "MESSAGE-ID CONFLICTS" .IX Header "MESSAGE-ID CONFLICTS" public-inbox v1 repositories currently do not resolve conflicting Message-IDs or messages with multiple Message-IDs. .SH "HEADERS" .IX Header "HEADERS" The Message-ID header is required. \&\*(L"Bytes\*(R", \*(L"Lines\*(R" and \*(L"Content-Length\*(R" headers are stripped and not allowed, they can interfere with further processing. When using ssoma with public-inbox-mda, the \*(L"Status\*(R" mbox header is also stripped as that header makes no sense in a public archive. .SH "LOCKING" .IX Header "LOCKING" \&\fBflock\fR\|(2) locking exclusively locks the empty \f(CW$GIT_DIR\fR/ssoma.lock file for all non-atomic operations. .SH "EXAMPLE INPUT FLOW (SERVER-SIDE MDA)" .IX Header "EXAMPLE INPUT FLOW (SERVER-SIDE MDA)" 1. Message is delivered to a mail transport agent (\s-1MTA\s0) .PP 1a. (optional) reject/discard spam, this should run before ssoma-mda .PP 1b. (optional) reject/strip unwanted attachments .PP ssoma-mda handles all steps once invoked. .PP 2. Mail transport agent invokes ssoma-mda .PP 3. reads message via stdin, extracting Message-ID .PP 4. acquires exclusive flock lock on \f(CW$GIT_DIR\fR/ssoma.lock .PP 5. creates or updates the blob of associated 2/38 \s-1SHA\-1\s0 path .PP 6. updates the index and commits .PP 7. releases \f(CW$GIT_DIR\fR/ssoma.lock .PP ssoma-mda can also be used as an \fBinotify\fR\|(7) trigger to monitor maildirs, and the ability to monitor \s-1IMAP\s0 mailboxes using \s-1IDLE\s0 will be available in the future. .SH "GIT REPOSITORIES (SERVERS)" .IX Header "GIT REPOSITORIES (SERVERS)" ssoma uses bare git repositories on both servers and clients. .PP Using the \fBgit\-init\fR\|(1) command with \-\-bare is the recommend method of creating a git repository on a server: .PP .Vb 1 \& git init \-\-bare /path/to/wherever/you/want.git .Ve .PP There are no standardized paths for servers, administrators make all the choices regarding git repository locations. .PP Special files in \f(CW$GIT_DIR\fR on the server: .ie n .IP "$GIT_DIR/ssoma.lock" 4 .el .IP "\f(CW$GIT_DIR\fR/ssoma.lock" 4 .IX Item "$GIT_DIR/ssoma.lock" An empty file for \fBflock\fR\|(2) locking. This is necessary to ensure the index and commits are updated consistently and multiple processes running \s-1MDA\s0 do not step on each other. .ie n .IP "$GIT_DIR/public\-inbox/msgmap.sqlite3" 4 .el .IP "\f(CW$GIT_DIR\fR/public\-inbox/msgmap.sqlite3" 4 .IX Item "$GIT_DIR/public-inbox/msgmap.sqlite3" SQLite3 database maintaining a stable mapping of Message-IDs to \s-1NNTP\s0 article numbers. Used by \fBpublic\-inbox\-nntpd\fR\|(1) and created and updated by \fBpublic\-inbox\-index\fR\|(1). .Sp Users of the PublicInbox::WWW interface will find it useful for attempting recovery from copy-paste truncations of URLs containing long Message-IDs. .Sp Automatically updated by \fBpublic\-inbox\-mda\fR\|(1), \&\fBpublic\-inbox\-learn\fR\|(1) and \fBpublic\-inbox\-watch\fR\|(1). .Sp Losing or damaging this file will cause synchronization problems for \&\s-1NNTP\s0 clients. This file is expected to be stable and require no updates to its schema. .Sp Requires DBD::SQLite. .ie n .IP "$GIT_DIR/public\-inbox/xapian$N/" 4 .el .IP "\f(CW$GIT_DIR\fR/public\-inbox/xapian$N/" 4 .IX Item "$GIT_DIR/public-inbox/xapian$N/" Xapian database for search indices in the \s-1PSGI\s0 web \s-1UI.\s0 .Sp \&\f(CW$N\fR is the value of PublicInbox::Search::SCHEMA_VERSION, and installations may have parallel versions on disk during upgrades or to roll-back upgrades. .Sp This is created and updated by \fBpublic\-inbox\-index\fR\|(1). .Sp Automatically updated by \fBpublic\-inbox\-mda\fR\|(1), \&\fBpublic\-inbox\-learn\fR\|(1) and \fBpublic\-inbox\-watch\fR\|(1). .Sp This directory can always be regenerated with \fBpublic\-inbox\-index\fR\|(1). If lost or damaged, there is no need to back it up unless the CPU/memory cost of regenerating it outweighs the storage/transfer cost. .Sp Since \s-1SCHEMA_VERSION 15\s0 and the development of the v2 format, the \*(L"overview\*(R" \s-1DB\s0 also exists in the xapian directory for v1 repositories. See \*(L"\s-1OVERVIEW DB\*(R"\s0 in \fBpublic\-inbox\-v2\-format\fR\|(5) .Sp Our use of the \*(L"\s-1OVERVIEW DB\*(R"\s0 requires Xapian document IDs to remain stable. Using \fBpublic\-inbox\-compact\fR\|(1) and \&\fBpublic\-inbox\-xcpdb\fR\|(1) wrappers are recommended over tools provided by Xapian. .Sp This directory is large, often two to three times the size of the objects stored in a packed git repository. .ie n .IP "$GIT_DIR/ssoma.index" 4 .el .IP "\f(CW$GIT_DIR\fR/ssoma.index" 4 .IX Item "$GIT_DIR/ssoma.index" This file is no longer used or created by public-inbox, but it is updated if it exists to remain compatible with ssoma installations. .Sp A git index file used for \s-1MDA\s0 updates. The normal git index (in \&\f(CW$GIT_DIR\fR/index) is not used at all as there is typically no working tree. .PP Each client \f(CW$GIT_DIR\fR may have multiple mbox/maildir/command targets. It is possible for a client to extract the mail stored in the git repository to multiple mboxes for compatibility with a variety of different tools. .SH "CAVEATS" .IX Header "CAVEATS" It is \s-1NOT\s0 recommended to check out the working directory of a git. there may be many files. .PP It is impossible to completely expunge messages, even spam, as git retains full history. Projects may (with adequate notice) cycle to new repositories/branches with history cleaned up via \fBgit\-filter\-repo\fR\|(1) or \fBgit\-filter\-branch\fR\|(1). This is up to the administrators. .SH "COPYRIGHT" .IX Header "COPYRIGHT" Copyright 2013\-2021 all contributors .PP License: \s-1AGPL\-3.0+\s0 .SH "SEE ALSO" .IX Header "SEE ALSO" \&\fBgitrepository\-layout\fR\|(5), \fBssoma\fR\|(1)