.\" Automatically generated by Pod::Man 4.14 (Pod::Simple 3.43) .\" .\" Standard preamble: .\" ======================================================================== .de Sp \" Vertical space (when we can't use .PP) .if t .sp .5v .if n .sp .. .de Vb \" Begin verbatim text .ft CW .nf .ne \\$1 .. .de Ve \" End verbatim text .ft R .fi .. .\" Set up some character translations and predefined strings. \*(-- will .\" give an unbreakable dash, \*(PI will give pi, \*(L" will give a left .\" double quote, and \*(R" will give a right double quote. \*(C+ will .\" give a nicer C++. Capital omega is used to do unbreakable dashes and .\" therefore won't be available. \*(C` and \*(C' expand to `' in nroff, .\" nothing in troff, for use with C<>. .tr \(*W- .ds C+ C\v'-.1v'\h'-1p'\s-2+\h'-1p'+\s0\v'.1v'\h'-1p' .ie n \{\ . ds -- \(*W- . ds PI pi . if (\n(.H=4u)&(1m=24u) .ds -- \(*W\h'-12u'\(*W\h'-12u'-\" diablo 10 pitch . if (\n(.H=4u)&(1m=20u) .ds -- \(*W\h'-12u'\(*W\h'-8u'-\" diablo 12 pitch . ds L" "" . ds R" "" . ds C` "" . ds C' "" 'br\} .el\{\ . ds -- \|\(em\| . ds PI \(*p . ds L" `` . ds R" '' . ds C` . ds C' 'br\} .\" .\" Escape single quotes in literal strings from groff's Unicode transform. .ie \n(.g .ds Aq \(aq .el .ds Aq ' .\" .\" If the F register is >0, we'll generate index entries on stderr for .\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index .\" entries marked with X<> in POD. Of course, you'll have to process the .\" output yourself in some meaningful fashion. .\" .\" Avoid warning from groff about undefined register 'F'. .de IX .. .nr rF 0 .if \n(.g .if rF .nr rF 1 .if (\n(rF:(\n(.g==0)) \{\ . if \nF \{\ . de IX . tm Index:\\$1\t\\n%\t"\\$2" .. . if !\nF==2 \{\ . nr % 0 . nr F 2 . \} . \} .\} .rr rF .\" .\" Accent mark definitions (@(#)ms.acc 1.5 88/02/08 SMI; from UCB 4.2). .\" Fear. Run. Save yourself. No user-serviceable parts. . \" fudge factors for nroff and troff .if n \{\ . ds #H 0 . ds #V .8m . ds #F .3m . ds #[ \f1 . ds #] \fP .\} .if t \{\ . ds #H ((1u-(\\\\n(.fu%2u))*.13m) . ds #V .6m . ds #F 0 . ds #[ \& . ds #] \& .\} . \" simple accents for nroff and troff .if n \{\ . ds ' \& . ds ` \& . ds ^ \& . ds , \& . ds ~ ~ . ds / .\} .if t \{\ . ds ' \\k:\h'-(\\n(.wu*8/10-\*(#H)'\'\h"|\\n:u" . ds ` \\k:\h'-(\\n(.wu*8/10-\*(#H)'\`\h'|\\n:u' . ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'^\h'|\\n:u' . ds , \\k:\h'-(\\n(.wu*8/10)',\h'|\\n:u' . ds ~ \\k:\h'-(\\n(.wu-\*(#H-.1m)'~\h'|\\n:u' . ds / \\k:\h'-(\\n(.wu*8/10-\*(#H)'\z\(sl\h'|\\n:u' .\} . \" troff and (daisy-wheel) nroff accents .ds : \\k:\h'-(\\n(.wu*8/10-\*(#H+.1m+\*(#F)'\v'-\*(#V'\z.\h'.2m+\*(#F'.\h'|\\n:u'\v'\*(#V' .ds 8 \h'\*(#H'\(*b\h'-\*(#H' .ds o \\k:\h'-(\\n(.wu+\w'\(de'u-\*(#H)/2u'\v'-.3n'\*(#[\z\(de\v'.3n'\h'|\\n:u'\*(#] .ds d- \h'\*(#H'\(pd\h'-\w'~'u'\v'-.25m'\f2\(hy\fP\v'.25m'\h'-\*(#H' .ds D- D\\k:\h'-\w'D'u'\v'-.11m'\z\(hy\v'.11m'\h'|\\n:u' .ds th \*(#[\v'.3m'\s+1I\s-1\v'-.3m'\h'-(\w'I'u*2/3)'\s-1o\s+1\*(#] .ds Th \*(#[\s+2I\s-2\h'-\w'I'u*3/5'\v'-.3m'o\v'.3m'\*(#] .ds ae a\h'-(\w'a'u*4/10)'e .ds Ae A\h'-(\w'A'u*4/10)'E . \" corrections for vroff .if v .ds ~ \\k:\h'-(\\n(.wu*9/10-\*(#H)'\s-2\u~\d\s+2\h'|\\n:u' .if v .ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'\v'-.4m'^\v'.4m'\h'|\\n:u' . \" for low resolution devices (crt and lpr) .if \n(.H>23 .if \n(.V>19 \ \{\ . ds : e . ds 8 ss . ds o a . ds d- d\h'-1'\(ga . ds D- D\h'-1'\(hy . ds th \o'bp' . ds Th \o'LP' . ds ae ae . ds Ae AE .\} .rm #[ #] #H #V #F C .\" ======================================================================== .\" .IX Title "PUBLIC-INBOX-TUNING 7" .TH PUBLIC-INBOX-TUNING 7 "1993-10-02" "public-inbox.git" "public-inbox user manual" .\" For nroff, turn off justification. Always turn off hyphenation; it makes .\" way too many mistakes in technical documents. .if n .ad l .nh .SH "NAME" public\-inbox\-tuning \- tuning public\-inbox .SH "DESCRIPTION" .IX Header "DESCRIPTION" public-inbox intends to support a wide variety of hardware. While we strive to provide the best out-of-the-box performance possible, tuning knobs are an unfortunate necessity in some cases. .IP "1." 4 New inboxes: public-inbox-init \-V2 .IP "2." 4 Optional Inline::C use .IP "3." 4 Performance on rotational hard disk drives .IP "4." 4 Btrfs (and possibly other copy-on-write filesystems) .IP "5." 4 Performance on solid state drives .IP "6." 4 Read-only daemons .IP "7." 4 Other \s-1OS\s0 tuning knobs .IP "8." 4 Scalability to many inboxes .SS "New inboxes: public-inbox-init \-V2" .IX Subsection "New inboxes: public-inbox-init -V2" If you're starting a new inbox (and not mirroring an existing one), the \-V2 requires DBD::SQLite, but is orders of magnitude more scalable than the original \f(CW\*(C`\-V1\*(C'\fR format. .SS "Optional Inline::C use" .IX Subsection "Optional Inline::C use" Our optional use of Inline::C speeds up subprocess spawning from large daemon processes. .PP To enable Inline::C, either set the \f(CW\*(C`PERL_INLINE_DIRECTORY\*(C'\fR environment variable to point to a writable directory, or create \&\f(CW\*(C`~/.cache/public\-inbox/inline\-c\*(C'\fR for any user(s) running public-inbox processes. .PP If libgit2 development files are installed and Inline::C is enabled (described above), per-inbox \f(CW\*(C`git cat\-file \-\-batch\*(C'\fR processes are replaced with a single \fBperl\fR\|(1) process running \&\f(CW\*(C`PublicInbox::Gcf2::loop\*(C'\fR in read-only daemons. libgit2 use will be available in public-inbox 1.7.0+ .PP More (optional) Inline::C use will be introduced in the future to lower memory use and improve scalability. .PP Note: Inline::C is required for \fBlei\fR\|(1), but not public\-inbox\-* .SS "Performance on rotational hard disk drives" .IX Subsection "Performance on rotational hard disk drives" Random I/O performance is poor on rotational HDDs. Xapian indexing performance degrades significantly as DBs grow larger than available \&\s-1RAM.\s0 Attempts to parallelize random I/O on HDDs leads to pathological slowdowns as inboxes grow. .PP While \f(CW\*(C`\-V2\*(C'\fR introduced Xapian shards as a parallelization mechanism for SSDs; enabling \f(CW\*(C`publicInbox.indexSequentialShard\*(C'\fR repurposes sharding as mechanism to reduce the kernel page cache footprint when indexing on HDDs. .PP Initializing a mirror with a high \f(CW\*(C`\-\-jobs\*(C'\fR count to create more shards (in \f(CW\*(C`\-V2\*(C'\fR inboxes) will keep each shard smaller and reduce its kernel page cache footprint. Keep in mind excessive sharding imposes a performance penalty for read-only queries. .PP Users with large amounts of \s-1RAM\s0 are advised to set a large value for \f(CW\*(C`publicinbox.indexBatchSize\*(C'\fR as documented in \&\fBpublic\-inbox\-index\fR\|(1). .PP \&\f(CW\*(C`dm\-crypt\*(C'\fR users on Linux 4.0+ are advised to try the \&\f(CW\*(C`\-\-perf\-same_cpu_crypt\*(C'\fR \f(CW\*(C`\-\-perf\-submit_from_crypt_cpus\*(C'\fR switches of \fBcryptsetup\fR\|(8) to reduce I/O contention from kernel workqueue threads. .SS "Btrfs (and possibly other copy-on-write filesystems)" .IX Subsection "Btrfs (and possibly other copy-on-write filesystems)" \&\fBbtrfs\fR\|(5) performance degrades from fragmentation when using large databases and random writes. The Xapian + SQLite indices used by public-inbox are no exception to that. .PP public-inbox 1.6.0+ disables copy-on-write (CoW) on Xapian and SQLite indices on btrfs to achieve acceptable performance (even on \s-1SSD\s0). Disabling copy-on-write also disables checksumming, thus \f(CW\*(C`raid1\*(C'\fR (or higher) configurations may be corrupt after unsafe shutdowns. .PP Fortunately, these SQLite and Xapian indices are designed to recoverable from git if missing. .PP Disabling CoW does not prevent all fragmentation. Large values of \f(CW\*(C`publicInbox.indexBatchSize\*(C'\fR also limit fragmentation during the initial index. .PP Avoid snapshotting subvolumes containing Xapian and/or SQLite indices. Snapshots use CoW despite our efforts to disable it, resulting in fragmentation. .PP \&\fBfilefrag\fR\|(8) can be used to monitor fragmentation, and \&\f(CW\*(C`btrfs filesystem defragment \-fr $INBOX_DIR\*(C'\fR may be necessary. .PP Large filesystems benefit significantly from the \f(CW\*(C`space_cache=v2\*(C'\fR mount option documented in \fBbtrfs\fR\|(5). .PP Older, non-CoW filesystems are generally work well out-of-the-box for our Xapian and SQLite indices. .SS "Performance on solid state drives" .IX Subsection "Performance on solid state drives" While \s-1SSD\s0 read performance is generally good, \s-1SSD\s0 write performance degrades as the drive ages and/or gets full. Issuing \f(CW\*(C`TRIM\*(C'\fR commands via \fBfstrim\fR\|(8) or similar is required to sustain write performance. .PP Users of the Flash-Friendly File System F2FS may benefit from optimizations found in SQLite 3.21.0+. Benchmarks are greatly appreciated. .SS "Read-only daemons" .IX Subsection "Read-only daemons" \&\fBpublic\-inbox\-httpd\fR\|(1), \fBpublic\-inbox\-imapd\fR\|(1), and \&\fBpublic\-inbox\-nntpd\fR\|(1) are all designed for C10K (or higher) levels of concurrency from a single process. \s-1SMP\s0 systems may use \f(CW\*(C`\-\-worker\-processes=NUM\*(C'\fR as documented in \fBpublic\-inbox\-daemon\fR\|(8) for parallelism. .PP The open file descriptor limit (\f(CW\*(C`RLIMIT_NOFILE\*(C'\fR, \f(CW\*(C`ulimit \-n\*(C'\fR in \fBsh\fR\|(1), \&\f(CW\*(C`LimitNOFILE=\*(C'\fR in \fBsystemd.exec\fR\|(5)) may need to be raised to accommodate many concurrent clients. .PP Transport Layer Security (\s-1IMAPS, NNTPS,\s0 or via \s-1STARTTLS\s0) significantly increases memory use of client sockets, sure to account for that in capacity planning. .SS "Other \s-1OS\s0 tuning knobs" .IX Subsection "Other OS tuning knobs" Linux users: the \f(CW\*(C`sys.vm.max_map_count\*(C'\fR sysctl may need to be increased if handling thousands of inboxes (with \fBpublic\-inbox\-extindex\fR\|(1)) to avoid out-of-memory errors from git. .PP Other OSes may have similar tuning knobs (patches appreciated). .SS "Scalability to many inboxes" .IX Subsection "Scalability to many inboxes" \&\fBpublic\-inbox\-extindex\fR\|(1) allows any number of public-inboxes to share the same Xapian indices. .PP git 2.33+ startup time is orders-of-magnitude faster and uses less memory when dealing with thousands of alternates required for thousands of inboxes with \fBpublic\-inbox\-extindex\fR\|(1). .PP Frequent packing (via \fBgit\-gc\fR\|(1)) both improves performance and reduces the need to increase \f(CW\*(C`sys.vm.max_map_count\*(C'\fR. .SH "CONTACT" .IX Header "CONTACT" Feedback encouraged via plain-text mail to .PP Information for *BSDs and non-traditional filesystems especially welcome. .PP Our archives are hosted at , , and other places .SH "COPYRIGHT" .IX Header "COPYRIGHT" Copyright all contributors .PP License: \s-1AGPL\-3.0+\s0