.\" Man page generated from reStructuredText.
.
.
.nr rst2man-indent-level 0
.
.de1 rstReportMargin
\\$1 \\n[an-margin]
level \\n[rst2man-indent-level]
level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
-
\\n[rst2man-indent0]
\\n[rst2man-indent1]
\\n[rst2man-indent2]
..
.de1 INDENT
.\" .rstReportMargin pre:
. RS \\$1
. nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin]
. nr rst2man-indent-level +1
.\" .rstReportMargin post:
..
.de UNINDENT
. RE
.\" indent \\n[an-margin]
.\" old: \\n[rst2man-indent\\n[rst2man-indent-level]]
.nr rst2man-indent-level -1
.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
..
.TH "SQUATTER" "8" "Mar 13, 2024" "3.8.2" "Cyrus IMAP"
.SH NAME
squatter \- Cyrus IMAP documentation
.sp
Create SQUAT and Xapian indexes for mailboxes
.SH SYNOPSIS
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
general:
\fBsquatter\fP [ \fB\-C\fP \fIconfig\-file\fP ] [\fBmode\fP] [\fBoptions\fP] [\fBsource\fP]

i.e.:
\fBsquatter\fP [ \fB\-C\fP \fIconfig\-file\fP ] [ \fB\-v\fP ] [ \fB\-a\fP ] [ \fB\-S\fP \fIseconds\fP ] [ \fB\-Z\fP ]
\fBsquatter\fP [ \fB\-C\fP \fIconfig\-file\fP ] [ \fB\-v\fP ] [ \fB\-a\fP ] [ \fB\-i\fP ] [ \fB\-N\fP \fIname\fP ] [ \fB\-S\fP \fIseconds\fP ] [ \fB\-r\fP ] [ \fB\-Z\fP ] \fImailbox\fP\&...
\fBsquatter\fP [ \fB\-C\fP \fIconfig\-file\fP ] [ \fB\-v\fP ] [ \fB\-a\fP ] [ \fB\-i\fP ] [ \fB\-N\fP \fIname\fP ] [ \fB\-S\fP \fIseconds\fP ] [ \fB\-r\fP ] [ \fB\-Z\fP ] \fB\-u\fP \fIuser\fP\&...
\fBsquatter\fP [ \fB\-C\fP \fIconfig\-file\fP ] [ \fB\-v\fP ] [ \fB\-a\fP ] \fB\-R\fP [ \fB\-n\fP \fIchannel\fP ] [ \fB\-d\fP ] [ \fB\-S\fP \fIseconds\fP ] [ \fB\-Z\fP ]
\fBsquatter\fP [ \fB\-C\fP \fIconfig\-file\fP ] [ \fB\-v\fP ] [ \fB\-a\fP ] \fB\-f\fP \fIsynclogfile\fP [ \fB\-S\fP \fIseconds\fP ] [ \fB\-Z\fP ]
\fBsquatter\fP [ \fB\-C\fP \fIconfig\-file\fP ] [ \fB\-v\fP ] \fB\-t\fP \fIsrctier(s)\fP\&... \fB\-z\fP \fIdesttier\fP [ \fB\-B\fP ] [ \fB\-F\fP ] [ \fB\-U\fP ] [ \fB\-T\fP \fIreindextiers\fP ] [ \fB\-X\fP ] [ \fB\-o\fP ] [ \fB\-S\fP \fIseconds\fP ] [ \fB\-u\fP \fIuser\fP\&... ]
.ft P
.fi
.UNINDENT
.UNINDENT
.SH DESCRIPTION
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
The name \(dq\fBsquatter\fP\(dq once referred both to the SQUAT indexing
engine and to the command used to create indexes.  Now that Cyrus
supports more than one index type \-\- SQUAT and Xapian, as of this
writing \-\- the name \(dq\fBsquatter\fP\(dq refers to the command used to
control index creation.  The terms \(dqSQUAT\(dq or \(dqSQUAT index(es)\(dq
refers to the indexes used by the older SQUAT indexing engine.
Post v3 the \fIsearch_engine\fP setting in \fIimapd.conf\fP determines
which search engine is used.
.UNINDENT
.UNINDENT
.sp
\fBsquatter\fP creates a new text index for one or more IMAP mailboxes.
The index is a unified index of all of the header and body text
of each message in a given mailbox.  This index is used to significantly
reduce IMAP SEARCH times on a mailbox.
.sp
\fBmode\fP is one of indexer, search, rolling, synclog, compact or audit.
.sp
By default, \fBsquatter\fP creates an index of ALL messages in the
mailbox, not just those since the last time that it was run.  The
\fB\-i\fP option is used to select incremental updates.  Any messages
appended to the mailbox after \fBsquatter\fP is run, will NOT be included
in the index.  To include new messages in the index, \fBsquatter\fP must
be run again, or on a regular basis via crontab, an entry in the EVENTS
section of \fI\%cyrus.conf(5)\fP or use \fIrolling\fP mode (\fB\-R\fP).
.sp
In the first synopsis, \fBsquatter\fP indexes all mailboxes.
.sp
In the second synopsis, \fBsquatter\fP indexes the specified mailbox(es).
The mailboxes are space\-separated.
.sp
In the third synopsis, \fBsquatter\fP indexes the specified user(s)
mailbox(es).
.sp
For the latter two index modes (mailbox, user) one
may optionally specify \fB\-r\fP to recurse from the specified start, or
\fB\-a\fP to limit action only to mailboxes which have the shared
\fI/vendor/cmu/cyrus\-imapd/squat\fP annotation set to \(dqtrue\(dq.
.sp
In the fourth synopsis, \fBsquatter\fP runs in rolling mode.  In this
mode \fBsquatter\fP backgrounds itself and runs as a daemon (unless
\fB\-d\fP is set), listening to a sync log channel chosen using the \fB\-n\fP
option, and set up using the \fIsync_log_channels\fP setting in
\fI\%imapd.conf(5)\fP\&.  Very soon after messages are delivered or
uploaded to mailboxes \fBsquatter\fP will incrementally index the
affected mailbox (see notes, below).
.sp
In the fifth synopsis, \fBsquatter\fP reads a single sync log file and
performs incremental indexing on the mailbox(es) listed therein.  This
is sometimes useful for cleaning up after problems with rolling mode.
.sp
In the sixth synopsis, \fBsquatter\fP will compact indices from
\fIsrctier(s)\fP to \fIdesttier\fP, optionally reindexing (\fB\-X\fP) or filtering
expunged records (\fB\-F\fP) in the process.  The optional \fB\-T\fP flag may be
used to specify members of srctiers which must be reindexed.  These files are
eventually copied with \fIrsync \-a\fP and then removed by \fIrm\fP\&.
\fIrsync\fP can increase the load average of the system, especially when the
temporary directory is on \fItmpfs\fP\&.  To throttle \fIrsync\fP it is possible to
modify the call in \fIimap/search_xapian.c\fP and pass \fI\-\e\-bwlimit=<number>\fP as further
parameter.  The \fB\-o\fP flag may be used to direct that a single index be
copied, rather than compacted, from \fIsrctier\fP to \fIdesttier\fP\&.  The \fB\-u\fP flag
may be used to restrict operation to the specified user(s).
.sp
For all modes, the \fB\-S\fP option may be specified, causing \fBsquatter\fP to
pause \fIseconds\fP seconds after each mailbox, to smooth loads.
.sp
When using the Xapian engine the \fB\-Z\fP option may be specified, for
the indexing modes.  This tells \fBsquatter\fP to consult the Xapian
internally indexed GUIDs, rather than relying on what\(aqs stored in
\fIcyrus.indexed.db\fP, allowing for recovery from broken
\fIcyrus.indexed.db\fP at the sacrifice of efficiency.
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
Incremental updates are very inefficient with the SQUAT search
engine.  If using SQUAT for large and active mailboxes, you should
run \fBsquatter\fP periodically as an EVENT in \fBcyrus.conf(5)\fP\&.
.UNINDENT
.UNINDENT
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
Messages and mailboxes that have not been indexed CAN still be
SEARCHed, just not as quickly as those with an index.
.UNINDENT
.UNINDENT
.sp
\fBsquatter\fP reads its configuration options out of the \fI\%imapd.conf(5)\fP file unless specified otherwise by \fB\-C\fP\&.
.SH OPTIONS
.INDENT 0.0
.TP
.B \-C config\-file
Use the specified configuration file \fIconfig\-file\fP rather than the default \fI\%imapd.conf(5)\fP\&.
.UNINDENT
.INDENT 0.0
.TP
.B \-a, \-\-squat\-annot
Only create indexes for mailboxes which have the shared
\fI/vendor/cmu/cyrus\-imapd/squat\fP annotation set to \(dqtrue\(dq.
.sp
The value of the \fI/vendor/cmu/cyrus\-imapd/squat\fP annotation is
inherited by all children of the given mailbox, so an entire
mailbox tree can be indexed (or not indexed) by setting a single
annotation on the root of that tree with a value of \(dqtrue\(dq (or
\(dqfalse\(dq).  If a mailbox does not have a
\fI/vendor/cmu/cyrus\-imapd/squat\fP annotation set on it (or does not
inherit one), then the mailbox is not indexed. In other words, the
implicit value of \fI/vendor/cmu/cyrus\-imapd/squat\fP is \(dqfalse\(dq.
.UNINDENT
.INDENT 0.0
.TP
.B \-A, \-\-audit
Audits the specified mailboxes (or all), reports any unindexed messages.
This feature is only available on the master branch.
.UNINDENT
.INDENT 0.0
.TP
.B \-d, \-\-nodaemon
In rolling mode, don\(aqt background and do emit log messages on
standard error.  Useful for debugging.
This feature was introduced in version 3.0.
.UNINDENT
.INDENT 0.0
.TP
.B \-B, \-\-skip\-locked
In compact mode, use non\-blocking lock to start and skip any
users who have their xapianactive file locked at the time (i.e
another reindex task)
This feature is only available on the master branch.
.UNINDENT
.INDENT 0.0
.TP
.B \-F, \-\-filter
In compact mode, filter the resulting database to only include
messages which are not expunged in mailboxes with existing
name/uidvalidity.
This feature was introduced in version 3.0.
.UNINDENT
.INDENT 0.0
.TP
.B \-f synclogfile, \-\-synclog=synclogfile
Read the \fIsynclogfile\fP and incrementally index all the mailboxes
listed therein, then exit.
This feature was introduced in version 3.0.
.UNINDENT
.INDENT 0.0
.TP
.B \-h, \-\-help
Display this usage information.
.UNINDENT
.INDENT 0.0
.TP
.B \-i, \-\-incremental
Incremental updates where indexes already exist.
.UNINDENT
.INDENT 0.0
.TP
.B \-N name, \-\-name=name
Only index mailboxes beginning with \fIname\fP while iterating through
the mailbox list derived from other options.
.UNINDENT
.INDENT 0.0
.TP
.B \-n channel, \-\-channel=channel
In rolling mode, specify the name of the sync log \fIchannel\fP that
\fBsquatter\fP will listen to.  The default is \(dqsquatter\(dq.  This
channel \fBmust\fP be defined in \fI\%imapd.conf(5)\fP before
being used.
This feature was introduced in version 3.0.
.UNINDENT
.INDENT 0.0
.TP
.B \-o, \-\-copydb
In compact mode, if only one source database is selected, just copy
it to the destination rather than compacting.
This feature was introduced in version 3.0.
.UNINDENT
.INDENT 0.0
.TP
.B \-p, \-\-allow\-partials
.INDENT 7.0
.INDENT 3.5
When indexing, allow messages to be partially indexed. This may
occur if attachment indexing is enabled but indexing failed for
one or more attachment body parts. If this flag is set, the
message is partially indexed and squatter continues. Otherwise
squatter aborts with an error. Also see \fB\-P\fP\&.
Xapian only.
This feature is only available on the master branch.
.UNINDENT
.UNINDENT
.INDENT 7.0
.TP
.B \-P, \-\-reindex\-partials
When reindexing, then attempt to reindex any partially indexed
messages (see \fB\-p\fP). Setting this flag implies \fB\-Z\fP\&.
Xapian only.
This feature is only available on the master branch.
.UNINDENT
.INDENT 7.0
.TP
.B \-L, \-\-reindex\-minlevel=level
When reindexing, index all messages that have an index level
less than level. Currently, Cyrus only supports two index levels:
A message for which attachment indexing was never attempted has
index level 1. A message that has indexed attachments, or does not
contain attachments, has index level 3. Consequently, running
squatter with minlevel set to 3 will cause it to attempt reindexing
all messages, for which attachment indexing never was attempted.
Future Cyrus versions may introduce additional levels. Setting
this flag implies \fB\-Z\fP\&.
Xapian only.
This feature is only available on the master branch.
.UNINDENT
.UNINDENT
.INDENT 0.0
.TP
.B \-R, \-\-rolling
Run in rolling mode; \fBsquatter\fP runs as a daemon listening to a
sync log channel and continuously incrementally indexing mailboxes.
See also \fB\-d\fP and \fB\-n\fP\&.
This feature was introduced in version 3.0.
.UNINDENT
.INDENT 0.0
.TP
.B \-r, \-\-recursive
Recursively create indexes for all sub\-mailboxes of the user,
mailboxes or mailbox prefixes given as arguments.
.UNINDENT
.INDENT 0.0
.TP
.B \-s delta, \-\-squat\-skip=delta
Skip mailboxes that have not been modified since last index. This is
achieved by comparing the last modification time of a mailbox to
the last time the squat index of this mailbox got updated. If the
mailbox modification time plus delta is less than the squat
index modification time, then the mailbox is skipped. The argument
value delta is defined in seconds and must be greater than or equal
to zero. The historical default delta was 60, and this remains a
good general choice, but for technical reasons it must now be
specified explicitly.
Squat only.
.UNINDENT
.INDENT 0.0
.TP
.B \-S seconds, \-\-sleep=seconds
After processing each mailbox, sleep for \(dqseconds\(dq before
continuing. Can be used to provide some load balancing.  Accepts
fractional amounts. This feature was introduced in version 3.0.
.UNINDENT
.INDENT 0.0
.TP
.B \-T reindextiers, \-\-reindex\-tier=reindextiers
In compact mode, a comma\-separated subset of the source tiers
(see \fB\-t\fP) to be reindexed.  Similar to \fB\-X\fP but allows
limiting the tiers that will be reindexed.
This feature was introduced in version 3.0.
.UNINDENT
.INDENT 0.0
.TP
.B \-t srctiers, \-\-srctier=srctiers
In compact mode, the comma\-separated source tier(s) for the compacted
indices.  At least one source tier must be specified in compact mode.
Xapian only.
This feature was introduced in version 3.0.
.UNINDENT
.INDENT 0.0
.TP
.B \-u name, \-\-user=name
Extra options refer to usernames (e.g. \fI\%foo@bar.com\fP) rather than
mailbox names.  Usernames are space\-separated.
This feature was introduced in version 3.0.
.UNINDENT
.INDENT 0.0
.TP
.B \-U, \-\-only\-upgrade
In compact mode, only compact if re\-indexing.
Xapian only.
This feature is only available on the master branch.
.UNINDENT
.INDENT 0.0
.TP
.B \-v, \-\-verbose
Increase the verbosity of progress/status messages.  Sometimes additional messages
are emitted on the terminal with this option and the messages are unconditionally sent
to syslog.  Sometimes messages are sent to syslog, only if \-v is provided.  In rolling and
synclog modes, \-vv sends even more messages to syslog.
.UNINDENT
.INDENT 0.0
.TP
.B \-X, \-\-reindex
Reindex all the messages before compacting.  This mode reads all
the lists of messages indexed by the listed tiers, and re\-indexes
them into a temporary database before compacting that into place.
Xapian only.
This feature was introduced in version 3.0.
.UNINDENT
.INDENT 0.0
.TP
.B \-z desttier, \-\-compact=desttier
In compact mode, the destination tier for the compacted indices.
This must be specified in compact mode.
Xapian only.
This feature was introduced in version 3.0.
.UNINDENT
.INDENT 0.0
.TP
.B \-Z, \-\-internalindex
When indexing messages, use the Xapian internal cyrusid rather than
referencing the ranges of already indexed messages to know if a
particular message is indexed.  Useful if the ranges get out of
sync with the actual messages (e.g. if files on a tier are lost)
Xapian only.
This feature is only available on the master branch.
.UNINDENT
.SH EXAMPLES
.sp
\fBsquatter\fP is typically deployed via entries in
\fI\%cyrus.conf(5)\fP, in either the DAEMON or EVENTS sections.
.sp
For the older SQUAT search engine, which offers poor performance in
rolling mode (\-R) we recommend triggering periodic runs via entries in
the EVENTS section, as follows:
.sp
Sample entries from the EVENTS section of \fI\%cyrus.conf(5)\fP for
periodic \fBsquatter\fP runs:
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
EVENTS {
    # reindex changed mailboxes (fulltext) approximately every three hours
    squatter1   cmd=\(dq/usr/bin/ionice \-c idle /usr/lib/cyrus/bin/squatter \-i\(dq period=180

    # reindex all mailboxes (fulltext) daily
    squattera   cmd=\(dq/usr/lib/cyrus/bin/squatter\(dq at=0117
}
.ft P
.fi
.UNINDENT
.UNINDENT
.UNINDENT
.UNINDENT
.sp
For the newer Xapian search engine, and with sufficiently fast storage,
the rolling mode (\-R) offers advantages.  Use of rolling mode requires
that \fBsquatter\fP be invoked in the DAEMON section.
.sp
Sample entries for the DAEMON section of \fI\%cyrus.conf(5)\fP for
rolling \fBsquatter\fP operation:
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
DAEMON {
  # run a rolling squatter using the default sync_log channel \(dqsquatter\(dq
  squatter cmd=\(dqsquatter \-R\(dq

  # run a rolling squatter using a specific sync_log channel
  squatter cmd=\(dqsquatter \-R \-n indexer\(dq
}
.ft P
.fi
.UNINDENT
.UNINDENT
.UNINDENT
.UNINDENT
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
When using the \fI\-R\fP rolling mode, you MUST enable sync_log
operation in \fI\%imapd.conf(5)\fP via the \fIsync_log: on\fP
setting, and MUST define a sync_log channel via the
\fIsync_log_channels:\fP setting.  If also using replication, you must
either explicitly specify your replication sync_log channel via the
\fIsync_log_channels\fP directive with a name, or specify the default
empty name with \(dq\(dq (the two\-character string U+22 U+22).  [Please
see \fI\%imapd.conf(5)\fP for details].
.UNINDENT
.UNINDENT
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
When configuring rolling search indexing on a \fBreplica\fP, one must
consider whether sync_logs will be written at all.  In this case,
please consider the setting \fIsync_log_unsuppressable_channels\fP to
ensure that the sync_log channel upon which one\(aqs squatter instance
depends will continue to be written.  See \fI\%imapd.conf(5)\fP
for details.
.UNINDENT
.UNINDENT
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
When using the Xapian search engine, you must define various
settings in \fI\%imapd.conf(5)\fP\&.  Please read all relevant
Xapian documentation in this release before using Xapian.
.UNINDENT
.UNINDENT
.sp
[NB: More examples needed]
.SH HISTORY
.sp
Support for additional search engines was added in version 3.0.
.sp
The following command\-line switches were added in version 3.0:
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
\fB\-F \-R \-X \-d \-f \-o \-u\fP
.ft P
.fi
.UNINDENT
.UNINDENT
.UNINDENT
.UNINDENT
.sp
The following command\-line settings were added in version 3.0:
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
\fB\-S\fP \fI<seconds>\fP, \fB\-T\fP \fI<directory>\fP, \fB\-f\fP \fI<synclogfile>\fP, \fB\-n\fP \fI<channel>\fP, \fB\-t\fP \fIsrctier\fP\&..., \fB\-z\fP \fIdesttier\fP
.ft P
.fi
.UNINDENT
.UNINDENT
.UNINDENT
.UNINDENT
.SH FILES
.sp
/etc/imapd.conf,
/etc/cyrus.conf
.SH SEE ALSO
.sp
\fI\%imapd.conf(5)\fP, \fI\%cyrus.conf(5)\fP
.SH AUTHOR
The Cyrus Team, Nic Bernstein (Onlight)
.SH COPYRIGHT
1993–2024, The Cyrus Team
.\" Generated by docutils manpage writer.
.