NAME¶
fetchnews - fetch news via NNTP
SYNOPSIS¶
fetchnews [
-q] [
-v] ... [
-x number]
[
-l] [
-n] [
-f] [
-P] [
-w]
DESCRIPTION¶
Leafnode is a USENET package intended for small sites, where there are
few users and little disk space, but where a large number of groups is
desired.
The design of
leafnode is intended to self-repair after problems, and to
require no manual maintenance.
Fetchnews (formerly called "fetch") is the program which
submits and retrieves new articles to or from the upstream NNTP server.
OPTIONS¶
- -q
- Be quiet. Suppresses some warnings. This flag cancels any
previously encountered -v flags.
- -v
- Be verbose. This flag cancels any previously encountered
-q flags. A maximum of four -v can be used (the more v, the
more verbose):
- -v
- Warn about non-critical errors.
- -vv
- Be verbose whenever a group command occurs.
- -vvv
- Be verbose whenever an article command occurs.
- -vvvv
- For debugging purposes. Don't expect the additional output
to make sense.
- -w
- Wait mode. This causes the XOVER update (which runs after
the article fetch phase) to run in the foreground. By default, the XOVER
update runs in the background, detached from the shell, on systems that
offer a working fork(2) call. Helpful for debugging and to avoid terminal
clobbering when fetchnews is run manually.
- -x number
- Go number articles back and try to re-fetch what's
missing. This can be useful to pull data from broken servers that have
either overview and group data out of synch or create articles out of
article number order.
- -l
- Don't use supplementary servers (-l is supposed to
mean "local"). This does speed up the operation quite a
bit.
- -n
- Do not expire newsgroup subscriptions. Technically: do not
unlink files in the interesting.groups directory.
- -f
- Erase the active.read file, thus force redownloading the
whole active file from all upstream servers. This is usually occurring
only very infrequently because it takes a long time. See
timeout_active in leafnode(8). DO NOT use this option
in cron jobs or otherwise regularly.
- -P
- Only post articles that have accumulated in the
out.going directory but do not fetch articles from upstream
servers.
EXIT STATUS¶
fetchnews will exit with code 0 if it could connect to all servers and
encountered no other errors. It exits with code 2 when it was unable to
connect to at least one server, and it exits with code 1 on other errors.
FILES AND CONFIGURATION¶
- /var/spool/news/leaf.node/active.read
- is a file to track when the active files have been
successfully downloaded from the upstream servers the last time. If it is
missing, this has the same effect as giving the -f option. It is
only created after the active file of each upstream servers has been
downloaded successfully.
See
leafnode(8) for the list of other files affected, and for information
on configuring
fetchnews.
ENVIRONMENT¶
- LN_SKIP_GROUPS=pattern[,pattern[...]] (since
v1.9.53)
- If set, all groups that match one of the wildmat(3)
patterns will be skipped during the fetch. This can be used as a
quick way of skipping binary groups during daytime, for instance,
example:
- env 'LN_SKIP_GROUPS=*binary*,alt.*' /usr/sbin/fetchnews -nq
- LN_LOCK_TIMEOUT
- This variable is parsed as an unsigned integer value and
determines how many seconds fetchnews will wait when trying to obtain the
lock file from another leafnode program. 0 means to wait indefinitely.
This variable takes precedence over the configuration file.
- LN_SUPPRESS_DATE
- (since v1.11.7) If this variable is defined (regardless of
its value, including empty), leafnode will skip checking the upstream
server's time by means of the NNTP DATE command.
ALGORITHM¶
Here is a brief description of what
fetchnews does.
First, after connecting to the upstream NNTP server, all articles that have the
u+r permission (0400) flag set in
/var/spool/news/out.going are posted
using POST. If a posting fails, that article is moved to
/var/spool/news/failed.postings which an administrator has to clean up
once in a while. Moving a file back to
/var/spool/news/out.going is
sufficient to retry the post it. The u+r flag is checked to prevent posting an
(incomplete) article that
leafnode(8) is still receiving at the same
time. If a posting succeeds, the article is removed from out.going in order to
not post it twice into a moderated newsgroup on a different server (which
would send multiple copies of the article to the moderator).
Note:
older fetchnews versions tried to post on all servers first, but failed to do
so when one of the servers could not be reached. If you need to work around a
dodgy upstream server that is fast but doesn't forward your posts reliably,
prohibit posting to this server using the nopost server option in the
configuration file, see
leafnode(8).
Second, the upstream server's list of newsgroups (obtained using LIST and LIST
NEWSGROUPS) is merged into
leafnode's (if timeout_active has expired,
the active.read file is not present or the active file is empty) or updated
(with NEWGROUPS).
Third,
fetchnews iterates over the list of newsgroups, performing a
GROUP, an XOVER (if supported) or XHDR Message-ID and a number of HEAD and
BODY or ARTICLE commands for each group which has been read recently. It
assumes that for the particular group it is looking at, "maxage" was
set to the groupexpire or global expire. As a consequence, it makes only sense
to set maxage lower than or equal to expire.
NOTES¶
Fetchnews does not use NEWNEWS. NEWNEWS is not used because it is often
very expensive and therefore also disabled at many sites.
After some experiments, I decided that it was better to fetch all the articles
in a group than to optimize away some; users would complain too loudly when
the optmizer guessed wrongly.
The only way to control what newsgroups are accessible to local users is to
control what the upstream server lets
fetchnews read, and that will not
keep out all crossposts (endemic in e.g. the alt.sex groups). If your site is
big enough that you need to control newsgroup access, consider using a news
server which is designed for a big site.
Fetchnews sets its real and effective uid to "news" when
started as root and refuses to run if it cannot get this user ID.
BUGS¶
Fetchnews ignores the "Supersedes:" and "Control:"
headers.
HISTORY¶
Older versions of
fetchnews posted any articles found in out.going. Since
v1.9.23,
fetchnews will only post article files that have their user
executable bit set.
Leafnode sets this bit after it has successfully
received a post. This behaviour avoids posting incomplete articles when
leafnode receives a post at the same time
fetchnews is posting.
AUTHOR¶
Written by Arnt Gulbrandsen <agulbra@troll.no> and copyright 1995-96 Troll
Tech AS, Postboks 6133 Etterstad, 0602 Oslo, Norway, fax +47 22646949.
Modified by Cornelius Krasel <krasel@wpxx02.toxi.uni-wuerzburg.de>, Markus
Enzenberger <enz@cip.physik.uni-muenchen.de> and Randolf Skerka
<Randolf.Skerka@gmx.de>. Copyright of the modifications 1997-1999.
Modified by Matthias Andree <matthias.andree@gmx.de>, copyright of the
modifications 2001-2004.
SEE ALSO¶
leafnode(8),
texpire(8),
newsq(1),
/etc/news/leafnode/leafnode/filters.example.