.\" Man page generated from reStructuredText.
.
.TH "FEED2EXEC" "1" "Feb 04, 2021" "0.17.1" "feed2exec"
.SH NAME
feed2exec \- The programmable feed reader
.
.nr rst2man-indent-level 0
.
.de1 rstReportMargin
\\$1 \\n[an-margin]
level \\n[rst2man-indent-level]
level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
-
\\n[rst2man-indent0]
\\n[rst2man-indent1]
\\n[rst2man-indent2]
..
.de1 INDENT
.\" .rstReportMargin pre:
. RS \\$1
. nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin]
. nr rst2man-indent-level +1
.\" .rstReportMargin post:
..
.de UNINDENT
. RE
.\" indent \\n[an-margin]
.\" old: \\n[rst2man-indent\\n[rst2man-indent-level]]
.nr rst2man-indent-level -1
.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
..
.SH SYNOPSIS
.sp
\fBfeed2exec\fP {add,ls,rm,fetch,import,export}
.SH DESCRIPTION
.sp
This command will take a configured set of feeds and fire specific
plugin for every new item found in the feed.
.SH OPTIONS
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.TP
.B \-\-version
Show the version and exit.
.TP
.B \-\-loglevel
choose specific log level [default: WARNING]
.TP
.B \-v\fP,\fB  \-\-verbose
show what is happening (loglevel: VERBOSE)
.TP
.B \-d\fP,\fB  \-\-debug
show debugging information (loglevel: DEBUG)
.TP
.BI \-\-syslog \ LEVEL
send LEVEL logs to syslog
.TP
.BI \-\-config \ TEXT
use a different configuration file
.TP
.BI \-\-database \ DB
use a different database
.TP
.B \-h\fP,\fB  \-\-help
Show this message and exit.
.UNINDENT
.UNINDENT
.UNINDENT
.SH EXAMPLES
.sp
Simple run with no side effects:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
feed2exec parse https://www.nasa.gov/rss/dyn/breaking_news.rss \-\-output echo \-\-args \(aq{item.title}\(aq
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Saving feed items to a Maildir folder:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
feed2exec add "NASA breaking news" https://www.nasa.gov/rss/dyn/breaking_news.rss \-\-folder nasa
feed2exec fetch
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
This creates the equivalent of this configuration file in \fB~/.config/feed2exec.ini\fP:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
[DEFAULT]
output = feed2exec.plugins.maildir
mailbox = ~/Maildir

[NASA breaking news]
folder = nasa
url = https://www.nasa.gov/rss/dyn/breaking_news.rss
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Send new feed items to Transmission:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
feed2exec add "Example torrent list" http://example.com/torrents/feed \-\-output transmission \-\-folder /srv/incoming
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Send new feed items to Mastodon, using the \fI\%toot\fP commandline
client:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
feed2exec add "My site" http://example.com/blog/feed \-\-output exec \-\-args \(aqtoot post "{item.title} {item.link}"\(aq
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Send new feed items to Twitter, using the tweet commandline client
from \fI\%python\-twitter\fP:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
feed2exec add "My site on twitter" http://example.com/blog/feed \-\-output exec \-\-args \(aqtweet "{item.title:40s} {item.link:100s}"\(aq
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Show feed contents:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
feed2exec add "NASA breaking news" https://www.nasa.gov/rss/dyn/breaking_news.rss \-\-output echo \-\-args "{item.title} {item.link}"
feed2exec fetch
.ft P
.fi
.UNINDENT
.UNINDENT
.SH COMMANDS
.INDENT 0.0
.IP \(bu 2
\fI\%parse\fP
.IP \(bu 2
\fI\%fetch\fP
.IP \(bu 2
\fI\%add\fP
.IP \(bu 2
\fI\%ls\fP
.IP \(bu 2
\fI\%rm\fP
.IP \(bu 2
\fI\%import\fP
.IP \(bu 2
\fI\%export\fP
.UNINDENT
.SS parse
.sp
Usage:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
parse URL
    [\-\-output PLUGIN [\-\-args ARG [ARG [...]]]
    [\-\-filter PLUGIN] [\-\-filter_args ARG [ARG [...]]]
    [\-\-mailbox PATH] [\-\-folder PATH]
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
The parse command loads and parses a single feed, without touching the
database. This is similar to calling \fIadd\fP then \fIfetch\fP on a single
feed, but the feed is not kept in the configuration. This is designed
to make quick tests with a new feed. The arguments are the same as the
\fIadd\fP command.
.SS fetch
.sp
Usage:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
fetch [\-\-parallel | \-p | \-\-jobs N | \-j N] [\-\-force | \-f] [\-\-pattern pattern]
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
The fetch command iterates through all the configured feeds or those
matching the \fBpattern\fP substring if provided.
.sp
Options:
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.TP
.BI \-\-pattern \ TEXT
only fetch feeds matchin name or URL
.TP
.B \-\-parallel
parse feeds in the background to improve performance
.TP
.BI \-j\fP,\fB  \-\-jobs \ N
start N jobs in parallel, implies
\fB\-\-parallel\fP which defaults to the number of CPUs
detected on the machine
.TP
.B \-f\fP,\fB  \-\-force
skip reading and writing the cache and
will consider all entries as new
.TP
.B \-n\fP,\fB  \-\-catchup
tell output plugins plugins to simulate their
actions
.UNINDENT
.UNINDENT
.UNINDENT
.SS add
.sp
Usage:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
add NAME URL
    [\-\-output PLUGIN [\-\-args ARG [ARG [...]]]
    [\-\-filter PLUGIN] [\-\-filter_args ARG [ARG [...]]]
    [\-\-mailbox PATH] [\-\-folder PATH]
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
The add command adds the given feed \fBNAME\fP that will be fetched from
the provided \fBURL\fP\&.
.sp
Options:
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.TP
.BI \-\-output \ PLUGIN
use PLUGIN as an output module. defaults to
\fBmaildir\fP to store in a mailbox. use
\fBnull\fP or \fBecho\fP to just fetch the feed without
doing anything. Modules are searched in the
\fIfeed2exec.plugins\fP package unless the name
contains a dot in which case the whole Python
search path is used.
.TP
.BI \-\-args \ ARGS
pass arguments ARGS to the output
plugin. supports interpolation of feed
parameters using, for example \fB{title}\fP
.TP
.BI \-\-filter \ PLUGIN
filter feed items through the PLUGIN filter
plugin
.TP
.BI \-\-filter_args \ ARGS
arguments passed to the filter plugin
.TP
.BI \-\-mailbox \ PATH
folder to store email into, defaults to
\fB~/Maildir\fP\&.
.TP
.BI \-\-folder \ PATH
subfolder to store the email into
.UNINDENT
.UNINDENT
.UNINDENT
.sp
Those parameters are documented more extensively in their equivalent
settings in the configuration file, see below.
.SS ls
.sp
The \fBls\fP command lists all configured feeds as JSON packets.
.SS rm
.sp
Usage:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
rm NAME
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Remove the feed named \fBNAME\fP from the configuration.
.SS import
.sp
Usage:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
import PATH
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Import feeds from the file named PATH. The file is expected to have
\fBoutline\fP elements and only the \fBtitle\fP and \fBxmlUrl\fP elements
are imported, as \fBNAME\fP and \fBURL\fP parameters, respectively.
.SS export
.sp
Usage:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
export PATH
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Export feeds into the file named PATH. The file will use the feed NAME
elements as \fBtitle\fP and the URL as \fBxmlUrl\fP\&.
.SH FILES
.SS Configuration file
.sp
The configuration file is loaded from (and written to, by \fBadd\fP)
\fB~/.config/feed2exec.ini\fP or \fB$XDG_CONFIG_HOME/feed2exec.ini\fP\&. It
can also be specified with the \fB\-\-config\fP commandline
parameter. This is an example configuration snippet:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
[NASA breaking news]
url = https://www.nasa.gov/rss/dyn/breaking_news.rss
output = feed2exec.plugins.echo
args = {title} {link}
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Naturally, those settings can be changed directly in the config
file. Note that there is a \fB[DEFAULT]\fP section that can be used to
apply settings to all feeds. For example, this will make all feeds
store new items in a maildir subfolder:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
[DEFAULT]
output = feed2exec.plugins.maildir
folder = feeds
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
This way individual feeds do not need to be individually configured.
.sp
\fBNOTE:\fP
.INDENT 0.0
.INDENT 3.5
feed2exec does not take care of adding the folder to
“subscriptions” in the mailbox. it is assumed that folders
are auto\-susbcribed or the user ignores subscription. if
that is a problem, you should subscribe to the folder by
hand in your email client when you add a new config. you can
also subscribe to a folder (say \fBfeeds\fP above) directly
using the \fBdoveadm mailbox subscribe feeds\fP command in
Dovecot, for example.
.UNINDENT
.UNINDENT
.sp
The following configuration parameters are supported:
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.TP
.B name
Human readable name for the feed. Equivalent to the \fBNAME\fP
argument in the \fBadd\fP command.
.TP
.B url
Address to fetch the feed from. Can be HTTP or HTTPS, but also
\fBfile://\fP resources for test purposes.
.TP
.B output
Output plugin to use. Equivalent to the \fB\-\-output\fP option in
the \fBadd\fP command.
.TP
.B args
Arguments to pass to the output plugin. Equivalent to the
\fB\-\-args\fP option in the \fBadd\fP command.
.TP
.B filter
Filter plugin to use. Equivalent to the \fB\-\-filter\fP option in
the \fBadd\fP command.
.TP
.B mailbox
Store emails in that mailbox prefix. Defaults to \fB~/Maildir\fP\&.
.TP
.B folder
Subfolder to use when writing to a mailbox. By default, a
\fIslugified\fP version of the feed name (where spaces and special
character are replaced by \fB\-\fP) is used. For example, the feed
named “NASA breaking news” would be stored in
\fB~/Maildir/nasa\-breaking\-news/\fP\&. Note that the \fBmailbox\fP
prefix is used \fBonly\fP if the \fBfolder\fP path is relative.
.TP
.B catchup
Skip to the latest feed items. The feed is still read and
parsed, and new feed items are added to the database, but output
plugins are never called.
.TP
.B pause
Completely skip feed during fetch or parse. Similar to catchup,
but doesn’t fetch the feed at all and doesn’t touch the cache.
.UNINDENT
.UNINDENT
.UNINDENT
.sp
Here is a more complete example configuration with all the settings
used:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
# this section will apply to all feeds
[DEFAULT]
# special folder location for maildir. I use this when I have multiple
# accounts synchronized with Offlineimap
mailbox = ~/Maildir/Remote/

# a feed to store NASA breaking news entry in a "nasa" subfolder
# this also demonstrates the droptitle filter
[NASA breaking news]
url = https://www.nasa.gov/rss/dyn/breaking_news.rss
folder = nasa
filter = feed2exec.plugins.droptitle
filter_args = trump

# some maildir storage require dots to get subfolders. for example,
# this will store messages in INBOX/feeds/images/ on Dovecot
[NASA image of the day]
url = https://www.nasa.gov/rss/dyn/lg_image_of_the_day.rss
folder = .feeds.images

# same feed, but save to wayback machine
[NASA IOTD wayback]
url = https://www.nasa.gov/rss/dyn/lg_image_of_the_day.rss
output = feed2exec.plugins.wayback

# this demonstrates the emptysummary filter, which fixes GitHub feeds
# that lack a proper summary
[restic]
url = https://github.com/restic/restic/tags.atom
filter = feed2exec.plugins.emptysummary

# saving to a mbox folder, one file per feed instead of one file per item
[International Space Station Reports]
url = http://blogs.nasa.gov/stationreport/feed/
mailbox = ~/Mail/
folder = stationreport.mbx

# simple generic exec call example: check for broken links using linkchecker
[NASA linkchecker]
url = https://www.nasa.gov/rss/dyn/breaking_news.rss
output = feed2exec.plugins.exec
args = linkchecker \-\-check\-extern \-\-no\-robots \-\-recursion\-level 1 \-\-quiet \(aq{item.link}\(aq

# same, but with a Ikiwiki RSS feed, which needs fixing
[Ikiwiki linkchecker]
url = http://ikiwiki.info/recentchanges/index.rss
output = feed2exec.plugins.exec
filter = feed2exec.plugins.ikiwiki_recentchanges
args = linkchecker \-\-check\-extern \-\-no\-robots \-\-recursion\-level 1 \-\-quiet \(aq{item.link}\(aq

# retweet hurricane news
[NASA Hurricane breaking news]
url = https://www.nasa.gov/rss/dyn/hurricaneupdate.rss
output = feed2exec.plugins.exec
args = tweet "{item.title:.40s} {item.link:.100s}"

# same, but on the mastodon network
#
# we can have multiple entries with the same URL without problems, as
# long as the feed name is different. it does mean that the feed will
# be fetched and parsed multiple times, unfortunately.
#
# this could be improved to include the \(aq{item.summary}\(aq and extra markup,
# for example.
[NASA Hurricane breaking news \- Mastodon]
url = https://www.nasa.gov/rss/dyn/hurricaneupdate.rss
output = feed2exec.plugins.exec
# unfortunately, this will noisily report the URL of the posted link,
# which you may not want. to avoid that, encourage upstream to do the
# right thing: https://github.com/ihabunek/toot/issues/46 ... or use
# another tool listed here:
# https://github.com/tootsuite/documentation/blob/master/Using\-Mastodon/Apps.md
args = toot post "{item.title} {item.link}"
# output is disabled here. feed will be fetched and parsed, but no
# toot will be sent
catchup = True

# same, but on the Pump.io network
[NASA Hurricane breaking news \- Pump]
url = https://www.nasa.gov/rss/dyn/hurricaneupdate.rss
output = feed2exec.plugins.exec
args = p post note "{item.title} {item.link}"

# crude podcast client
[NASA Whats up?]
url = https://www.nasa.gov/rss/dyn/whats_up.rss
output = feed2exec.plugins.exec
# XXX: this doesn\(aqt handle errors properly: if there is a feed without
# enclosures, the whole thing will crash.
args = wget \-P /srv/podcasts/nasa/ "{item.enclosures[0].href}"
# feed is paused here. feed will not be fetched and parsed at all and
# no post will be sent.
pause = True

# download torrents linked from a RSS feed
[torrents]
url = http://example.com/torrents.rss
output = feed2exec.plugins.exec
args = transmission\-remote \-a \(aq{item.link}\(aq \-w \(aq/srv/incoming\(aq

# same thing with an actual plugin
[torrents]
url = http://example.com/torrents.rss
output = feed2exec.plugins.transmission
args = seedbox.example.com
folder = /srv/incoming

.ft P
.fi
.UNINDENT
.UNINDENT
.SS Cache database
.sp
The feeds cache is stored in a \fBfeed2exec.db\fP file. It is a
SQLite database and can be inspected using standard sqlite
tools. It is used to keep track of which feed and items have been
processed. To clear the cache, you can simply remove the file, which
will make the program process all feeds items from scratch again. In
this case, you should use the \fB\-\-catchup\fP argument to avoid
duplicate processing. You can also use the \fBnull\fP output plugin to
the same effect.
.SH LIMITATIONS
.sp
Feed support is only as good as \fBfeedparser\fP library which isn’t as solid as
I expected. In particular, I had issues with \fI\%feeds without dates\fP and
\fI\%without guid\fP\&.
.INDENT 0.0
.INDENT 3.5
.UNINDENT
.UNINDENT
.sp
Unit test coverage is incomplete, but still pretty decent, above 90%.
.sp
The \fBexec\fP plugin itself is not well tested and may have serious
security issues.
.sp
API, commandline interface, configuration file syntax and database format can
be changed until the 1.0 release is published, at which point normal \fI\%Semantic
Versioning\fP semantics apply.
.sp
The program is written mainly targeting Python 3.5 and 3.7, but should
support later releases as well. See the \fBsetup.py\fP classification
for an authoritative reference. Python 2.7 is not supported anymore.
.sp
The SQL storage layer is badly written and is known to trigger locking
issues with SQLite when doing multiprocessing. The global LOCK object
could be used to work around this issue but that could mean pretty bad
coupling. A good inspiration may be the \fI\%beets story about this
problem\fP\&. And of course,
another alternative would be to considering something like SQLalchemy
instead of rolling our own ORM. There has, however, been some
improvements to the locking recently, although that has been done with
\fIthread\fP instead of \fIprocess\fP\-specific locks.
.sp
Older feed items are not purged from the database when they disappear from the
feed, which may lead to database bloat in the long term. Similarly,
there is no way for plugins to remove old entry that expire from the feed.
.SH SEE ALSO
.sp
\fBfeed2exec\-plugins(1)\fP, \fBfeed2imap(1)\fP, \fBrss2email(1)\fP
.SH AUTHOR
Antoine Beaupré
.SH COPYRIGHT
Copyright (C) 2016-2019  Antoine Beaupré
.\" Generated by docutils manpage writer.
.