.\" Automatically generated by Pod::Man 4.10 (Pod::Simple 3.35) .\" .\" Standard preamble: .\" ======================================================================== .de Sp \" Vertical space (when we can't use .PP) .if t .sp .5v .if n .sp .. .de Vb \" Begin verbatim text .ft CW .nf .ne \\$1 .. .de Ve \" End verbatim text .ft R .fi .. .\" Set up some character translations and predefined strings. \*(-- will .\" give an unbreakable dash, \*(PI will give pi, \*(L" will give a left .\" double quote, and \*(R" will give a right double quote. \*(C+ will .\" give a nicer C++. Capital omega is used to do unbreakable dashes and .\" therefore won't be available. \*(C` and \*(C' expand to `' in nroff, .\" nothing in troff, for use with C<>. .tr \(*W- .ds C+ C\v'-.1v'\h'-1p'\s-2+\h'-1p'+\s0\v'.1v'\h'-1p' .ie n \{\ . ds -- \(*W- . ds PI pi . if (\n(.H=4u)&(1m=24u) .ds -- \(*W\h'-12u'\(*W\h'-12u'-\" diablo 10 pitch . if (\n(.H=4u)&(1m=20u) .ds -- \(*W\h'-12u'\(*W\h'-8u'-\" diablo 12 pitch . ds L" "" . ds R" "" . ds C` "" . ds C' "" 'br\} .el\{\ . ds -- \|\(em\| . ds PI \(*p . ds L" `` . ds R" '' . ds C` . ds C' 'br\} .\" .\" Escape single quotes in literal strings from groff's Unicode transform. .ie \n(.g .ds Aq \(aq .el .ds Aq ' .\" .\" If the F register is >0, we'll generate index entries on stderr for .\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index .\" entries marked with X<> in POD. Of course, you'll have to process the .\" output yourself in some meaningful fashion. .\" .\" Avoid warning from groff about undefined register 'F'. .de IX .. .nr rF 0 .if \n(.g .if rF .nr rF 1 .if (\n(rF:(\n(.g==0)) \{\ . if \nF \{\ . de IX . tm Index:\\$1\t\\n%\t"\\$2" .. . if !\nF==2 \{\ . nr % 0 . nr F 2 . \} . \} .\} .rr rF .\" ======================================================================== .\" .IX Title "AUTOSEARCH 1p" .TH AUTOSEARCH 1p "2018-11-25" "perl v5.28.0" "User Contributed Perl Documentation" .\" For nroff, turn off justification. Always turn off hyphenation; it makes .\" way too many mistakes in technical documents. .if n .ad l .nh .SH "NAME" AutoSearch \-\- a web\-search tracking application .SH "SYNOPSIS" .IX Header "SYNOPSIS" AutoSearch [\-\-stats] [\-\-verbose] \-n \*(L"Query Name\*(R" \-s \*(L"query string\*(R" \-\-engine engine [\-\-mail you@where.com] [\-\-options \*(L"opt=val\*(R"]... [\-\-filter \*(L"filter\*(R"] [\-\-host host] [\-\-port port] [\-\-userid bbunny \-\-password c4rr0t5] [\-\-ignore_channels \s-1KABC,KCBS,KNBC\s0] qid .PP AutoSearch \-\-VERSION AutoSearch \-\-help AutoSearch \-\-man .SH "DESCRIPTION" .IX Header "DESCRIPTION" \&\fBAutoSearch\fR performs a web-based search and puts the results set in \fIqid/index.html\fR. Subsequent searches (i.e., the second form above) \&\fBAutoSearch\fR determine what changes (if any) occurred to the results sent since the last run. These incremental changes are recorded in \fIqid/YYYYMMDD.html\fR. .PP \&\fBAutoSearch\fR is amenable to be run as a \fBcron\fR job because all the input parameters are saved in the web pages. \fBAutoSearch\fR can act as a automated query agent for a particular search. The output files are designed to be a set of web pages to easily display the results set with a web browser. .PP Example: .PP .Vb 4 \& AutoSearch \-n \*(AqLSAM Replication\*(Aq \& \-s \*(Aq"lsam replication"\*(Aq \& \-e AltaVista \& replication_query .Ve .PP This query (which should be all on one line) creates a directory replication_query and fills it with the fascinating output of the AltaVista query on \f(CW"lsam replication"\fR, with pages titled ``\s-1LSAM\s0 Replication''. (Note the quoting: the single quotes in \&\f(CW\*(Aq"lsam replication"\*(Aq\fR are for the shell, the double quotes are for AltaVista to search for the phrase rather than the separate words.) .PP A more complicated example: .PP .Vb 4 \& AutoSearch \-n \*(AqExternal Links to LSAM\*(Aq \& \-s \*(Aq(link:www.isi.edu/lsam or link:www.isi.edu/~lsam) \-url:isi.edu\*(Aq \& \-e AltaVista::AdvancedWeb \& \-o coolness=hot .Ve .PP This query does an advanced AltaVista search and specifies the (hypothetical) ``coolness'' option to the search engine. .SH "OPTIONS" .IX Header "OPTIONS" .ie n .IP """qid""" 4 .el .IP "\f(CWqid\fR" 4 .IX Item "qid" The \fIquery identifer\fR specifies the directory in which all the files that relate to this query and search results will live. It can be an absolute path, or a relative path from cwd. If the directory does not exist, it will be created and a new search started. .ie n .IP """\-\-stats""" 4 .el .IP "\f(CW\-\-stats\fR" 4 .IX Item "--stats" Show search statistics: the query string, number of hits, number of filtered hits, filter string, number of suspended (deleted) hits, previous set size, current set size, etc. .ie n .IP """\-v"" or ""\-\-verbose""" 4 .el .IP "\f(CW\-v\fR or \f(CW\-\-verbose\fR" 4 .IX Item "-v or --verbose" Verbose: output additional messages and warnings. .ie n .IP """\-n"" or ""\-\-qn"" or ""\-\-queryname""" 4 .el .IP "\f(CW\-n\fR or \f(CW\-\-qn\fR or \f(CW\-\-queryname\fR" 4 .IX Item "-n or --qn or --queryname" Specify the query name. The query name is used as a heading in the web pages, therefore it should be a 'nice' looking version of the query string. .ie n .IP """\-s"" or ""\-\-qs"" or ""\-\-querystring""" 4 .el .IP "\f(CW\-s\fR or \f(CW\-\-qs\fR or \f(CW\-\-querystring\fR" 4 .IX Item "-s or --qs or --querystring" Specify the query string. The query string is the character string which will be submitted to the search engine. You may include special characters to group or to qualify the search. .ie n .IP """\-e"" or ""\-\-engine""" 4 .el .IP "\f(CW\-e\fR or \f(CW\-\-engine\fR" 4 .IX Item "-e or --engine" Specify the search engine. The query string will be submitted to the user specified search engine. .Sp In many cases there are specialized versions of search engines. For example, \fBAltaVista::AdvancedWeb\fR and \fBAltaVista::News\fR allow more powerful and Usenet searches. See AltaVista or the man page for your search engine for details about specialized variations. .ie n .IP """\-\-listnewurls""" 4 .el .IP "\f(CW\-\-listnewurls\fR" 4 .IX Item "--listnewurls" In addition to all the normal file maintenance, print all new URLs to \s-1STDOUT,\s0 one per line. .ie n .IP """\-o"" or ""\-\-options""" 4 .el .IP "\f(CW\-o\fR or \f(CW\-\-options\fR" 4 .IX Item "-o or --options" Specify the query options. The query options will be submitted to the user search engine with the query string. This feature permits modification of the query string for a specific search engine or option. More than one query option may be specified. .Sp Example: \&\f(CW\*(C`\-o what=news\*(C'\fR causes AltaVista to search Usenet. Although this works, the preferred mechanism in this case would be \f(CW\*(C`\-e AltaVista::News\*(C'\fR or \f(CW\*(C`\-e AltaVista::AdvancedNews\*(C'\fR. Options are intended for internal or expert use. .ie n .IP """\-f"" or ""\-\-uf"" or ""\-\-urlfilter""" 4 .el .IP "\f(CW\-f\fR or \f(CW\-\-uf\fR or \f(CW\-\-urlfilter\fR" 4 .IX Item "-f or --uf or --urlfilter" This option specifies a regular expression which will be compared against the URLs of any results; if they match the case-insensitive regular expression, they will be removed from the hit set. .Sp Example: \&\f(CW\*(C`\-f \*(Aq.*\e.isi\e.edu\*(Aq\*(C'\fR avoids all of \s-1ISI\s0's web pages. .ie n .IP """\-\-cleanup i""" 4 .el .IP "\f(CW\-\-cleanup i\fR" 4 .IX Item "--cleanup i" Delete all traces of query results from more than i days ago. If \&\-\-cleanup is given, all other options other than the qid will be ignored. .ie n .IP """\-\-cmdline""" 4 .el .IP "\f(CW\-\-cmdline\fR" 4 .IX Item "--cmdline" Reconstruct the complete command line (AutoSearch and all its arguments) that was used to create the query results. Command line will be shown on \s-1STDERR.\s0 If \-\-cmdline is given, all other options other than the qid will be ignored. .ie n .IP """\-\-mail user@address"" or ""\-m user@address""" 4 .el .IP "\f(CW\-\-mail user@address\fR or \f(CW\-m user@address\fR" 4 .IX Item "--mail user@address or -m user@address" After search is complete, send email to that user, listing the \s-1NEW\s0 results. Email is \s-1HTML\s0 format. Requires the Email::Send and related modules. If you send email through an \s-1SMTP\s0 server, you must set environment variable \s-1SMTPSERVER\s0 to your server name or \s-1IP\s0 address. If your \s-1SMTP\s0 server requires password, you must set environment variables \s-1SMTPUSERNAME\s0 and \s-1SMTPPASSWORD.\s0 If you send email via sendmail, you should set environment variable \s-1SENDMAIL\s0 if the sendmail executable is not in the path. .ie n .IP """\-\-emailfrom user@address""" 4 .el .IP "\f(CW\-\-emailfrom user@address\fR" 4 .IX Item "--emailfrom user@address" If your outgoing mail server rejects email from certain users, you can use this argument to set the From: header. .ie n .IP """\-\-userid bbunny""" 4 .el .IP "\f(CW\-\-userid bbunny\fR" 4 .IX Item "--userid bbunny" If the search engine requires a login/password (e.g. Ebay::Completed), use this. .ie n .IP """\-\-password Carr0t5""" 4 .el .IP "\f(CW\-\-password Carr0t5\fR" 4 .IX Item "--password Carr0t5" If the search engine requires a login/password (e.g. Ebay::Mature), use this. .SH "DESCRIPTION" .IX Header "DESCRIPTION" \&\fBAutoSearch\fR submits a query to a search engine, produces \s-1HTML\s0 pages that reflect the set of 'hits' (filtered search results) returned by the search engine, and tracks these results over time. The \s-1URL\s0 and title are displayed in the \fIqid/index.html\fR, the \s-1URL,\s0 the title, and description are displayed in the 'weekly' files. .PP To organize these results, each search result is placed in a query information directory (qid). The directory becomes the search results \&'handle', an easy way to track a set of results. Thus a qid of \&\f(CW\*(C`/usr/local/htdocs/lsam/autosearch/load_balancing\*(C'\fR might locate the results on your web server at \&\f(CW\*(C`http://www.isi.edu/lsam/autosearch/load_balancing\*(C'\fR. .PP Inside the qid directory you will find files relating to this query. The primary file is \fIindex.html\fR, which reflects the latest search results. Every not-filtered hit for every search is stored in \&\fIindex.html\fR. When a hit is no longer found by the search engine it a removed from \fIindex.html\fR. As new results for a search are returned from the search engine they are placed in \fIindex.html\fR. .PP At the bottom of \fIindex.html\fR, there is a heading \*(L"Weekly Search Results\*(R", which is updated each time the search is submitted (see \&\*(L"\s-1AUTOMATED SEARCHING\*(R"\s0). The list of search runs is stored in reverse chronological order. Runs which provide no new information are identified with .PP .Vb 1 \& No Unique Results found for search on .Ve .PP Runs which contain changes are identified by .PP .Vb 1 \& Web search results for search on .Ve .PP which will be linked a page detailing the changes from that run. .PP Detailed search results are noted in weekly files. These files are named \fI\s-1YYYYMMDD\s0.html\fR and are stored in the qid directory. The weekly files include \s-1THE URL,\s0 title, and a the description (if available). The title is a link to the original web page. .SH "AUTOMATED SEARCHING" .IX Header "AUTOMATED SEARCHING" On UNIX-like systems, \fBcron\fR\|(1) may be used to establish periodic searches and the web pages will be maintained by \fBAutoSearch\fR. To establish the first search, use the first example under \s-1SYNOPSIS.\s0 You must specify the qid, query name and query string. If any of the items are missing, you will be interactively prompted for the missing item(s). .PP Once the first search is complete you can re-run the search with the second form under \s-1SYNOPSIS.\s0 .PP A cron entry like: .PP .Vb 1 \& 0 3 * * 1 /nfs/u1/wls/AutoSearch.pl /www/div7/lsam/autosearch/caching .Ve .PP might be used to run the search each Monday at 3:00 \s-1AM.\s0 The query name and query string may be repeated; but they will not be used. This means that with a cron line like: .PP .Vb 1 \& 0 3 * * 1 /nfs/u1/wls/AutoSearch.pl /www/div7/lsam/autosearch/caching \-n caching \-s caching .Ve .PP a whole new search series can be originated by .PP .Vb 1 \& rm \-r /www/div7/lsam/autosearch/caching .Ve .PP However, the only reason to start a new search series would be to throw away the old weekly files. .PP We don't recommend running searches more than once per day, but if so the per-run files will be updated in-place. Any changes are added to the page with a comment that \*(L"Recently Added:\*(R"; and deletions are indicated with \*(L"Recently Suspended:.\*(R" .SH "CHANGING THE LOOK OF THE PAGES" .IX Header "CHANGING THE LOOK OF THE PAGES" The basic format of these two pages is simple and customizable. One requirement is that the basic structure remain unchanged. \s-1HTML\s0 comments are used to identify sections of the document. Almost everything can be changed except for the strings which identify the section starts and ends. .PP Noteworthy tags and their meaning: .IP ".*" 16 .IX Item ".*" The text contained within this tag is placed at the top of the output page. If the text contains \fIAutoSearch \s-1WEB\s0 Searching\fR, then the query name will replace it. If the text does not contain this magic string and it is the first ever search, the user will be asked for a query name. .IP "" 16 .IX Item "" The text contained between the braces is the query string. This is how \fBAutoSearch\fR maintains the query string. You may edit this string to change the query string; but only in \fIqid/index.html\fR. The text \fIask user\fR is special and will force \fBAutoSearch\fR to request the search string from the user. .IP "" 16 .IX Item "" The text contained between the braces is the search engine. Other engines supported are HotBot and Lycos. You may edit this string to change the engine used; but only in \fIqid/index.html\fR. The text \fIask user\fR is special and will force \fBAutoSearch\fR to to request the search string from the user. .IP "" 16 .IX Item "" The text contained between the braces specifies a query options. Multiple occurrences of this command are allowed to specify multiple options. .IP "" 16 .IX Item "" The text contained between the braces is the \s-1URL\s0 filter. This is how \&\fBAutoSearch\fR maintains the filter. Again you may edit this string to change the query string; but only in \fIqid/index.html\fR. The text \&\fIask user\fR is special and will force \fBAutoSearch\fR to ask the user (\s-1STDIN\s0) for the query string. When setting up the first search, you must edit \fIfirst_index.html\fR, not \fIqid/index.html\fR. The \s-1URL\s0 filter is a standard perl5 regular expression. URLs which do not match will be kept. .IP ".*" 16 .IX Item ".*" The text contained within this tag is placed at the bottom of the output page. This is a good place to put navigation, page owner information, etc. .PP The remainder of the tags fall into a triplet of \fI~Heading\fR, \&\fI~Template\fR, and \fI~\fR, where ~ is Summary, Weekly, Appended, and Suspended. The sub-sections appear in the order given. To produce a section \fBAutoSearch\fR outputs the heading, the template, the section, n copies of the formatted data, and an /section. The tags and their function are: .IP "~Heading" 16 .IX Item "~Heading" The heading tag identifies the heading for a section of the output file. The SummaryHeading is for the summary portion, etc. The section may be empty (e.g., Suspended) and thus no heading is output. .IP "~Template" 16 .IX Item "~Template" The template tag identifies how each item is to be formatted. Simple text replacement is used to change the template into the actual output text. The text to be replaced is noted in \s-1ALLCAPS.\s0 .IP "~" 16 This tag is used to locate the section (Summary, Weekly, etc.). This section represents the actual n\-items of data. .PP You can edit these values in the \fIqid/index.html\fR page of an existing search. The file \fIfirst_index.html\fR (in the directory above \fIqid\fR) will be used as a default template for new queries. .PP Examples of these files can be seen in the pages under \&\f(CW\*(C`http://www.isi.edu/lsam/tools/autosearch/\*(C'\fR, or in the output generated by a new AutoSearch. .SH "FILES" .IX Header "FILES" .IP "\fIfirst_index.html\fR" 20 .IX Item "first_index.html" optional file to determine the default format of the \fIindex.html\fR file of a new query. .IP "\fIfirst_date.html\fR" 20 .IX Item "first_date.html" optional file to determine the default format of the \fI\s-1YYYYMMDD\s0.html\fR file for a new query. .IP "\fIqid/index.html\fR" 20 .IX Item "qid/index.html" (automatically created) latest search results, and reverse chronological list of periodic searches. .IP "\fIqid/date.html\fR" 20 .IX Item "qid/date.html" file used as a template for the \fI\s-1YYYYMMDD\s0.html\fR files. .IP "\fIqid/YYYYMMDD.html\fR" 20 .IX Item "qid/YYYYMMDD.html" (automatically created) summary of changes for a particular date (\s-1AKA\s0 \&'Weekly' file). .PP Optional files \fIfirst_index.html\fR and \fIfirst_date.html\fR are used for the initial search as a template for \fIqid/index.html\fR and \&\fIdate.html\fR, respectively. If either of these files does not exist; a default-default template is stored within the \fIAutoSearch\fR source. The intention of these two files is to permit a user to establish a framework for a group of search sets which have a common format. By leaving the default query name and query string alone, they will be overridden by command line inputs. .SH "SEE ALSO" .IX Header "SEE ALSO" For the library, see WWW::Search, for the perl regular expressions, see perlre. .SH "AUTHORS" .IX Header "AUTHORS" Wm. L. Scheding .PP \&\fBAutoSearch\fR is a re-implementation of an earlier version written by Kedar Jog. .SH "COPYRIGHT" .IX Header "COPYRIGHT" Copyright (C) 1996\-1997 University of Southern California. All rights reserved. .PP Redistribution and use in source and binary forms are permitted provided that the above copyright notice and this paragraph are duplicated in all such forms and that any documentation, advertising materials, and other materials related to such distribution and use acknowledge that the software was developed by the University of Southern California, Information Sciences Institute. The name of the University may not be used to endorse or promote products derived from this software without specific prior written permission. .PP \&\s-1THIS SOFTWARE IS PROVIDED \*(L"AS IS\*(R" AND WITHOUT ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.\s0 .SH "DESIRED FEATURES" .IX Header "DESIRED FEATURES" These are good ideas that people have suggested. .IP "\s-1URL\s0 validation." 4 .IX Item "URL validation." Validate the status of each \s-1URL\s0 (with \s-1HTTP HEAD\s0 requests) and indicate this status in the output. .IP "Multi-search." 4 .IX Item "Multi-search." It should be possible to merge the results of searches from two search-engines. If this merger were done as a new search engine, this operation would be transparent to AutoSearch. .SH "BUGS" .IX Header "BUGS" None known at this time; please inform the maintainer mthurn@cpan.org if any crop up.