NAME¶
HTTP::Proxy::BodyFilter::save - A filter that saves transferred data to a file
SYNOPSIS¶
use HTTP::Proxy;
use HTTP::Proxy::BodyFilter::save;
my $proxy = HTTP::Proxy->new;
# save RFC files as we browse them
$proxy->push_filter(
path => qr!/rfc\d+.txt!,
mime => 'text/plain',
response => HTTP::Proxy::BodyFilter::save->new(
template => '%f',
prefix => 'rfc',
keep_old => 1,
)
);
$proxy->start;
DESCRIPTION¶
The HTTP::Proxy::BodyFilter::save filter can save HTTP messages (responses or
request) bodies to files. The name of the file is determined by a template and
the URI of the request.
Simply insert this filter in a filter stack, and it will save the data as it
flows through the proxy. Depending on where the filter is located in the
stack, the saved data can be more or less modified.
This filter
will create directories if it needs to!
Note: Remember that the default "mime" parameter for
"push_filter()" is "text/*" and that you may need to
change it for other MIME types.
Constructor¶
The constructor accepts quite a few options. Most of them control the
construction of the filename that will be used to save the response body.
There are two options to compute this filename:
- •
- use a template
- •
- use your own filename creation routine
The template option uses the following options:
- template => string
- The file name is build from the "template" option. The following
placeholders are available:
%% a percent sign
%h the host
%p the path (no leading separator)
%d the path (filename removed)
%f the filename (or 'index.html' if absent)
%q the query string
%P the path and the query string,
separated by '?' (if the query string is not empty)
"/" in the URI path are replaced by the separator used by
File::Spec.
The result of the template is modified by the no_host, no_dirs
and cut_dirs.
The default template is the local equivalent of the "%h/%P" Unix
path.
- no_host => boolean
- The "no_host" option makes %h empty. Default is
false.
- no_dirs => boolean
- The "no_dirs" option removes all directories from %p, %P and %d.
Default is false.
- cut_dirs => number
- The "cut_dirs" options removes the first n directories
from the content of %p, %P and %d. Default is 0.
- prefix => string
- The prefix option prepends the given prefix to the filename created
from the template. Default is "".
Using your own subroutine is also possible, with the following parameter:
- filename => coderef
- When the "filename" option is used, the "template"
option and the other template-related options ("no_host",
"no_dirs", "cut_dirs" and "prefix") are
ignored.
The "filename" option expects a reference to a subroutine. The
subroutine will receive the HTTP::Message object and must return a string
which is the path of the file to be created (an absolute path is
recommended, but a relative path is accepted).
Returning "" or "undef" will prevent the creation of the
file. This lets a filter decide even more precisely what to save or not,
even though this should be done in the match subroutine (see HTTP::Proxy's
"push_filter()" method).
Other options help the filter decide where and when to save:
- multiple => boolean
- With the multiple option, saving the same file in the same
directory will result in the original copy of file being preserved and the
second copy being named file.1. If that a file is saved yet again
with the same name, the third copy will be named file.2, and so on.
Default is true.
If multiple is set to false then a file will be overwritten by
the next one with the same name.
- timestamp => boolean
- With the "timestamp" option, the decision as to whether or not
to save a newer copy of a file depends on the local and remote timestamp
and size of the file.
The file is saved only if the date given in the "Last-Modified" is
more recent than the local file's timestamp.
Default is false.
This option is not implemented.
- keep_old => boolean
- The "keep_old" option will prevent the file to be saved if a
file with the same name already exists. Default is false.
No matter if multiple is set or not, the file will not be
saved if keep_old is set to true.
- status => \@codes
- The "status" option limits the status codes for which a response
body will be saved. The default is "[ 200 ]", which prevent
saving error pages (for 404 codes).
Examples¶
Given a request for the <
http://search.cpan.org/dist/HTTP-Proxy/> URI, the
filename is computed as follows, depending on the constructor options:
No options -> search.cpan.org/dist/HTTP-Proxy/index.html
no_host => 1 -> dist/HTTP-Proxy/index.html
no_dirs => 1 -> search.cpan.org/index.html
no_host => 1,
no_dirs => 1,
prefix => 'data' -> data/index.html
cut_dirs => 1 -> search.cpan.org/HTTP-Proxy/index.html
cut_dirs => 2 -> search.cpan.org/index.html
METHODS¶
This filter implements several methods, which are all called atuomatically:
- init()
- Handle all the parameters passed to the constructor to define the filter
behaviour.
- begin()
- Open the file to which the data will be saved.
- filter()
- Save all the data that goes through to the opened file.
- end()
- Close the file when the whole message body has been processed.
- will_modify()
- This method returns a false value, thus indicating to the system
that it will not modify data passing through.
SEE ALSO¶
HTTP::Proxy, HTTP::Proxy::BodyFilter.
AUTHOR¶
Philippe "BooK" Bruhat, <book@cpan.org>.
ACKNOWLEDGMENTS¶
Thanks to Mat Proud for asking how to store all pages which go through the proxy
to disk, without any processing. The further discussion we had led to the
writing of this class.
Wget(1) provided the inspiration for many of the file naming options.
Thanks to Nicolas Chuche for telling me about "O_EXCL".
Thanks to RafaXl Garcia-Suarez and David Rigaudiere for their help on irc while
coding the nasty "begin()" method. ";-)"
Thanks to Howard Jones for the inspiration and initial patch for the
"filename" option. Lucas Gonze provided a patch to make
"status" actually work.
Thanks to Max Maischein for detecting a bug in the parameter validation for
"filename"
(<
http://rt.cpan.org/Ticket/Display.html?id=14548>).
Thanks to Mark Tilford, who found out that the "filename" option was
incorrectly used internally
(<
http://rt.cpan.org/Ticket/Display.html?id=18644>).
Thanks to Roland Stigge and Gunnar Wolf for reporting and forwarding Debian bug
#433951 to CPAN RT
(<
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=433951>,
<
http://rt.cpan.org/Ticket/Display.html?id=33018>).
COPYRIGHT¶
Copyright 2004-2013, Philippe Bruhat.
LICENSE¶
This module is free software; you can redistribute it or modify it under the
same terms as Perl itself.