.\" Automatically generated by Pod::Man 4.14 (Pod::Simple 3.40)
.\"
.\" Standard preamble:
.\" ========================================================================
.de Sp \" Vertical space (when we can't use .PP)
.if t .sp .5v
.if n .sp
..
.de Vb \" Begin verbatim text
.ft CW
.nf
.ne \\$1
..
.de Ve \" End verbatim text
.ft R
.fi
..
.\" Set up some character translations and predefined strings.  \*(-- will
.\" give an unbreakable dash, \*(PI will give pi, \*(L" will give a left
.\" double quote, and \*(R" will give a right double quote.  \*(C+ will
.\" give a nicer C++.  Capital omega is used to do unbreakable dashes and
.\" therefore won't be available.  \*(C` and \*(C' expand to `' in nroff,
.\" nothing in troff, for use with C<>.
.tr \(*W-
.ds C+ C\v'-.1v'\h'-1p'\s-2+\h'-1p'+\s0\v'.1v'\h'-1p'
.ie n \{\
.    ds -- \(*W-
.    ds PI pi
.    if (\n(.H=4u)&(1m=24u) .ds -- \(*W\h'-12u'\(*W\h'-12u'-\" diablo 10 pitch
.    if (\n(.H=4u)&(1m=20u) .ds -- \(*W\h'-12u'\(*W\h'-8u'-\"  diablo 12 pitch
.    ds L" ""
.    ds R" ""
.    ds C` ""
.    ds C' ""
'br\}
.el\{\
.    ds -- \|\(em\|
.    ds PI \(*p
.    ds L" ``
.    ds R" ''
.    ds C`
.    ds C'
'br\}
.\"
.\" Escape single quotes in literal strings from groff's Unicode transform.
.ie \n(.g .ds Aq \(aq
.el       .ds Aq '
.\"
.\" If the F register is >0, we'll generate index entries on stderr for
.\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index
.\" entries marked with X<> in POD.  Of course, you'll have to process the
.\" output yourself in some meaningful fashion.
.\"
.\" Avoid warning from groff about undefined register 'F'.
.de IX
..
.nr rF 0
.if \n(.g .if rF .nr rF 1
.if (\n(rF:(\n(.g==0)) \{\
.    if \nF \{\
.        de IX
.        tm Index:\\$1\t\\n%\t"\\$2"
..
.        if !\nF==2 \{\
.            nr % 0
.            nr F 2
.        \}
.    \}
.\}
.rr rF
.\"
.\" Accent mark definitions (@(#)ms.acc 1.5 88/02/08 SMI; from UCB 4.2).
.\" Fear.  Run.  Save yourself.  No user-serviceable parts.
.    \" fudge factors for nroff and troff
.if n \{\
.    ds #H 0
.    ds #V .8m
.    ds #F .3m
.    ds #[ \f1
.    ds #] \fP
.\}
.if t \{\
.    ds #H ((1u-(\\\\n(.fu%2u))*.13m)
.    ds #V .6m
.    ds #F 0
.    ds #[ \&
.    ds #] \&
.\}
.    \" simple accents for nroff and troff
.if n \{\
.    ds ' \&
.    ds ` \&
.    ds ^ \&
.    ds , \&
.    ds ~ ~
.    ds /
.\}
.if t \{\
.    ds ' \\k:\h'-(\\n(.wu*8/10-\*(#H)'\'\h"|\\n:u"
.    ds ` \\k:\h'-(\\n(.wu*8/10-\*(#H)'\`\h'|\\n:u'
.    ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'^\h'|\\n:u'
.    ds , \\k:\h'-(\\n(.wu*8/10)',\h'|\\n:u'
.    ds ~ \\k:\h'-(\\n(.wu-\*(#H-.1m)'~\h'|\\n:u'
.    ds / \\k:\h'-(\\n(.wu*8/10-\*(#H)'\z\(sl\h'|\\n:u'
.\}
.    \" troff and (daisy-wheel) nroff accents
.ds : \\k:\h'-(\\n(.wu*8/10-\*(#H+.1m+\*(#F)'\v'-\*(#V'\z.\h'.2m+\*(#F'.\h'|\\n:u'\v'\*(#V'
.ds 8 \h'\*(#H'\(*b\h'-\*(#H'
.ds o \\k:\h'-(\\n(.wu+\w'\(de'u-\*(#H)/2u'\v'-.3n'\*(#[\z\(de\v'.3n'\h'|\\n:u'\*(#]
.ds d- \h'\*(#H'\(pd\h'-\w'~'u'\v'-.25m'\f2\(hy\fP\v'.25m'\h'-\*(#H'
.ds D- D\\k:\h'-\w'D'u'\v'-.11m'\z\(hy\v'.11m'\h'|\\n:u'
.ds th \*(#[\v'.3m'\s+1I\s-1\v'-.3m'\h'-(\w'I'u*2/3)'\s-1o\s+1\*(#]
.ds Th \*(#[\s+2I\s-2\h'-\w'I'u*3/5'\v'-.3m'o\v'.3m'\*(#]
.ds ae a\h'-(\w'a'u*4/10)'e
.ds Ae A\h'-(\w'A'u*4/10)'E
.    \" corrections for vroff
.if v .ds ~ \\k:\h'-(\\n(.wu*9/10-\*(#H)'\s-2\u~\d\s+2\h'|\\n:u'
.if v .ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'\v'-.4m'^\v'.4m'\h'|\\n:u'
.    \" for low resolution devices (crt and lpr)
.if \n(.H>23 .if \n(.V>19 \
\{\
.    ds : e
.    ds 8 ss
.    ds o a
.    ds d- d\h'-1'\(ga
.    ds D- D\h'-1'\(hy
.    ds th \o'bp'
.    ds Th \o'LP'
.    ds ae ae
.    ds Ae AE
.\}
.rm #[ #] #H #V #F C
.\" ========================================================================
.\"
.IX Title "LW2 3perl"
.TH LW2 3perl "2021-01-02" "perl v5.32.0" "User Contributed Perl Documentation"
.\" For nroff, turn off justification.  Always turn off hyphenation; it makes
.\" way too many mistakes in technical documents.
.if n .ad l
.nh
.SH "NAME"
LW2 \- Perl HTTP library version 2.5
.SH "SYNOPSIS"
.IX Header "SYNOPSIS"
use \s-1LW2\s0;
.PP
require '\s-1LW2\s0.pm';
.SH "DESCRIPTION"
.IX Header "DESCRIPTION"
Libwhisker is a Perl library useful for \s-1HTTP\s0 testing scripts.  It
contains a pure-Perl reimplementation of functionality found in the \f(CW\*(C`LWP\*(C'\fR,
\&\f(CW\*(C`URI\*(C'\fR, \f(CW\*(C`Digest::MD5\*(C'\fR, \f(CW\*(C`Digest::MD4\*(C'\fR, \f(CW\*(C`Data::Dumper\*(C'\fR, \f(CW\*(C`Authen::NTLM\*(C'\fR, 
\&\f(CW\*(C`HTML::Parser\*(C'\fR, \f(CW\*(C`HTML::FormParser\*(C'\fR, \f(CW\*(C`CGI::Upload\*(C'\fR, \f(CW\*(C`MIME::Base64\*(C'\fR,
and \f(CW\*(C`GetOpt::Std\*(C'\fR modules.
.PP
Libwhisker is designed to be portable (a single perl file), fast (general
benchmarks show libwhisker is faster than \s-1LWP\s0), and flexible (great care
was taken to ensure the library does exactly what you want to do, even
if it means breaking the protocol).
.SH "FUNCTIONS"
.IX Header "FUNCTIONS"
The following are the functions contained in Libwhisker:
.IP "\fBauth_brute_force\fR" 4
.IX Item "auth_brute_force"
Params: \f(CW$auth_method\fR, \e%req, \f(CW$user\fR, \e@passwords [, \f(CW$domain\fR, \f(CW$fail_code\fR ]
.Sp
Return: \f(CW$first_valid_password\fR, undef if error/none found
.Sp
Perform a \s-1HTTP\s0 authentication brute force against a server (host and \s-1URI\s0 
defined in \f(CW%req\fR).  It will try every password in the password array for 
the given user.  The first password (in conjunction with the given user) 
that doesn't return \s-1HTTP 401\s0 is returned (and the brute force is stopped 
at that point).  You should retry the request with the given password and
double-check that you got a useful \s-1HTTP\s0 return code that indicates
successful authentication (200, 302), and not something a bit more 
abnormal (407, 500, etc).  \f(CW$domain\fR is optional, and is only used for \s-1NTLM\s0
auth.
.Sp
Note: set up any proxy settings and proxy auth in \f(CW%req\fR before calling
this function.
.Sp
You can brute-force proxy authentication by setting up the target proxy
as proxy_host and proxy_port in \f(CW%req\fR, using an arbitrary host and uri
(preferably one that is reachable upon successful proxy authorization),
and setting the \f(CW$fail_code\fR to 407.  The \f(CW$auth_method\fR passed to this
function should be a proxy-based one ('proxy\-basic', 'proxy\-ntlm', etc).
.Sp
if your server returns something other than 401 upon auth failure, then
set \f(CW$fail_code\fR to whatever is returned (and it needs to be something
*different* than what is received on auth success, or this function
won't be able to tell the difference).
.IP "\fBauth_unset\fR" 4
.IX Item "auth_unset"
Params: \e%req
.Sp
Return: nothing (modifies \f(CW%req\fR)
.Sp
Modifes \f(CW%req\fR to disable all authentication (regular and proxy).
.Sp
Note: it only removes the values set by \fBauth_set()\fR.  Manually-defined
[Proxy\-]Authorization headers will also be deleted (but you shouldn't 
be using the auth_* functions if you're manually handling your own auth...)
.IP "\fBauth_set\fR" 4
.IX Item "auth_set"
Params: \f(CW$auth_method\fR, \e%req, \f(CW$user\fR, \f(CW$password\fR [, \f(CW$domain\fR]
.Sp
Return: nothing (modifies \f(CW%req\fR)
.Sp
Modifes \f(CW%req\fR to use the indicated authentication info.
.Sp
Auth_method can be: 'basic', 'proxy\-basic', 'ntlm', 'proxy\-ntlm'.
.Sp
Note: this function may not necessarily set any headers after being called.
Also, proxy-ntlm with \s-1SSL\s0 is not currently supported.
.IP "\fBcookie_new_jar\fR" 4
.IX Item "cookie_new_jar"
Params: none
.Sp
Return: \f(CW$jar\fR
.Sp
Create a new cookie jar, for use with the other functions.  Even though
the jar is technically just a hash, you should still use this function
in order to be future-compatible (should the jar format change).
.IP "\fBcookie_read\fR" 4
.IX Item "cookie_read"
Params: \f(CW$jar\fR, \e%response [, \e%request, \f(CW$reject\fR ]
.Sp
Return: \f(CW$num_of_cookies_read\fR
.Sp
Read in cookies from an \f(CW%response\fR hash, and put them in \f(CW$jar\fR.
.Sp
Notice: cookie_read uses internal magic done by http_do_request
in order to read cookies regardless of 'Set\-Cookie[2]' header
appearance.
.Sp
If the optional \f(CW%request\fR hash is supplied, then it will be used to
calculate default host and path values, in case the cookie doesn't
specify them explicitly.  If \f(CW$reject\fR is set to 1, then the \f(CW%request\fR
hash values are used to calculate and reject cookies which are not
appropriate for the path and domains of the given request.
.IP "\fBcookie_parse\fR" 4
.IX Item "cookie_parse"
Params: \f(CW$jar\fR, \f(CW$cookie\fR [, \f(CW$default_domain\fR, \f(CW$default_path\fR, \f(CW$reject\fR ]
.Sp
Return: nothing
.Sp
Parses the cookie into the various parts and then sets the appropriate 
values in the cookie \f(CW$jar\fR. If the cookie value is blank, it will delete 
it from the \f(CW$jar\fR.  See the 'docs/cookies.txt' document for a full
explanation of how Libwhisker parses cookies and what \s-1RFC\s0 aspects are 
supported.
.Sp
The optional \f(CW$default_domain\fR value is taken literally.  Values with no 
leading dot (e.g. 'www.host.com') are considered to be strict hostnames 
and will only match the identical hostname.  Values with leading dots (e.g. 
\&'.host.com') are treated as sub-domain matches for a single domain level.
If the cookie does not indicate a domain, and a \f(CW$default_domain\fR is not
provided, then the cookie is considered to match all domains/hosts.
.Sp
The optional \f(CW$default_path\fR is used when the cookie does not specify a path.
\&\f(CW$default_path\fR must be absolute (start with '/'), or it will be ignored.  If
the cookie does not specify a path, and \f(CW$default_path\fR is not provided, then
the default value '/' will be used.
.Sp
Set \f(CW$reject\fR to 1 if you wish to reject cookies based upon the provided
\&\f(CW$default_domain\fR and \f(CW$default_path\fR.  Note that \f(CW$default_domain\fR and 
\&\f(CW$default_path\fR must be specified for \f(CW$reject\fR to actually do something 
meaningful.
.IP "\fBcookie_write\fR" 4
.IX Item "cookie_write"
Params: \f(CW$jar\fR, \e%request, \f(CW$override\fR
.Sp
Return: nothing
.Sp
Goes through the given \f(CW$jar\fR and sets the Cookie header in \f(CW%req\fR pending the 
correct domain and path.  If \f(CW$override\fR is true, then the secure, domain and 
path restrictions of the cookies are ignored and all cookies are essentially
included.
.Sp
Notice: cookie expiration is currently not implemented.  \s-1URL\s0 restriction
comparision is also case-insensitive.
.IP "\fBcookie_get\fR" 4
.IX Item "cookie_get"
Params: \f(CW$jar\fR, \f(CW$name\fR
.Sp
Return: \f(CW@elements\fR
.Sp
Fetch the named cookie from the \f(CW$jar\fR, and return the components.  The
returned items will be an array in the following order:
.Sp
value, domain, path, expire, secure
.Sp
value  = cookie value, should always be non-empty string
domain = domain root for cookie, can be undefined
path   = \s-1URL\s0 path for cookie, should always be a non-empty string
expire = undefined (depreciated, but exists for backwards-compatibility)
secure = whether or not the cookie is limited to HTTPs; value is 0 or 1
.IP "\fBcookie_get_names\fR" 4
.IX Item "cookie_get_names"
Params: \f(CW$jar\fR
.Sp
Return: \f(CW@names\fR
.Sp
Fetch all the cookie names from the jar, which then let you \fBcooke_get()\fR
them individually.
.IP "\fBcookie_get_valid_names\fR" 4
.IX Item "cookie_get_valid_names"
Params: \f(CW$jar\fR, \f(CW$domain\fR, \f(CW$url\fR, \f(CW$ssl\fR
.Sp
Return: \f(CW@names\fR
.Sp
Fetch all the cookie names from the jar which are valid for the given
\&\f(CW$domain\fR, \f(CW$url\fR, and \f(CW$ssl\fR values.  \f(CW$domain\fR should be string scalar of the
target host domain ('www.example.com', etc.).  \f(CW$url\fR should be the absolute 
\&\s-1URL\s0 for the page ('/index.html', '/cgi\-bin/foo.cgi', etc.).  \f(CW$ssl\fR should be 
0 for non-secure cookies, or 1 for all (secure and normal) cookies.  The 
return value is an array of names compatible with \fBcookie_get()\fR.
.IP "\fBcookie_set\fR" 4
.IX Item "cookie_set"
Params: \f(CW$jar\fR, \f(CW$name\fR, \f(CW$value\fR, \f(CW$domain\fR, \f(CW$path\fR, \f(CW$expire\fR, \f(CW$secure\fR
.Sp
Return: nothing
.Sp
Set the named cookie with the provided values into the \f(CW%jar\fR.  \f(CW$name\fR is 
required to be a non-empty string.  \f(CW$value\fR is required, and will delete
the named cookie from the \f(CW$jar\fR if it is an empty string.  \f(CW$domain\fR and
\&\f(CW$path\fR can be strings or undefined.  \f(CW$expire\fR is ignored (but exists
for backwards-compatibility).  \f(CW$secure\fR should be the numeric value of
0 or 1.
.IP "\fBcrawl_new\fR" 4
.IX Item "crawl_new"
Params: \f(CW$START\fR, \f(CW$MAX_DEPTH\fR, \e%request_hash [, \e%tracking_hash ]
.Sp
Return: \f(CW$crawl_object\fR
.Sp
The \fBcrawl_new()\fR functions initializes a crawl object (hash) to the default
values, and then returns it for later use by \fBcrawl()\fR.  \f(CW$START\fR is the starting
\&\s-1URL\s0 (in the form of 'http://www.host.com/url'), and \s-1MAX_DEPTH\s0 is the maximum
number of levels to crawl (the \s-1START URL\s0 counts as 1, so a value of 2 will
crawl the \s-1START URL\s0 and all URLs found on that page).  The request_hash
is a standard initialized request hash to be used for requests; you should
set any authentication information or headers in this hash in order for
the crawler to use them.  The optional tracking_hash lets you supply a
hash for use in tracking \s-1URL\s0 results (otherwise \fBcrawl_new()\fR will allocate
a new anon hash).
.IP "\fBcrawl\fR" 4
.IX Item "crawl"
Params: \f(CW$crawl_object\fR [, \f(CW$START\fR, \f(CW$MAX_DEPTH\fR ]
.Sp
Return: \f(CW$count\fR [ undef on error ]
.Sp
The heart of the crawl package.  Will perform an \s-1HTTP\s0 crawl on the
specified \s-1HOST,\s0 starting at \s-1START URI,\s0 proceeding up to \s-1MAX_DEPTH.\s0
.Sp
Crawl_object needs to be the variable returned by \fBcrawl_new()\fR.  You can
also indirectly call \fBcrawl()\fR via the crawl_object itself:
.Sp
.Vb 1
\&        $crawl_object\->{crawl}\->($START,$MAX_DEPTH)
.Ve
.Sp
Returns the number of URLs actually crawled (not including those skipped).
.IP "\fBdump\fR" 4
.IX Item "dump"
Params: \f(CW$name\fR, \e@array [, \f(CW$name\fR, \e%hash, \f(CW$name\fR, \e$scalar ]
.Sp
Return: \f(CW$code\fR [ undef on error ]
.Sp
The dump function will take the given \f(CW$name\fR and data reference, and
will create an \s-1ASCII\s0 perl code representation suitable for eval'ing
later to recreate the same structure.  \f(CW$name\fR is the name of the variable
that it will be saved as.  Example:
.Sp
.Vb 1
\& $output = LW2::dump(\*(Aqrequest\*(Aq,\e%request);
.Ve
.Sp
\&\s-1NOTE:\s0 \fBdump()\fR creates anonymous structures under the name given.  For
example, if you dump the hash \f(CW%hin\fR under the name 'hin', then when you
eval the dumped code you will need to use %$hin, since \f(CW$hin\fR is now a
*reference* to a hash.
.IP "\fBdump_writefile\fR" 4
.IX Item "dump_writefile"
Params: \f(CW$file\fR, \f(CW$name\fR, \e@array [, \f(CW$name\fR, \e%hash, \f(CW$name\fR, \e@scalar ]
.Sp
Return: 0 if success; 1 if error
.Sp
This calls \fBdump()\fR and saves the output to the specified \f(CW$file\fR.
.Sp
Note: \s-1LW\s0 does not checking on the validity of the file name, it's
creation, or anything of the sort.  Files are opened in overwrite
mode.
.IP "\fBencode_base64\fR" 4
.IX Item "encode_base64"
Params: \f(CW$data\fR [, \f(CW$eol\fR]
.Sp
Return: \f(CW$b64_encoded_data\fR
.Sp
This function does Base64 encoding.  If the binary MIME::Base64 module
is available, it will use that; otherwise, it falls back to an internal
perl version.  The perl version carries the following copyright:
.Sp
.Vb 1
\& Copyright 1995\-1999 Gisle Aas <gisle@aas.no>
.Ve
.Sp
\&\s-1NOTE:\s0 the \f(CW$eol\fR parameter will be inserted every 76 characters.  This is
used to format the data for output on a 80 character wide terminal.
.IP "\fBdecode_base64\fR" 4
.IX Item "decode_base64"
Params: \f(CW$data\fR
.Sp
Return: \f(CW$b64_decoded_data\fR
.Sp
A perl implementation of base64 decoding.  The perl code for this function
was actually taken from an older MIME::Base64 perl module, and bears the 
following copyright:
.Sp
Copyright 1995\-1999 Gisle Aas <gisle@aas.no>
.IP "\fBencode_uri_hex\fR" 4
.IX Item "encode_uri_hex"
Params: \f(CW$data\fR
.Sp
Return: \f(CW$result\fR
.Sp
This function encodes every character (except the / character) with normal 
\&\s-1URL\s0 hex encoding.
.IP "\fBencode_uri_randomhex\fR" 4
.IX Item "encode_uri_randomhex"
Params: \f(CW$data\fR
.Sp
Return: \f(CW$result\fR
.Sp
This function randomly encodes characters (except the / character) with 
normal \s-1URL\s0 hex encoding.
.IP "\fBencode_uri_randomcase\fR" 4
.IX Item "encode_uri_randomcase"
Params: \f(CW$data\fR
.Sp
Return: \f(CW$result\fR
.Sp
This function randomly changes the case of characters in the string.
.IP "\fBencode_unicode\fR" 4
.IX Item "encode_unicode"
Params: \f(CW$data\fR
.Sp
Return: \f(CW$result\fR
.Sp
This function converts a normal string into Windows unicode format
(non-overlong or anything fancy).
.IP "\fBdecode_unicode\fR" 4
.IX Item "decode_unicode"
Params: \f(CW$unicode_string\fR
.Sp
Return: \f(CW$decoded_string\fR
.Sp
This function attempts to decode a unicode (\s-1UTF\-8\s0) string by
converting it into a single-byte-character string.  Overlong 
characters are converted to their standard characters in place; 
non-overlong (aka multi-byte) characters are substituted with the 
0xff; invalid encoding characters are left as-is.
.Sp
Note: this function is useful for dealing with the various unicode
exploits/vulnerabilities found in web servers; it is *not* good for
doing actual \s-1UTF\-8\s0 parsing, since characters over a single byte are
basically dropped/replaced with a placeholder.
.IP "\fBencode_anti_ids\fR" 4
.IX Item "encode_anti_ids"
Params: \e%request, \f(CW$modes\fR
.Sp
Return: nothing
.Sp
encode_anti_ids computes the proper anti-ids encoding/tricks 
specified by \f(CW$modes\fR, and sets up \f(CW%hin\fR in order to use those tricks.  
Valid modes are (the mode numbers are the same as those found in whisker 
1.4):
.RS 4
.IP "1 Encode some of the characters via normal \s-1URL\s0 encoding" 4
.IX Item "1 Encode some of the characters via normal URL encoding"
.PD 0
.IP "2 Insert directory self-references (/./)" 4
.IX Item "2 Insert directory self-references (/./)"
.IP "3 Premature \s-1URL\s0 ending (make it appear the request line is done)" 4
.IX Item "3 Premature URL ending (make it appear the request line is done)"
.ie n .IP "4 Prepend a long random string in the form of ""/string/../URL""" 4
.el .IP "4 Prepend a long random string in the form of ``/string/../URL''" 4
.IX Item "4 Prepend a long random string in the form of /string/../URL"
.IP "5 Add a fake \s-1URL\s0 parameter" 4
.IX Item "5 Add a fake URL parameter"
.IP "6 Use a tab instead of a space as a request spacer" 4
.IX Item "6 Use a tab instead of a space as a request spacer"
.IP "7 Change the case of the \s-1URL\s0 (works against Windows and Novell)" 4
.IX Item "7 Change the case of the URL (works against Windows and Novell)"
.IP "8 Change normal seperators ('/') to Windows version ('\e')" 4
.IX Item "8 Change normal seperators ('/') to Windows version ('')"
.IP "9 Session splicing [\s-1NOTE:\s0 not currently available]" 4
.IX Item "9 Session splicing [NOTE: not currently available]"
.IP "A Use a carriage return (0x0d) as a request spacer" 4
.IX Item "A Use a carriage return (0x0d) as a request spacer"
.IP "B Use binary value 0x0b as a request spacer" 4
.IX Item "B Use binary value 0x0b as a request spacer"
.RE
.RS 4
.PD
.Sp
You can set multiple modes by setting the string to contain all the modes
desired; i.e. \f(CW$modes\fR=\*(L"146\*(R" will use modes 1, 4, and 6.
.RE
.IP "\fB\s-1FORMS FUNCTIONS\s0\fR" 4
.IX Item "FORMS FUNCTIONS"
The goal is to parse the variable, human-readable \s-1HTML\s0 into concrete
structures useable by your program.  The forms functions does do a good job
at making these structures, but I will admit: they are not exactly simple,
and thus not a cinch to work with.  But then again, representing something
as complex as a \s-1HTML\s0 form is not a simple thing either.  I think the
results are acceptable for what's trying to be done.  Anyways...
.Sp
Forms are stored in perl hashes, with elements in the following format:
.Sp
.Vb 1
\& $form{\*(Aqelement_name\*(Aq}=@([ \*(Aqtype\*(Aq, \*(Aqvalue\*(Aq, @params ])
.Ve
.Sp
Thus every element in the hash is an array of anonymous arrays.  The first
array value contains the element type (which is 'select', 'textarea',
\&'button', or an 'input' value of the form 'input\-text', 'input\-hidden',
\&'input\-radio', etc).
.Sp
The second value is the value, if applicable (it could be undef if no
value was specified).  Note that select elements will always have an undef
value\*(--the actual values are in the subsequent options elements.
.Sp
The third value, if defined, is an anonymous array of additional tag
parameters found in the element (like 'onchange=\*(L"blah\*(R"', 'size=\*(L"20\*(R"',
\&'maxlength=\*(L"40\*(R"', 'selected', etc).
.Sp
The array does contain one special element, which is stored in the hash
under a \s-1NULL\s0 character (\*(L"\e0\*(R") key.  This element is of the format:
.Sp
.Vb 1
\& $form{"\e0"}=[\*(Aqname\*(Aq, \*(Aqmethod\*(Aq, \*(Aqaction\*(Aq, @parameters];
.Ve
.Sp
The element is an anonymous array that contains strings of the form's
name, method, and action (values can be undef), and a \f(CW@parameters\fR array
similar to that found in normal elements (above).
.Sp
Accessing individual values stored in the form hash becomes a test of your
perl referencing skills.  Hint: to access the 'value' of the third element
named 'choices', you would need to do:
.Sp
.Vb 1
\& $form{\*(Aqchoices\*(Aq}\->[2]\->[1];
.Ve
.Sp
The '[2]' is the third element (normal array starts with 0), and the
actual value is '[1]' (the type is '[0]', and the parameter array is
\&'[2]').
.IP "\fBforms_read\fR" 4
.IX Item "forms_read"
Params: \e$html_data
.Sp
Return: \e@found_forms
.Sp
This function parses the given \f(CW$html_data\fR into libwhisker form hashes.  
It returns a reference to an array of hash references to the found 
forms.
.IP "\fBforms_write\fR" 4
.IX Item "forms_write"
Params: \e%form_hash
.Sp
Return: \f(CW$html_of_form\fR [undef on error]
.Sp
This function will take the given \f(CW%form\fR hash and compose a generic \s-1HTML\s0
representation of it, formatted with tabs and newlines in order to make it
neat and tidy for printing.
.Sp
Note: this function does *not* escape any special characters that were
embedded in the element values.
.IP "\fBhtml_find_tags\fR" 4
.IX Item "html_find_tags"
Params: \e$data, \e&callback_function [, \f(CW$xml_flag\fR, \f(CW$funcref\fR, \e%tag_map]
.Sp
Return: nothing
.Sp
html_find_tags parses a piece of \s-1HTML\s0 and 'extracts' all found tags,
passing the info to the given callback function.  The callback function 
must accept two parameters: the current tag (as a scalar), and a hash ref 
of all the tag's elements. For example, the tag <a href=\*(L"/file\*(R"> will
pass 'a' as the current tag, and a hash reference which contains
{'href'=>\*(L"/file\*(R"}.
.Sp
The xml_flag, when set, causes the parser to do some extra processing
and checks to accomodate \s-1XML\s0 style tags such as <tag foo=\*(L"bar\*(R"/>.
.Sp
The optional \f(CW%tagmap\fR is a hash of lowercase tag names.  If a tagmap is
supplied, then the parser will only call the callback function if the
tag name exists in the tagmap.
.Sp
The optional \f(CW$funcref\fR variable is passed straight to the callback
function, allowing you to pass flags or references to more complex
structures to your callback function.
.IP "\fBhtml_find_tags_rewrite\fR" 4
.IX Item "html_find_tags_rewrite"
Params: \f(CW$position\fR, \f(CW$length\fR, \f(CW$replacement\fR
.Sp
Return: nothing
.Sp
\&\fBhtml_find_tags_rewrite()\fR is used to 'rewrite' an \s-1HTML\s0 stream from
within an \fBhtml_find_tags()\fR callback function.  In general, you can
think of html_find_tags_rewrite working as:
.Sp
substr(\s-1DATA,\s0 \f(CW$position\fR, \f(CW$length\fR) = \f(CW$replacement\fR
.Sp
Where \s-1DATA\s0 is the current \s-1HTML\s0 string the html parser is using.
The reason you need to use this function and not \fBsubstr()\fR is
because a few internal parser pointers and counters need to be
adjusted to accomodate the changes.
.Sp
If you want to remove a piece of the string, just set the
replacement to an empty string ('').  If you wish to insert a
string instead of overwrite, just set \f(CW$length\fR to 0; your string
will be inserted at the indicated \f(CW$position\fR.
.IP "\fBhtml_link_extractor\fR" 4
.IX Item "html_link_extractor"
Params: \e$html_data
.Sp
Return: \f(CW@urls\fR
.Sp
The \fBhtml_link_extractor()\fR function uses the internal crawl tests to
extract all the \s-1HTML\s0 links from the given \s-1HTML\s0 data stream.
.Sp
Note: \fBhtml_link_extractor()\fR does not unique the returned array of
discovered links, nor does it attempt to remove javascript links
or make the links absolute.  It just extracts every raw link from
the \s-1HTML\s0 stream and returns it.  You'll have to do your own
post-processing.
.IP "\fBhttp_new_request\fR" 4
.IX Item "http_new_request"
Params: \f(CW%parameters\fR
.Sp
Return: \e%request_hash
.Sp
This function basically 'objectifies' the creation of whisker
request hash objects.  You would call it like:
.Sp
.Vb 1
\& $req = http_new_request( host=>\*(Aqwww.example.com\*(Aq, uri=>\*(Aq/\*(Aq )
.Ve
.Sp
where 'host' and 'uri' can be any number of {whisker} hash
control values (see http_init_request for default list).
.IP "\fBhttp_new_response\fR" 4
.IX Item "http_new_response"
Params: [none]
.Sp
Return: \e%response_hash
.Sp
This function basically 'objectifies' the creation of whisker
response hash objects.  You would call it like:
.Sp
.Vb 1
\&        $resp = http_new_response()
.Ve
.IP "\fBhttp_init_request\fR" 4
.IX Item "http_init_request"
Params: \e%request_hash_to_initialize
.Sp
Return: Nothing (modifies input hash)
.Sp
Sets default values to the input hash for use.  Sets the host to
\&'localhost', port 80, request \s-1URI\s0 '/', using \s-1HTTP 1.1\s0 with \s-1GET\s0
method.  The timeout is set to 10 seconds, no proxies are defined, and all
\&\s-1URI\s0 formatting is set to standard \s-1HTTP\s0 syntax.  It also sets the
Connection (Keep-Alive) and User-Agent headers.
.Sp
\&\s-1NOTICE\s0!!  It's important to use http_init_request before calling 
http_do_request, or http_do_request might puke.  Thus, a special magic 
value is placed in the hash to let http_do_request know that the hash has 
been properly initialized.  If you really must 'roll your own' and not use 
http_init_request before you call http_do_request, you will at least need 
to set the \s-1MAGIC\s0 value (amongst other things).
.IP "\fBhttp_do_request\fR" 4
.IX Item "http_do_request"
Params: \e%request, \e%response [, \e%configs]
.Sp
Return: >=1 if error; 0 if no error (also modifies response hash)
.Sp
*THE* core function of libwhisker.  http_do_request actually performs
the \s-1HTTP\s0 request, using the values submitted in \f(CW%request\fR, and placing result
values in \f(CW%response\fR.  This allows you to resubmit \f(CW%request\fR in subsequent 
requests (%response is automatically cleared upon execution).  You can 
submit 'runtime' config directives as \f(CW%configs\fR, which will be spliced into
\&\f(CW$hin\fR{whisker}\->{} before anything else.  That means you can do:
.Sp
LW2::http_do_request(\e%req,\e%resp,{'uri'=>'/cgi\-bin/'});
.Sp
This will set \f(CW$req\fR{whisker}\->{'uri'}='/cgi\-bin/' before execution, and
provides a simple shortcut (note: it does modify \f(CW%req\fR).
.Sp
This function will also retry any requests that bomb out during the 
transaction (but not during the connecting phase).  This is controlled
by the {whisker}\->{retry} value.  Also note that the returned error
message in hout is the *last* error received.  All retry errors are
put into {whisker}\->{retry_errors}, which is an anonymous array.
.Sp
Also note that all \s-1NTLM\s0 auth logic is implemented in \fBhttp_do_request()\fR.
\&\s-1NTLM\s0 requires multiple requests in order to work correctly, and so this
function attempts to wrap that and make it all transparent, so that the
final end result is what's passed to the application.
.Sp
This function will return 0 on success, 1 on \s-1HTTP\s0 protocol error, and 2
on non-recoverable network connection error (you can retry error 1, but
error 2 means that the server is totally unreachable and there's no
point in retrying).
.IP "\fBhttp_req2line\fR" 4
.IX Item "http_req2line"
Params: \e%request, \f(CW$uri_only_switch\fR
.Sp
Return: \f(CW$request\fR
.Sp
req2line is used internally by http_do_request, as well as provides a
convienient way to turn a \f(CW%request\fR configuration into an actual \s-1HTTP\s0 request
line.  If \f(CW$switch\fR is set to 1, then the returned \f(CW$request\fR will be the \s-1URI\s0
only ('/requested/page.html'), versus the entire \s-1HTTP\s0 request ('\s-1GET\s0
/requested/page.html \s-1HTTP/1\s0.0\en\en').  Also, if the 'full_request_override'
whisker config variable is set in \f(CW%hin\fR, then it will be returned instead
of the constructed \s-1URI.\s0
.IP "\fBhttp_resp2line\fR" 4
.IX Item "http_resp2line"
Params: \e%response
.Sp
Return: \f(CW$response\fR
.Sp
http_resp2line provides a convienient way to turn a \f(CW%response\fR hash back 
into the original \s-1HTTP\s0 response line.
.IP "\fBhttp_fixup_request\fR" 4
.IX Item "http_fixup_request"
Params: \f(CW$hash_ref\fR
.Sp
Return: Nothing
.Sp
This function takes a \f(CW%hin\fR hash reference and makes sure the proper 
headers exist (for example, it will add the Host: header, calculate the 
Content-Length: header for \s-1POST\s0 requests, etc).  For standard requests 
(i.e. you want the request to be \s-1HTTP\s0 RFC-compliant), you should call this 
function right before you call http_do_request.
.IP "\fBhttp_reset\fR" 4
.IX Item "http_reset"
Params: Nothing
.Sp
Return: Nothing
.Sp
The http_reset function will walk through the \f(CW%http_host_cache\fR, 
closing all open sockets and freeing \s-1SSL\s0 resources.  It also clears
out the host cache in case you need to rerun everything fresh.
.Sp
Note: if you just want to close a single connection, and you have
a copy of the \f(CW%request\fR hash you used, you should use the \fBhttp_close()\fR
function instead.
.IP "\fBssl_is_available\fR" 4
.IX Item "ssl_is_available"
Params: Nothing
.Sp
Return: \f(CW$boolean\fR [, \f(CW$lib_name\fR, \f(CW$version\fR]
.Sp
The \fBssl_is_available()\fR function will inform you whether \s-1SSL\s0 requests
are allowed, which is dependant on whether the appropriate \s-1SSL\s0
libraries are installed on the machine.  In scalar context, the
function will return 1 or 0.  In array context, the second element
will be the \s-1SSL\s0 library name that is currently being used by \s-1LW2,\s0
and the third elment will be the \s-1SSL\s0 library version number.
Elements two and three (name and version) will be undefined if
called in array context and no \s-1SSL\s0 libraries are available.
.IP "\fBhttp_read_headers\fR" 4
.IX Item "http_read_headers"
Params: \f(CW$stream\fR, \e%in, \e%out
.Sp
Return: \f(CW$result_code\fR, \f(CW$encoding\fR, \f(CW$length\fR, \f(CW$connection\fR
.Sp
Read \s-1HTTP\s0 headers from the given stream, storing the results in \f(CW%out\fR.  On
success, \f(CW$result_code\fR will be 1 and \f(CW$encoding\fR, \f(CW$length\fR, and \f(CW$connection\fR
will hold the values of the Transfer-Encoding, Content-Length, and
Connection headers, respectively.  If any of those headers are not present,
then it will have an 'undef' value.  On an error, the \f(CW$result_code\fR will
be 0 and \f(CW$encoding\fR will contain an error message.
.Sp
This function can be used to parse both request and response headers.
.Sp
Note: if there are multiple Transfer-Encoding, Content-Length, or
Connection headers, then only the last header value is the one returned
by the function.
.IP "\fBhttp_read_body\fR" 4
.IX Item "http_read_body"
Params: \f(CW$stream\fR, \e%in, \e%out, \f(CW$encoding\fR, \f(CW$length\fR
.Sp
Return: 1 on success, 0 on error (and sets \f(CW$hout\fR\->{whisker}\->{error})
.Sp
Read the body from the given stream, placing it in \f(CW$out\fR\->{whisker}\->{data}.
Handles chunked encoding.  Can be used to read \s-1HTTP\s0 (\s-1POST\s0) request or \s-1HTTP\s0
response bodies.  \f(CW$encoding\fR parameter should be lowercase encoding type.
.Sp
\&\s-1NOTE:\s0 \f(CW$out\fR\->{whisker}\->{data} is erased/cleared when this function is called,
leaving {data} to just contain this particular \s-1HTTP\s0 body.
.IP "\fBhttp_construct_headers\fR" 4
.IX Item "http_construct_headers"
Params: \e%in
.Sp
Return: \f(CW$data\fR
.Sp
This function assembles the headers in the given hash into a data
string.
.IP "\fBhttp_close\fR" 4
.IX Item "http_close"
Params: \e%request
.Sp
Return: nothing
.Sp
This function will close any open streams for the given request.
.Sp
Note: in order for \fBhttp_close()\fR to find the right connection, all
original host/proxy/port parameters in \f(CW%request\fR must be the exact
same as when the original request was made.
.IP "\fBhttp_do_request_timeout\fR" 4
.IX Item "http_do_request_timeout"
Params: \e%request, \e%response, \f(CW$timeout\fR
.Sp
Return: \f(CW$result\fR
.Sp
This function is identical to \fBhttp_do_request()\fR, except that it
wraps the entire request in a timeout wrapper.  \f(CW$timeout\fR is the
number of seconds to allow for the entire request to be completed.
.Sp
Note: this function uses \fBalarm()\fR and signals, and thus will only
work on Unix-ish platforms.  It should be safe to call on any
platform though.
.IP "\fBmd5\fR" 4
.IX Item "md5"
Params: \f(CW$data\fR
.Sp
Return: \f(CW$hex_md5_string\fR
.Sp
This function takes a data scalar, and composes a \s-1MD5\s0 hash of it, and 
returns it in a hex ascii string.  It will use the fastest \s-1MD5\s0 function
available.
.IP "\fBmd4\fR" 4
.IX Item "md4"
Params: \f(CW$data\fR
.Sp
Return: \f(CW$hex_md4_string\fR
.Sp
This function takes a data scalar, and composes a \s-1MD4\s0 hash of it, and 
returns it in a hex ascii string.  It will use the fastest \s-1MD4\s0 function
available.
.IP "\fBmultipart_set\fR" 4
.IX Item "multipart_set"
Params: \e%multi_hash, \f(CW$param_name\fR, \f(CW$param_value\fR
.Sp
Return: nothing
.Sp
This function sets the named parameter to the given value within the
supplied multipart hash.
.IP "\fBmultipart_get\fR" 4
.IX Item "multipart_get"
Params: \e%multi_hash, \f(CW$param_name\fR
.Sp
Return: \f(CW$param_value\fR, undef on error
.Sp
This function retrieves the named parameter to the given value within the
supplied multipart hash.  There is a special case where the named
parameter is actually a file\*(--in which case the resulting value will be
\&\*(L"\e0FILE\*(R".  In general, all special values will be prefixed with a \s-1NULL\s0
character.  In order to get a file's info, use \fBmultipart_getfile()\fR.
.IP "\fBmultipart_setfile\fR" 4
.IX Item "multipart_setfile"
Params: \e%multi_hash, \f(CW$param_name\fR, \f(CW$file_path\fR [, \f(CW$filename\fR]
.Sp
Return: undef on error, 1 on success
.Sp
\&\s-1NOTE:\s0 this function does not actually add the contents of \f(CW$file_path\fR into
the \f(CW%multi_hash\fR; instead, \fBmultipart_write()\fR inserts the content when
generating the final request.
.IP "\fBmultipart_getfile\fR" 4
.IX Item "multipart_getfile"
Params: \e%multi_hash, \f(CW$file_param_name\fR
.Sp
Return: \f(CW$path\fR, \f(CW$name\fR ($path=undef on error)
.Sp
multipart_getfile is used to retrieve information for a file
parameter contained in \f(CW%multi_hash\fR.  To use this you would most
likely do:
.Sp
.Vb 1
\& ($path,$fname)=LW2::multipart_getfile(\e%multi,"param_name");
.Ve
.IP "\fBmultipart_boundary\fR" 4
.IX Item "multipart_boundary"
Params: \e%multi_hash [, \f(CW$new_boundary_name\fR]
.Sp
Return: \f(CW$current_boundary_name\fR
.Sp
multipart_boundary is used to retrieve, and optionally set, the
multipart boundary used for the request.
.Sp
\&\s-1NOTE:\s0 the function does no checking on the supplied boundary, so if 
you want things to work make sure it's a legit boundary.  Libwhisker
does *not* prefix it with any '\-\-\-' characters.
.IP "\fBmultipart_write\fR" 4
.IX Item "multipart_write"
Params: \e%multi_hash, \e%request
.Sp
Return: 1 if successful, undef on error
.Sp
multipart_write is used to parse and construct the multipart data
contained in \f(CW%multi_hash\fR, and place it ready to go in the given whisker
hash (%request) structure, to be sent to the server.
.Sp
\&\s-1NOTE:\s0 file contents are read into the final \f(CW%request\fR, so it's possible for
the hash to get *very* large if you have (a) large file(s).
.IP "\fBmultipart_read\fR" 4
.IX Item "multipart_read"
Params: \e%multi_hash, \e%hout_response [, \f(CW$filepath\fR ]
.Sp
Return: 1 if successful, undef on error
.Sp
multipart_read will parse the data contents of the supplied
\&\f(CW%hout_response\fR hash, by passing the appropriate info to
\&\fBmultipart_read_data()\fR.  Please see \fBmultipart_read_data()\fR for more
info on parameters and behaviour.
.Sp
\&\s-1NOTE:\s0 this function will return an error if the given \f(CW%hout_response\fR
Content-Type is not set to \*(L"multipart/form\-data\*(R".
.IP "\fBmultipart_read_data\fR" 4
.IX Item "multipart_read_data"
Params: \e%multi_hash, \e$data, \f(CW$boundary\fR [, \f(CW$filepath\fR ]
.Sp
Return: 1 if successful, undef on error
.Sp
multipart_read_data parses the contents of the supplied data using 
the given boundary and puts the values in the supplied \f(CW%multi_hash\fR.  
Embedded files will *not* be saved unless a \f(CW$filepath\fR is given, which
should be a directory suitable for writing out temporary files.
.Sp
\&\s-1NOTE:\s0 currently only application/octet\-stream is the only supported
file encoding.  All other file encodings will not be parsed/saved.
.IP "\fBmultipart_files_list\fR" 4
.IX Item "multipart_files_list"
Params: \e%multi_hash
.Sp
Return: \f(CW@files\fR
.Sp
multipart_files_list returns an array of parameter names for all
the files that are contained in \f(CW%multi_hash\fR.
.IP "\fBmultipart_params_list\fR" 4
.IX Item "multipart_params_list"
Params: \e%multi_hash
.Sp
Return: \f(CW@params\fR
.Sp
multipart_files_list returns an array of parameter names for all
the regular parameters (non-file) that are contained in \f(CW%multi_hash\fR.
.IP "\fBntlm_new\fR" 4
.IX Item "ntlm_new"
Params: \f(CW$username\fR, \f(CW$password\fR [, \f(CW$domain\fR, \f(CW$ntlm_only\fR]
.Sp
Return: \f(CW$ntlm_object\fR
.Sp
Returns a reference to an array (otherwise known as the 'ntlm object')
which contains the various informations specific to a user/pass combo.
If \f(CW$ntlm_only\fR is set to 1, then only the \s-1NTLM\s0 hash (and not the LanMan
hash) will be generated.  This results in a speed boost, and is typically
fine for using against \s-1IIS\s0 servers.
.Sp
The array contains the following items, in order:
username, password, domain, lmhash(password), ntlmhash(password)
.IP "\fBntlm_decode_challenge\fR" 4
.IX Item "ntlm_decode_challenge"
Params: \f(CW$challenge\fR
.Sp
Return: \f(CW@challenge_parts\fR
.Sp
Splits the supplied challenge into the various parts.  The returned array
contains elements in the following order:
.Sp
unicode_domain, ident, packet_type, domain_len, domain_maxlen,
domain_offset, flags, challenge_token, reserved, empty, raw_data
.IP "\fBntlm_client\fR" 4
.IX Item "ntlm_client"
Params: \f(CW$ntlm_obj\fR [, \f(CW$server_challenge\fR]
.Sp
Return: \f(CW$response\fR
.Sp
\&\fBntlm_client()\fR is responsible for generating the base64\-encoded text you
include in the \s-1HTTP\s0 Authorization header.  If you call \fBntlm_client()\fR
without a \f(CW$server_challenge\fR, the function will return the initial \s-1NTLM\s0
request packet (message packet #1).  You send this to the server, and
take the server's response (message packet #2) and pass that as
\&\f(CW$server_challenge\fR, causing \fBntlm_client()\fR to generate the final response
packet (message packet #3).
.Sp
Note: \f(CW$server_challenge\fR is expected to be base64 encoded.
.IP "\fBget_page\fR" 4
.IX Item "get_page"
Params: \f(CW$url\fR [, \e%request]
.Sp
Return: \f(CW$code\fR, \f(CW$data\fR ($code will be set to undef on error, \f(CW$data\fR will
			contain error message)
.Sp
This function will fetch the page at the given \s-1URL,\s0 and return the \s-1HTTP\s0 response code
and page contents.  Use this in the form of:
($code,$html)=LW2::get_page(\*(L"http://host.com/page.html\*(R")
.Sp
The optional \f(CW%request\fR will be used if supplied.  This allows you to set
headers and other parameters.
.IP "\fBget_page_hash\fR" 4
.IX Item "get_page_hash"
Params: \f(CW$url\fR [, \e%request]
.Sp
Return: \f(CW$hash_ref\fR (undef on no \s-1URL\s0)
.Sp
This function will fetch the page at the given \s-1URL,\s0 and return the whisker
\&\s-1HTTP\s0 response hash.  The return code of the function is set to
\&\f(CW$hash_ref\fR\->{whisker}\->{get_page_hash}, and uses the \fBhttp_do_request()\fR
return values.
.Sp
Note: undef is returned if no \s-1URL\s0 is supplied
.IP "\fBget_page_to_file\fR" 4
.IX Item "get_page_to_file"
Params: \f(CW$url\fR, \f(CW$filepath\fR [, \e%request]
.Sp
Return: \f(CW$code\fR ($code will be set to undef on error)
.Sp
This function will fetch the page at the given \s-1URL,\s0 place the resulting \s-1HTML\s0
in the file specified, and return the \s-1HTTP\s0 response code.  The optional
\&\f(CW%request\fR hash sets the default parameters to be used in the request.
.Sp
\&\s-1NOTE:\s0 libwhisker does not do any file checking; libwhisker will open the
supplied filepath for writing, overwriting any previously-existing files.
Libwhisker does not differentiate between a bad request, and a bad file
open.  If you're having troubles making this function work, make sure
that your \f(CW$filepath\fR is legal and valid, and that you have appropriate
write permissions to create/overwrite that file.
.IP "\fBtime_mktime\fR" 4
.IX Item "time_mktime"
Params: \f(CW$seconds\fR, \f(CW$minutes\fR, \f(CW$hours\fR, \f(CW$day_of_month\fR, \f(CW$month\fR, \f(CW$year_minus_1900\fR
.Sp
Return: \f(CW$seconds\fR [ \-1 on error ]
.Sp
Performs a general mktime calculation with the given time components.
Note that the input parameter values are expected to be in the format
output by localtime/gmtime.  Namely, \f(CW$seconds\fR is 0\-60 (yes, there can
be a leap second value of 60 occasionally), \f(CW$minutes\fR is 0\-59, \f(CW$hours\fR
is 0\-23, \f(CW$days\fR is 1\-31, \f(CW$month\fR is 0\-11, and \f(CW$year\fR is 70\-127.  This
function is limited in that it will not process dates prior to 1970 or
after 2037 (that way 32\-bit time_t overflow calculations aren't required).
.Sp
Additional parameters passed to the function are ignored, so it is
safe to use the full localtime/gmtime output, such as:
.Sp
.Vb 1
\&        $seconds = LW2::time_mktime( localtime( time ) );
.Ve
.Sp
Note: this function does not adjust for time zone, daylight savings
time, etc.  You must do that yourself.
.IP "\fBtime_gmtolocal\fR" 4
.IX Item "time_gmtolocal"
Params: \f(CW$seconds_gmt\fR
.Sp
Return: \f(CW$seconds_local_timezone\fR
.Sp
Takes a seconds value in \s-1UTC/GMT\s0 time and adjusts it to reflect the current
timezone.  This function is slightly expensive; it takes the \fBgmtime()\fR and
\&\fBlocaltime()\fR representations of the current time, calculates the delta 
difference by turning them back into seconds via time_mktime, and then 
applies this delta difference to \f(CW$seconds_gmt\fR.
.Sp
Note that if you give this function a time and subtract the return value from
the original time, you will get the delta value.  At that point, you can just
apply the delta directly and skip calling this function, which is a massive
performance boost.  However, this will cause problems if you have a long
running program which crosses daylight savings time boundaries, as the \s-1DST\s0
adjustment will not be accounted for unless you recalculate the new delta.
.IP "\fBuri_split\fR" 4
.IX Item "uri_split"
Params: \f(CW$uri_string\fR [, \e%request_hash]
.Sp
Return: \f(CW@uri_parts\fR
.Sp
Return an array of the following values, in order:  uri, protocol, host,
port, params, frag, user, password.  Values not defined are given an undef
value.  If a \f(CW%request\fR hash is passed in, then \fBuri_split()\fR will also set 
the appropriate values in the hash.
.Sp
Note:  \fBuri_split()\fR will only set the \f(CW%request\fR hash if the protocol
is \s-1HTTP\s0 or \s-1HTTPS\s0!
.IP "\fBuri_join\fR" 4
.IX Item "uri_join"
Params: \f(CW@vals\fR
.Sp
Return: \f(CW$url\fR
.Sp
Takes the \f(CW@vals\fR array output from http_split_uri, and returns a single 
scalar/string with them joined again, in the form of:
protocol://user:pass@host:port/uri?params#frag
.IP "\fBuri_absolute\fR" 4
.IX Item "uri_absolute"
Params: \f(CW$uri\fR, \f(CW$base_uri\fR [, \f(CW$normalize_flag\fR ]
.Sp
Return: \f(CW$absolute_uri\fR
.Sp
Double checks that the given \f(CW$uri\fR is in absolute form (that is,
\&\*(L"http://host/file\*(R"), and if not (it's in the form \*(L"/file\*(R"), then
it will append the given \f(CW$base_uri\fR to make it absolute.  This
provides a compatibility similar to that found in the \s-1URI\s0
subpackage.
.Sp
If \f(CW$normalize_flag\fR is set to 1, then the output will be passed
through uri_normalize before being returned.
.IP "\fBuri_normalize\fR" 4
.IX Item "uri_normalize"
Params: \f(CW$uri\fR [, \f(CW$fix_windows_slashes\fR ]
.Sp
Return: \f(CW$normalized_uri\fR [ undef on error ]
.Sp
Takes the given \f(CW$uri\fR and does any /./ and /../ dereferencing in
order to come up with the correct absolute \s-1URL.\s0  If the \f(CW$fix_\fR
windows_slashes parameter is set to 1, all \e (back slashes) will
be converted to / (forward slashes).
.Sp
Non\-http/https URIs return an error.
.IP "\fBuri_get_dir\fR" 4
.IX Item "uri_get_dir"
Params: \f(CW$uri\fR
.Sp
Return: \f(CW$uri_directory\fR
.Sp
Will take a \s-1URI\s0 and return the directory base of it, i.e. /rfp/page.php 
will return /rfp/.
.IP "\fBuri_strip_path_parameters\fR" 4
.IX Item "uri_strip_path_parameters"
Params: \f(CW$uri\fR [, \e%param_hash]
.Sp
Return: \f(CW$stripped_uri\fR
.Sp
This function removes all \s-1URI\s0 path parameters of the form
.Sp
.Vb 1
\& /blah1;foo=bar/blah2;baz
.Ve
.Sp
and returns the stripped \s-1URI\s0 ('/blah1/blah2').  If the optional
parameter hash reference is provided, the stripped parameters
are saved in the form of 'blah1'=>'foo=bar', 'blah2'=>'baz'.
.Sp
Note: only the last value of a duplicate name is saved into the 
param_hash, if provided.  So a \f(CW$uri\fR of '/foo;A/foo;B/' will result 
in a single hash entry of 'foo'=>'B'.
.IP "\fBuri_parse_parameters\fR" 4
.IX Item "uri_parse_parameters"
Params: \f(CW$parameter_string\fR [, \f(CW$decode\fR, \f(CW$multi_flag\fR ]
.Sp
Return: \e%parameter_hash
.Sp
This function takes a string in the form of:
.Sp
.Vb 1
\& foo=1&bar=2&baz=3&foo=4
.Ve
.Sp
And parses it into a hash.  In the above example, the element 'foo'
has two values (1 and 4).  If \f(CW$multi_flag\fR is set to 1, then the
\&'foo' hash entry will hold an anonymous array of both values. 
Otherwise, the default is to just contain the last value (in this
case, '4').
.Sp
If \f(CW$decode\fR is set to 1, then normal hex decoding is done on the
characters, where needed (both the name and value are decoded).
.Sp
Note: if a \s-1URL\s0 parameter name appears without a value, then the
value will be set to undef.  E.g. for the string \*(L"foo=1&bar&baz=2\*(R",
the 'bar' hash element will have an undef value.
.IP "\fBuri_escape\fR" 4
.IX Item "uri_escape"
Params: \f(CW$data\fR
.Sp
Return: \f(CW$encoded_data\fR
.Sp
This function encodes the given \f(CW$data\fR so it is safe to be used in URIs.
.IP "\fBuri_unescape\fR" 4
.IX Item "uri_unescape"
Params: \f(CW$encoded_data\fR
.Sp
Return: \f(CW$data\fR
.Sp
This function decodes the given \f(CW$data\fR out of \s-1URI\s0 format.
.IP "\fButils_recperm\fR" 4
.IX Item "utils_recperm"
Params: \f(CW$uri\fR, \f(CW$depth\fR, \e@dir_parts, \e@valid, \e&func, \e%track, \e%arrays, \e&cfunc
.Sp
Return: nothing
.Sp
This is a special function which is used to recursively-permutate through
a given directory listing.  This is really only used by whisker, in order
to traverse down directories, testing them as it goes.  See whisker 2.0 for
exact usage examples.
.IP "\fButils_array_shuffle\fR" 4
.IX Item "utils_array_shuffle"
Params: \e@array
.Sp
Return: nothing
.Sp
This function will randomize the order of the elements in the given array.
.IP "\fButils_randstr\fR" 4
.IX Item "utils_randstr"
Params: [ \f(CW$size\fR, \f(CW$chars\fR ]
.Sp
Return: \f(CW$random_string\fR
.Sp
This function generates a random string between 10 and 20 characters
long, or of \f(CW$size\fR if specified.  If \f(CW$chars\fR is specified, then the
random function picks characters from the supplied string.  For example,
to have a random string of 10 characters, composed of only the characters
\&'abcdef', then you would run:
.Sp
.Vb 1
\& utils_randstr(10,\*(Aqabcdef\*(Aq);
.Ve
.Sp
The default character string is alphanumeric.
.IP "\fButils_port_open\fR" 4
.IX Item "utils_port_open"
Params: \f(CW$host\fR, \f(CW$port\fR
.Sp
Return: \f(CW$result\fR
.Sp
Quick function to attempt to make a connection to the given host and
port.  If a connection was successfully made, function will return true
(1).  Otherwise it returns false (0).
.Sp
Note: this uses standard \s-1TCP\s0 connections, thus is not recommended for use
in port-scanning type applications.  Extremely slow.
.IP "\fButils_lowercase_keys\fR" 4
.IX Item "utils_lowercase_keys"
Params: \e%hash
.Sp
Return: \f(CW$number_changed\fR
.Sp
Will lowercase all the header names (but not values) of the given hash.
.IP "\fButils_find_lowercase_key\fR" 4
.IX Item "utils_find_lowercase_key"
Params: \e%hash, \f(CW$key\fR
.Sp
Return: \f(CW$value\fR, undef on error or not exist
.Sp
Searches the given hash for the \f(CW$key\fR (regardless of case), and
returns the value. If the return value is placed into an array, the
will dereference any multi-value references and return an array of
all values.
.Sp
\&\s-1WARNING\s0!  In scalar context, \f(CW$value\fR can either be a single-value
scalar or an array reference for multiple scalar values.  That means
you either need to check the return value and act appropriately, or
use an array context (even if you only want a single value).  This is
very important, even if you know there are no multi-value hash keys.
This function may still return an array of multiple values even if
all hash keys are single value, since lowercasing the keys could result
in multiple keys matching.  For example, a hash with the values
{ 'Foo'=>'a', 'fOo'=>'b' } technically has two keys with the lowercase
name 'foo', and so this function will either return an array or array
reference with both 'a' and 'b'.
.IP "\fButils_find_key\fR" 4
.IX Item "utils_find_key"
Params: \e%hash, \f(CW$key\fR
.Sp
Return: \f(CW$value\fR, undef on error or not exist
.Sp
Searches the given hash for the \f(CW$key\fR (case-sensitive), and
returns the value. If the return value is placed into an array, the
will dereference any multi-value references and return an array of
all values.
.IP "\fButils_delete_lowercase_key\fR" 4
.IX Item "utils_delete_lowercase_key"
Params: \e%hash, \f(CW$key\fR
.Sp
Return: \f(CW$number_found\fR
.Sp
Searches the given hash for the \f(CW$key\fR (regardless of case), and
deletes the key out of the hash if found.  The function returns
the number of keys found and deleted (since multiple keys can
exist under the names 'Key', 'key', 'keY', '\s-1KEY\s0', etc.).
.IP "\fButils_getline\fR" 4
.IX Item "utils_getline"
Params: \e$data [, \f(CW$resetpos\fR ]
.Sp
Return: \f(CW$line\fR (undef if no more data)
.Sp
Fetches the next \en terminated line from the given data.  Use
the optional \f(CW$resetpos\fR to reset the internal position pointer.
Does *NOT* return trialing \en.
.IP "\fButils_getline_crlf\fR" 4
.IX Item "utils_getline_crlf"
Params: \e$data [, \f(CW$resetpos\fR ]
.Sp
Return: \f(CW$line\fR (undef if no more data)
.Sp
Fetches the next \er\en terminated line from the given data.  Use
the optional \f(CW$resetpos\fR to reset the internal position pointer.
Does *NOT* return trialing \er\en.
.IP "\fButils_save_page\fR" 4
.IX Item "utils_save_page"
Params: \f(CW$file\fR, \e%response
.Sp
Return: 0 on success, 1 on error
.Sp
Saves the data portion of the given whisker \f(CW%response\fR hash to the
indicated file.  Can technically save the data portion of a
\&\f(CW%request\fR hash too.  A file is not written if there is no data.
.Sp
Note: \s-1LW\s0 does not do any special file checking; files are opened
in overwrite mode.
.IP "\fButils_getopts\fR" 4
.IX Item "utils_getopts"
Params: \f(CW$opt_str\fR, \e%opt_results
.Sp
Return: 0 on success, 1 on error
.Sp
This function is a general implementation of GetOpts::Std.  It will
parse \f(CW@ARGV\fR, looking for the options specified in \f(CW$opt_str\fR, and will
put the results in \f(CW%opt_results\fR.  Behavior/parameter values are
similar to GetOpts::Std's \fBgetopts()\fR.
.Sp
Note: this function does *not* support long options (\-\-option),
option grouping (\-opq), or options with immediate values (\-ovalue).
If an option is indicated as having a value, it will take the next
argument regardless.
.IP "\fButils_text_wrapper\fR" 4
.IX Item "utils_text_wrapper"
Params: \f(CW$long_text_string\fR [, \f(CW$crlf\fR, \f(CW$width\fR ]
.Sp
Return: \f(CW$formatted_test_string\fR
.Sp
This is a simple function used to format a long line of text for
display on a typical limited-character screen, such as a unix
shell console.
.Sp
\&\f(CW$crlf\fR defaults to \*(L"\en\*(R", and \f(CW$width\fR defaults to 76.
.IP "\fButils_bruteurl\fR" 4
.IX Item "utils_bruteurl"
Params: \e%req, \f(CW$pre\fR, \f(CW$post\fR, \e@values_in, \e@values_out
.Sp
Return: Nothing (adds to \f(CW@out\fR)
.Sp
Bruteurl will perform a brute force against the host/server specified in
\&\f(CW%req\fR.  However, it will make one request per entry in \f(CW@in\fR, taking the
value and setting \f(CW$hin\fR{'whisker'}\->{'uri'}= \f(CW$pre\fR.value.$post.  Any \s-1URI\s0
responding with an \s-1HTTP 200\s0 or 403 response is pushed into \f(CW@out\fR.  An
example of this would be to brute force usernames, putting a list of
common usernames in \f(CW@in\fR, setting \f(CW$pre\fR='/~' and \f(CW$post\fR='/'.
.IP "\fButils_join_tag\fR" 4
.IX Item "utils_join_tag"
Params: \f(CW$tag_name\fR, \e%attributes
.Sp
Return: \f(CW$tag_string\fR [undef on error]
.Sp
This function takes the \f(CW$tag_name\fR (like 'A') and a hash full of
attributes (like {href=>'http://foo/'}) and returns the 
constructed \s-1HTML\s0 tag string (<A href=\*(L"http://foo\*(R">).
.IP "\fButils_request_clone\fR" 4
.IX Item "utils_request_clone"
Params: \e%from_request, \e%to_request
.Sp
Return: 1 on success, 0 on error
.Sp
This function takes the connection/request\-specific values from the
given from_request hash, and copies them to the to_request hash.
.IP "\fButils_request_fingerprint\fR" 4
.IX Item "utils_request_fingerprint"
Params: \e%request [, \f(CW$hash\fR ]
.Sp
Return: \f(CW$fingerprint\fR [undef on error]
.Sp
This function constructs a 'fingerprint' of the given request by
using a cryptographic hashing function on the constructed original
\&\s-1HTTP\s0 request.
.Sp
Note: \f(CW$hash\fR can be 'md5' (default) or 'md4'.
.IP "\fButils_flatten_lwhash\fR" 4
.IX Item "utils_flatten_lwhash"
Params: \e%lwhash
.Sp
Return: \f(CW$flat_version\fR [undef on error]
.Sp
This function takes a \f(CW%request\fR or \f(CW%response\fR libwhisker hash, and
creates an approximate flat data string of the original request/
response (i.e. before it was parsed into components and placed
into the libwhisker hash).
.IP "\fButils_carp\fR" 4
.IX Item "utils_carp"
Params: [ \f(CW$package_name\fR ]
.Sp
Return: nothing
.Sp
This function acts like Carp's carp function.  It warn's with the file and 
line number of user's code which causes a problem.  It traces up the call 
stack and reports the first function that is not in the \s-1LW2\s0 or optional 
\&\f(CW$package_name\fR package package.
.IP "\fButils_croak\fR" 4
.IX Item "utils_croak"
Params: [ \f(CW$package_name\fR ]
.Sp
Return: nothing
.Sp
This function acts like Carp's croak function.  It die's with the file and 
line number of user's code which causes a problem.  It traces up the call 
stack and reports the first function that is not in the \s-1LW2\s0 or optional 
\&\f(CW$package_name\fR package package.
.SH "SEE ALSO"
.IX Header "SEE ALSO"
\&\s-1LWP\s0
.SH "COPYRIGHT"
.IX Header "COPYRIGHT"
Copyright 2009 Jeff Forristal