NAME¶
HTML::TableExtract - Perl module for extracting the content contained in tables
within an HTML document, either as text or encoded element trees.
SYNOPSIS¶
# Matched tables are returned as table objects; tables can be matched
# using column headers, depth, count within a depth, table tag
# attributes, or some combination of the four.
# Example: Using column header information.
# Assume an HTML document with tables that have "Date", "Price", and
# "Cost" somewhere in a row. The columns beneath those headings are
# what you want to extract. They will be returned in the same order as
# you specified the headers since 'automap' is enabled by default.
use HTML::TableExtract;
$te = HTML::TableExtract->new( headers => [qw(Date Price Cost)] );
$te->parse($html_string);
# Examine all matching tables
foreach $ts ($te->tables) {
print "Table (", join(',', $ts->coords), "):\n";
foreach $row ($ts->rows) {
print join(',', @$row), "\n";
}
}
# Shorthand...top level rows() method assumes the first table found in
# the document if no arguments are supplied.
foreach $row ($te->rows) {
print join(',', @$row), "\n";
}
# Example: Using depth and count information.
# Every table in the document has a unique depth and count tuple, so
# when both are specified it is a unique table. Depth and count both
# begin with 0, so in this case we are looking for a table (depth 2)
# within a table (depth 1) within a table (depth 0, which is the top
# level HTML document). In addition, it must be the third (count 2)
# such instance of a table at that depth.
$te = HTML::TableExtract->new( depth => 2, count => 2 );
$te->parse_file($html_file);
foreach $ts ($te->tables) {
print "Table found at ", join(',', $ts->coords), ":\n";
foreach $row ($ts->rows) {
print " ", join(',', @$row), "\n";
}
}
# Example: Using table tag attributes.
# If multiple attributes are specified, all must be present and equal
# for match to occur.
$te = HTML::TableExtract->new( attribs => { border => 1 } );
$te->parse($html_string);
foreach $ts ($te->tables) {
print "Table with border=1 found at ", join(',', $ts->coords), ":\n";
foreach $row ($ts->rows) {
print " ", join(',', @$row), "\n";
}
}
# Example: Extracting as an HTML::Element tree structure
# Rather than extracting raw text, the html can be converted into a
# tree of element objects. The HTML document is composed of
# HTML::Element objects and the tables are HTML::ElementTable
# structures. Using this, the contents of tables within a document can
# be edited in-place.
use HTML::TableExtract qw(tree);
$te = HTML::TableExtract->new( headers => qw(Fee Fie Foe Fum) );
$te->parse_file($html_file);
$table = $te->first_table_found;
$table_tree = $table->tree;
$table_tree->cell(4,4)->replace_content('Golden Goose');
$table_html = $table_tree->as_HTML;
$table_text = $table_tree->as_text;
$document_tree = $te->tree;
$document_html = $document_tree->as_HTML;
DESCRIPTION¶
HTML::TableExtract is a subclass of HTML::Parser that serves to extract the
information from tables of interest contained within an HTML document. The
information from each extracted table is stored in table objects. Tables can
be extracted as text, HTML, or HTML::ElementTable structures (for in-place
editing or manipulation).
There are currently four constraints available to specify which tables you would
like to extract from a document:
Headers,
Depth,
Count,
and
Attributes.
Headers, the most flexible and adaptive of the techniques, involves
specifying text in an array that you expect to appear above the data in the
tables of interest. Once all headers have been located in a row of that table,
all further cells beneath the columns that matched your headers are extracted.
All other columns are ignored: think of it as vertical slices through a table.
In addition, TableExtract automatically rearranges each row in the same order
as the headers you provided. If you would like to disable this, set
automap to 0 during object creation, and instead rely on the
column_map() method to find out the order in which the headers were
found. Furthermore, TableExtract will automatically compensate for cell span
issues so that columns are really the same columns as you would visually see
in a browser. This behavior can be disabled by setting the
gridmap
parameter to 0. HTML is stripped from the entire textual content of a cell
before header matches are attempted -- unless the
keep_html parameter
was enabled.
Depth and
Count are more specific ways to specify tables in
relation to one another.
Depth represents how deeply a table resides in
other tables. The depth of a top-level table in the document is 0. A table
within a top-level table has a depth of 1, and so on. Each depth can be
thought of as a layer; tables sharing the same depth are on the same layer.
Within each of these layers,
Count represents the order in which a
table was seen at that depth, starting with 0. Providing both a
depth
and a
count will uniquely specify a table within a document.
Attributes match based on the attributes of the html <table> tag,
for example, boder widths or background color.
Each of the
Headers,
Depth,
Count, and
Attributes
specifications are cumulative in their effect on the overall extraction. For
instance, if you specify only a
Depth, then you get all tables at that
depth (note that these could very well reside in separate higher- level tables
throughout the document since depth extends across tables). If you specify
only a
Count, then the tables at that
Count from all depths are
returned (i.e., the
nth occurrence of a table at each depth). If you
only specify
Headers, then you get all tables in the document
containing those column headers. If you have specified multiple constraints of
Headers,
Depth,
Count, and
Attributes, then each
constraint has veto power over whether a particular table is extracted.
If no
Headers,
Depth,
Count, or
Attributes are
specified, then all tables match.
When extracting only text from tables, the text is decoded with HTML::Entities
by default; this can be disabled by setting the
decode parameter to 0.
The default mode of extraction for HTML::TableExtract is raw text or HTML. In
this mode, embedded tables are completely decoupled from one another. In this
case, HTML::TableExtract is a subclass of HTML::Parser:
use HTML::TableExtract;
Alternativevly, tables can be extracted as HTML::ElementTable structures, which
are in turn embedded in an HTML::Element tree representing the entire HTML
document. Embedded tables are not decoupled from one another since this tree
structure must be manitained. In this case, HTML::TableExtract is a subclass
of HTML::TreeBuilder (itself a subclass of HTML:::Parser):
use HTML::TableExtract qw(tree);
In either case, the basic interface for HTML::TableExtract and the resulting
table objects remains the same -- all that changes is what you can do with the
resulting data.
HTML::TableExtract is a subclass of HTML::Parser, and as such inherits all of
its basic methods such as "parse()" and "parse_file()".
During scans, "start()", "end()", and "text()"
are utilized. Feel free to override them, but if you do not eventually invoke
them in the SUPER class with some content, results are not guaranteed.
Advice¶
The main point of this module was to provide a flexible method of extracting
tabular information from HTML documents without relying to heavily on the
document layout. For that reason, I suggest using
Headers whenever
possible -- that way, you are anchoring your extraction on what the document
is trying to communicate rather than some feature of the HTML comprising the
document (other than the fact that the data is contained in a table).
METHODS¶
The following are the top-level methods of the HTML::TableExtract object. Tables
that have matched a query are actually returned as separate objects of type
HTML::TableExtract::Table. These table objects have their own methods,
documented further below.
CONSTRUCTOR¶
- new()
- Return a new HTML::TableExtract object. Valid attributes are:
- headers
- Passed as an array reference, headers specify strings of interest at the
top of columns within targeted tables. They can be either strings or
regular expressions (qr//). If they are strings, they will eventually be
passed through a non-anchored, case-insensitive regular expression, so
regexp special characters are allowed.
The table row containing the headers is not returned, unless
"keep_headers" was specified or you are extracting into an
element tree. In either case the header row can be accessed via the
hrow() method from within the table object.
Columns that are not beneath one of the provided headers will be ignored
unless "slice_columns" was set to 0. Columns will, by default,
be rearranged into the same order as the headers you provide (see the
automap parameter for more information) unless
"slice_columns" is 0.
Additionally, by default columns are considered what you would see visually
beneath that header when the table is rendered in a browser. See the
"gridmap" parameter for more information.
HTML within a header is stripped before the match is attempted, unless the
"keep_html" parameter was specified and
"strip_html_on_match" is false.
- depth
- Specify how embedded in other tables your tables of interest should be.
Top-level tables in the HTML document have a depth of 0, tables within
top-level tables have a depth of 1, and so on.
- count
- Specify which table within each depth you are interested in, beginning
with 0.
- attribs
- Passed as a hash reference, attribs specify attributes of interest within
the HTML <table> tag itself.
- automap
- Automatically applies the ordering reported by column_map() to the
rows returned by rows(). This only makes a difference if you have
specified Headers and they turn out to be in a different order in
the table than what you specified. Automap will rearrange the columns in
the same order as the headers appear. To get the original ordering, you
will need to take another slice of each row using column_map().
automap is enabled by default.
- slice_columns
- Enabled by default, this option controls whether vertical slices are
returned from under headers that match. When disabled, all columns of the
matching table are retained, regardles of whether they had a matching
header above them. Disabling this also disables "automap".
- keep_headers
- Disabled by default, and only applicable when header constraints have been
specified, "keep_headers" will retain the matching header row as
the first row of table data when enabled. This option has no effect if
extracting into an element tree tructure. In any case, the header row is
accessible from the table method "hrow()".
- gridmap
- Controls whether the table contents are returned as a grid or a tree.
ROWSPAN and COLSPAN issues are compensated for, and columns really are
columns. Empty phantom cells are created where they would have been
obscured by ROWSPAN or COLSPAN settings. This really becomes an issue when
extracting columns beneath headers. Enabled by default.
- subtables
- Extract all tables embedded within matched tables.
- decode
- Automatically decode retrieved text with
HTML::Entities::decode_entities(). Enabled by default. Has no
effect if "keep_html" was specified or if extracting into an
element tree structure.
- br_translate
- Translate <br> tags into newlines. Sometimes the remaining text can
be hard to parse if the <br> tag is simply dropped. Enabled by
default. Has no effect if keep_html is enabled or if extracting
into an element tree structure.
- keep_html
- Return the raw HTML contained in the cell, rather than just the visible
text. Embedded tables are not retained in the HTML extracted from a
cell. Patterns for header matches must take into account HTML in the
string if this option is enabled. This option has no effect if extracting
into an elment tree structure.
- strip_html_on_match
- When "keep_html" is enabled, HTML is stripped by default during
attempts at matching header strings (so if "strip_html_on_match"
is not enabled and "keep_html" is, you would have to include
potential HTML tags in the regexp for header matches). Stripped header
tags are replaced with an empty string, e.g. 'hot
d<em>og</em>' would become 'hot dog' before attempting a
match.
- error_handle
- Filehandle where error messages are printed. STDERR by default.
- debug
- Prints some debugging information to STDERR, more for higher values. If
"error_handle" was provided, messages are printed there rather
than STDERR.
REGULAR METHODS¶
The following methods are invoked directly from an HTML::TableExtract object.
- depths()
- Returns all depths that contained matched tables in the document.
- counts($depth)
- For a particular depth, returns all counts that contained matched
tables.
- table($depth, $count)
- For a particular depth and count, return the table object for the table
found, if any.
- tables()
- Return table objects for all tables that matched. Returns an empty list if
no tables matched.
- first_table_found()
- Return the table state object for the first table matched in the document.
Returns undef if no tables were matched.
- current_table()
- Returns the current table object while parsing the HTML. Only useful if
you're messing around with overriding HTML::Parser methods.
- tree()
- If the module was invoked in tree extraction mode, returns a reference to
the top node of the HTML::Element tree structure for the entire document
(which includes, ultimately, all tables within the document).
- tables_report([$show_content, $col_sep])
- Return a string summarizing extracted tables, along with their depth and
count. Optionally takes a $show_content flag which will dump the extracted
contents of each table as well with columns separated by $col_sep. Default
$col_sep is ':'.
- tables_dump([$show_content, $col_sep])
- Same as "tables_report()" except dump the information to
STDOUT.
- start
- end
- text
- These are the hooks into HTML::Parser. If you want to subclass this module
and have things work, you must at some point call these with content.
DEPRECATED METHODS¶
Tables used to be called 'table states'. Accordingly, the following methods
still work but have been deprecated:
- table_state()
- Is now table()
- table_states()
- Is now tables()
- first_table_state_found()
- Is now first_table_found()
TABLE METHODS¶
The following methods are invoked from an HTML::TableExtract::Table object, such
as those returned from the "tables()" method.
- rows()
- Return all rows within a matched table. Each row returned is a reference
to an array containing the text, HTML, or reference to the HTML::Element
object of each cell depending the mode of extraction. Tables with rowspan
or colspan attributes will have some cells containing undef. Returns a
list or a reference to an array depending on context.
- columns()
- Return all columns within a matched table. Each column returned is a
reference to an array containing the text, HTML, or reference to
HTML::Element object of each cell depending on the mode of extraction.
Tables with rowspan or colspan attributes will have some cells containing
undef.
- row($row)
- Return a particular row from within a matched table either as a list or an
array reference, depending on context.
- column($col)
- Return a particular column from within a matched table as a list or an
array reference, depending on context.
- cell($row,$col)
- Return a particular item from within a matched table, whether it be the
text, HTML, or reference to the HTML::Element object of that cell,
depending on the mode of extraction. If the cell was covered due to
rowspan or colspan effects, will return undef.
- space($row,$col)
- The same as cell(), except in cases where the given coordinates
were covered due to rowspan or colspan issues, in which case the content
of the covering cell is returned rather than undef.
- depth()
- Return the depth at which this table was found.
- count()
- Return the count for this table within the depth it was found.
- coords()
- Return depth and count in a list.
- tree()
- If the module was invoked in tree extraction mode, this accessor provides
a reference to the HTML::ElementTable structure encompassing the
table.
- hrow()
- Returns the header row as a list when headers were specified as a
constraint. If "keep_headers" was specified initially, this is
equivalent to the first row returned by the "rows()"
method.
- column_map()
- Return the order (via indices) in which the provided headers were found.
These indices can be used as slices on rows to either order the rows in
the same order as headers or restore the rows to their natural order,
depending on whether the rows have been pre-adjusted using the
automap parameter.
- lineage()
- Returns the path of matched tables that led to matching this table. The
path is a list of array refs containing depth, count, row, and column
values for each ancestor table involved. Note that corresponding table
objects will not exist for ancestral tables that did not match specified
constraints.
As mentioned above, HTML::TableExtract can be invoked in 'tree' mode where the
resulting HTML and extracted tables are encoded in HTML::Element tree
structures:
use HTML::TableExtract 'tree';
There are a number of things to take note of while using this mode. The entire
HTML document is encoded into an HTML::Element tree. Each table is part of
this structure, but nevertheless is tracked separately via an
HTML::ElementTable structure, which is a specialized form of HTML::Element
tree.
The HTML::ElementTable objects are accessible by invoking the
tree()
method from within each table object returned by HTML::TableExtract. The
HTML::ElementTable objects have their own
row(),
col(), and
cell() methods (among others). These are not to be confused with the
row() and
column() methods provided by the
HTML::TableExtract::Table objects.
For example, the
row() method from HTML::ElementTable will provide a
reference to a 'glob' of all the elements in that row. Actions (such as
setting attributes) performed on that row reference will affect all elements
within that row. On the other hand, the
row() method from the
HTML::TableExtract::Table object will return an array (either by reference or
list, depending on context) of the contents of each cell within the row. In
tree mode, the content is represented by individual references to each cell --
these are references to the same HTML::Element objects that reside in the
HTML::Element tree.
The
cell() methods provided in both cases will therefore return
references to the same object. The exception to this is when a 'cell' in the
table grid was originally 'covered' due to rowspan or colspan issues -- in
this case the cell content will be undef. Likewise, the
row() or
column() methods from HTML::TableExtract::Table objects will return
arrays potentially containing a mixture of object references and undefs. If
you're going to be doing lots of manipulation of the table elements, it might
be more efficient to access them via the methods provided by the
HTML::ElementTable object instead. See HTML::ElementTable for more information
on how to manipulate those objects.
An alternative to the
cell() method in HTML::TableExtract::Table is the
space() method. It is largely similar to
cell(), except when
given coordinates of a cell that was covered due to rowspan or colspan
effects, it will return the contents of the cell that was covering that space
rather than undef. So if, for example, cell (0,0) had a rowspan of 2 and
colspan of 2, cell(1,1) would return undef and space(1,1) would return the
same content as cell(0,0) or space(0,0).
REQUIRES¶
HTML::Parser(3),
HTML::Entities(3)
OPTIONALLY REQUIRES¶
HTML::TreeBuilder(3),
HTML::ElementTable(3)
AUTHOR¶
Matthew P. Sisk, <
sisk@mojotoad.com>
COPYRIGHT¶
Copyright (c) 2000-2006 Matthew P. Sisk. All rights reserved. All wrongs
revenged. This program is free software; you can redistribute it and/or modify
it under the same terms as Perl itself.
SEE ALSO¶
HTML::Parser(3),
HTML::TreeBuilder(3),
HTML::ElementTable(3),
perl(1).