table of contents
libwget-robots(3) | wget2 | libwget-robots(3) |
NAME¶
libwget-robots
SYNOPSIS¶
Functions¶
ROBOTS * wget_robots_parse (const char *data, const
char *client)
void wget_robots_free (ROBOTS **robots)
Detailed Description¶
The purpose of this set of functions is to parse a Robots Exlusion Standard file into a data structure for easy access.
Function Documentation¶
ROBOTS* wget_robots_parse (const char * data, const char * client)¶
Parameters:
client Name of the client / user-agent
Returns:
The function parses the robots.txt data and returns a ROBOTS structure including a list of the disallowed paths and including a list of the sitemap files.
The ROBOTS structure has to be freed by calling wget_robots_free().
void wget_robots_free (ROBOTS ** robots)¶
Parameters:
wget_robots_free() free's the formerly allocated ROBOTS structure.
Author¶
Generated automatically by Doxygen for wget2 from the source code.
Mon Aug 7 2017 | Version 1.0.0 |