table of contents
libwget-robots(3) | wget2 | libwget-robots(3) |
NAME¶
libwget-robots - Robots Exclusion file parser
SYNOPSIS¶
Functions¶
wget_robots_t * wget_robots_parse (const char *data,
const char *client)
void wget_robots_free (wget_robots_t **robots)
Detailed Description¶
The purpose of this set of functions is to parse a Robots Exlusion Standard file into a data structure for easy access.
Function Documentation¶
wget_robots_t* wget_robots_parse (const char * data, const char * client)¶
Parameters
client Name of the client / user-agent
Returns
The function parses the robots.txt data and returns a ROBOTS structure including a list of the disallowed paths and including a list of the sitemap files.
The ROBOTS structure has to be freed by calling wget_robots_free().
void wget_robots_free (wget_robots_t ** robots)¶
Parameters
wget_robots_free() free's the formerly allocated ROBOTS structure.
Author¶
Generated automatically by Doxygen for wget2 from the source code.
Tue Jan 26 2021 | Version 1.99.1 |