.TH "libwget-robots" 3 "Mon Aug 7 2017" "Version 1.0.0" "wget2" \" -*- nroff -*- .ad l .nh .SH NAME libwget-robots .SH SYNOPSIS .br .PP .SS "Functions" .in +1c .ti -1c .RI "\fBROBOTS\fP * \fBwget_robots_parse\fP (const char *data, const char *client)" .br .ti -1c .RI "void \fBwget_robots_free\fP (\fBROBOTS\fP **robots)" .br .in -1c .SH "Detailed Description" .PP The purpose of this set of functions is to parse a Robots Exlusion Standard file into a data structure for easy access\&. .SH "Function Documentation" .PP .SS "\fBROBOTS\fP* wget_robots_parse (const char * data, const char * client)" .PP \fBParameters:\fP .RS 4 \fIdata\fP Memory with robots\&.txt content (with trailing 0-byte) .br \fIclient\fP Name of the client / user-agent .RE .PP \fBReturns:\fP .RS 4 Return an allocated \fBROBOTS\fP structure or NULL on error .RE .PP The function parses the robots\&.txt \fCdata\fP and returns a \fBROBOTS\fP structure including a list of the disallowed paths and including a list of the sitemap files\&. .PP The \fBROBOTS\fP structure has to be freed by calling \fBwget_robots_free()\fP\&. .SS "void wget_robots_free (\fBROBOTS\fP ** robots)" .PP \fBParameters:\fP .RS 4 \fIrobots\fP Pointer to Pointer to \fBROBOTS\fP structure .RE .PP \fBwget_robots_free()\fP free's the formerly allocated \fBROBOTS\fP structure\&. .SH "Author" .PP Generated automatically by Doxygen for wget2 from the source code\&.