.\" Automatically generated by Pod::Man 4.09 (Pod::Simple 3.35)
.\"
.\" Standard preamble:
.\" ========================================================================
.de Sp \" Vertical space (when we can't use .PP)
.if t .sp .5v
.if n .sp
..
.de Vb \" Begin verbatim text
.ft CW
.nf
.ne \\$1
..
.de Ve \" End verbatim text
.ft R
.fi
..
.\" Set up some character translations and predefined strings.  \*(-- will
.\" give an unbreakable dash, \*(PI will give pi, \*(L" will give a left
.\" double quote, and \*(R" will give a right double quote.  \*(C+ will
.\" give a nicer C++.  Capital omega is used to do unbreakable dashes and
.\" therefore won't be available.  \*(C` and \*(C' expand to `' in nroff,
.\" nothing in troff, for use with C<>.
.tr \(*W-
.ds C+ C\v'-.1v'\h'-1p'\s-2+\h'-1p'+\s0\v'.1v'\h'-1p'
.ie n \{\
.    ds -- \(*W-
.    ds PI pi
.    if (\n(.H=4u)&(1m=24u) .ds -- \(*W\h'-12u'\(*W\h'-12u'-\" diablo 10 pitch
.    if (\n(.H=4u)&(1m=20u) .ds -- \(*W\h'-12u'\(*W\h'-8u'-\"  diablo 12 pitch
.    ds L" ""
.    ds R" ""
.    ds C` ""
.    ds C' ""
'br\}
.el\{\
.    ds -- \|\(em\|
.    ds PI \(*p
.    ds L" ``
.    ds R" ''
.    ds C`
.    ds C'
'br\}
.\"
.\" Escape single quotes in literal strings from groff's Unicode transform.
.ie \n(.g .ds Aq \(aq
.el       .ds Aq '
.\"
.\" If the F register is >0, we'll generate index entries on stderr for
.\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index
.\" entries marked with X<> in POD.  Of course, you'll have to process the
.\" output yourself in some meaningful fashion.
.\"
.\" Avoid warning from groff about undefined register 'F'.
.de IX
..
.if !\nF .nr F 0
.if \nF>0 \{\
.    de IX
.    tm Index:\\$1\t\\n%\t"\\$2"
..
.    if !\nF==2 \{\
.        nr % 0
.        nr F 2
.    \}
.\}
.\"
.\" Accent mark definitions (@(#)ms.acc 1.5 88/02/08 SMI; from UCB 4.2).
.\" Fear.  Run.  Save yourself.  No user-serviceable parts.
.    \" fudge factors for nroff and troff
.if n \{\
.    ds #H 0
.    ds #V .8m
.    ds #F .3m
.    ds #[ \f1
.    ds #] \fP
.\}
.if t \{\
.    ds #H ((1u-(\\\\n(.fu%2u))*.13m)
.    ds #V .6m
.    ds #F 0
.    ds #[ \&
.    ds #] \&
.\}
.    \" simple accents for nroff and troff
.if n \{\
.    ds ' \&
.    ds ` \&
.    ds ^ \&
.    ds , \&
.    ds ~ ~
.    ds /
.\}
.if t \{\
.    ds ' \\k:\h'-(\\n(.wu*8/10-\*(#H)'\'\h"|\\n:u"
.    ds ` \\k:\h'-(\\n(.wu*8/10-\*(#H)'\`\h'|\\n:u'
.    ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'^\h'|\\n:u'
.    ds , \\k:\h'-(\\n(.wu*8/10)',\h'|\\n:u'
.    ds ~ \\k:\h'-(\\n(.wu-\*(#H-.1m)'~\h'|\\n:u'
.    ds / \\k:\h'-(\\n(.wu*8/10-\*(#H)'\z\(sl\h'|\\n:u'
.\}
.    \" troff and (daisy-wheel) nroff accents
.ds : \\k:\h'-(\\n(.wu*8/10-\*(#H+.1m+\*(#F)'\v'-\*(#V'\z.\h'.2m+\*(#F'.\h'|\\n:u'\v'\*(#V'
.ds 8 \h'\*(#H'\(*b\h'-\*(#H'
.ds o \\k:\h'-(\\n(.wu+\w'\(de'u-\*(#H)/2u'\v'-.3n'\*(#[\z\(de\v'.3n'\h'|\\n:u'\*(#]
.ds d- \h'\*(#H'\(pd\h'-\w'~'u'\v'-.25m'\f2\(hy\fP\v'.25m'\h'-\*(#H'
.ds D- D\\k:\h'-\w'D'u'\v'-.11m'\z\(hy\v'.11m'\h'|\\n:u'
.ds th \*(#[\v'.3m'\s+1I\s-1\v'-.3m'\h'-(\w'I'u*2/3)'\s-1o\s+1\*(#]
.ds Th \*(#[\s+2I\s-2\h'-\w'I'u*3/5'\v'-.3m'o\v'.3m'\*(#]
.ds ae a\h'-(\w'a'u*4/10)'e
.ds Ae A\h'-(\w'A'u*4/10)'E
.    \" corrections for vroff
.if v .ds ~ \\k:\h'-(\\n(.wu*9/10-\*(#H)'\s-2\u~\d\s+2\h'|\\n:u'
.if v .ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'\v'-.4m'^\v'.4m'\h'|\\n:u'
.    \" for low resolution devices (crt and lpr)
.if \n(.H>23 .if \n(.V>19 \
\{\
.    ds : e
.    ds 8 ss
.    ds o a
.    ds d- d\h'-1'\(ga
.    ds D- D\h'-1'\(hy
.    ds th \o'bp'
.    ds Th \o'LP'
.    ds ae ae
.    ds Ae AE
.\}
.rm #[ #] #H #V #F C
.\" ========================================================================
.\"
.IX Title "Lucy::Docs::IRTheory 3pm"
.TH Lucy::Docs::IRTheory 3pm "2017-08-02" "perl v5.26.0" "User Contributed Perl Documentation"
.\" For nroff, turn off justification.  Always turn off hyphenation; it makes
.\" way too many mistakes in technical documents.
.if n .ad l
.nh
.SH "NAME"
Lucy::Docs::IRTheory \- Crash course in information retrieval.
.SH "ABSTRACT"
.IX Header "ABSTRACT"
Just enough Information Retrieval theory to find your way around Apache Lucy.
.SH "Terminology"
.IX Header "Terminology"
Lucy uses some terminology from the field of information retrieval which
may be unfamiliar to many users.  \*(L"Document\*(R" and \*(L"term\*(R" mean pretty much what
you'd expect them to, but others such as \*(L"posting\*(R" and \*(L"inverted index\*(R" need a
formal introduction:
.IP "\(bu" 4
\&\fIdocument\fR \- An atomic unit of retrieval.
.IP "\(bu" 4
\&\fIterm\fR \- An attribute which describes a document.
.IP "\(bu" 4
\&\fIposting\fR \- One term indexing one document.
.IP "\(bu" 4
\&\fIterm list\fR \- The complete list of terms which describe a document.
.IP "\(bu" 4
\&\fIposting list\fR \- The complete list of documents which a term indexes.
.IP "\(bu" 4
\&\fIinverted index\fR \- A data structure which maps from terms to documents.
.PP
Since Lucy is a practical implementation of \s-1IR\s0 theory, it loads these
abstract, distilled definitions down with useful traits.  For instance, a
\&\*(L"posting\*(R" in its most rarefied form is simply a term-document pairing; in
Lucy, the class Lucy::Index::Posting::MatchPosting fills this
role.  However, by associating additional information with a posting like the
number of times the term occurs in the document, we can turn it into a
ScorePosting, making it possible
to rank documents by relevance rather than just list documents which happen to
match in no particular order.
.SH "TF/IDF ranking algorithm"
.IX Header "TF/IDF ranking algorithm"
Lucy uses a variant of the well-established \*(L"Term Frequency / Inverse
Document Frequency\*(R" weighting scheme.  A thorough treatment of \s-1TF/IDF\s0 is too
ambitious for our present purposes, but in a nutshell, it means that...
.IP "\(bu" 4
in a search for \f(CW\*(C`skate park\*(C'\fR, documents which score well for the
comparatively rare term \f(CW\*(C`skate\*(C'\fR will rank higher than documents which score
well for the more common term \f(CW\*(C`park\*(C'\fR.
.IP "\(bu" 4
a 10\-word text which has one occurrence each of both \f(CW\*(C`skate\*(C'\fR and \f(CW\*(C`park\*(C'\fR will
rank higher than a 1000\-word text which also contains one occurrence of each.
.PP
A web search for \*(L"tf idf\*(R" will turn up many excellent explanations of the
algorithm.