.\" Automatically generated by Pod::Man 4.14 (Pod::Simple 3.42)
.\"
.\" Standard preamble:
.\" ========================================================================
.de Sp \" Vertical space (when we can't use .PP)
.if t .sp .5v
.if n .sp
..
.de Vb \" Begin verbatim text
.ft CW
.nf
.ne \\$1
..
.de Ve \" End verbatim text
.ft R
.fi
..
.\" Set up some character translations and predefined strings.  \*(-- will
.\" give an unbreakable dash, \*(PI will give pi, \*(L" will give a left
.\" double quote, and \*(R" will give a right double quote.  \*(C+ will
.\" give a nicer C++.  Capital omega is used to do unbreakable dashes and
.\" therefore won't be available.  \*(C` and \*(C' expand to `' in nroff,
.\" nothing in troff, for use with C<>.
.tr \(*W-
.ds C+ C\v'-.1v'\h'-1p'\s-2+\h'-1p'+\s0\v'.1v'\h'-1p'
.ie n \{\
.    ds -- \(*W-
.    ds PI pi
.    if (\n(.H=4u)&(1m=24u) .ds -- \(*W\h'-12u'\(*W\h'-12u'-\" diablo 10 pitch
.    if (\n(.H=4u)&(1m=20u) .ds -- \(*W\h'-12u'\(*W\h'-8u'-\"  diablo 12 pitch
.    ds L" ""
.    ds R" ""
.    ds C` ""
.    ds C' ""
'br\}
.el\{\
.    ds -- \|\(em\|
.    ds PI \(*p
.    ds L" ``
.    ds R" ''
.    ds C`
.    ds C'
'br\}
.\"
.\" Escape single quotes in literal strings from groff's Unicode transform.
.ie \n(.g .ds Aq \(aq
.el       .ds Aq '
.\"
.\" If the F register is >0, we'll generate index entries on stderr for
.\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index
.\" entries marked with X<> in POD.  Of course, you'll have to process the
.\" output yourself in some meaningful fashion.
.\"
.\" Avoid warning from groff about undefined register 'F'.
.de IX
..
.nr rF 0
.if \n(.g .if rF .nr rF 1
.if (\n(rF:(\n(.g==0)) \{\
.    if \nF \{\
.        de IX
.        tm Index:\\$1\t\\n%\t"\\$2"
..
.        if !\nF==2 \{\
.            nr % 0
.            nr F 2
.        \}
.    \}
.\}
.rr rF
.\" ========================================================================
.\"
.IX Title "SQL::SplitStatement::Tokenizer 3pm"
.TH SQL::SplitStatement::Tokenizer 3pm "2022-10-16" "perl v5.34.0" "User Contributed Perl Documentation"
.\" For nroff, turn off justification.  Always turn off hyphenation; it makes
.\" way too many mistakes in technical documents.
.if n .ad l
.nh
.SH "NAME"
SQL::SplitStatement::Tokenizer \- A simple SQL tokenizer.
.SH "SYNOPSIS"
.IX Header "SYNOPSIS"
.Vb 1
\& use SQL::SplitStatement::Tokenizer qw(tokenize_sql);
\&
\& my $query= q{SELECT 1 + 1};
\& my @tokens= tokenize_sql($query);
\&
\& # @tokens now contains (\*(AqSELECT\*(Aq, \*(Aq \*(Aq, \*(Aq1\*(Aq, \*(Aq \*(Aq, \*(Aq+\*(Aq, \*(Aq \*(Aq, \*(Aq1\*(Aq)
.Ve
.SH "DESCRIPTION"
.IX Header "DESCRIPTION"
SQL::SplitStatement::Tokenizer is a simple tokenizer for \s-1SQL\s0 queries. It does
not claim to be a parser or query verifier. It just creates sane tokens from a
valid \s-1SQL\s0 query.
.PP
It supports \s-1SQL\s0 with comments like:
.PP
.Vb 3
\& \-\- This query is used to insert a message into
\& \-\- logs table
\& INSERT INTO log (application, message) VALUES (?, ?)
.Ve
.PP
Also supports \f(CW\*(Aq\*(Aq\fR, \f(CW""\fR and \f(CW\*(C`\e\*(Aq\*(C'\fR escaping methods, so tokenizing queries
like the one below should not be a problem:
.PP
.Vb 2
\& INSERT INTO log (application, message)
\& VALUES (\*(Aqmyapp\*(Aq, \*(AqHey, this is a \*(Aq\*(Aqsingle quoted string\*(Aq\*(Aq!\*(Aq)
.Ve
.SH "API"
.IX Header "API"
.IP "tokenize_sql" 4
.IX Item "tokenize_sql"
.Vb 1
\&    use SQL::SplitStatement::Tokenizer qw(tokenize_sql);
\&
\&    my @tokens = tokenize_sql($query);
\&    my $tokens = tokenize_sql($query);
\&
\&    $tokens = tokenize_sql( $query, $remove_white_tokens );
.Ve
.Sp
\&\f(CW\*(C`tokenize_sql\*(C'\fR can be imported to current namespace on request. It receives a
\&\s-1SQL\s0 query, and returns an array of tokens if called in list context, or an
arrayref if called in scalar context.
.Sp
If \f(CW$remove_white_tokens\fR is true, white spaces only tokens will be removed from
result.
.SH "ACKNOWLEDGEMENTS"
.IX Header "ACKNOWLEDGEMENTS"
.IP "\(bu" 4
Igor Sutton Lopes for writing SQL::Tokenizer, which this was forked from.
.IP "\(bu" 4
Evan Harris, for implementing Shell comment style and \s-1SQL\s0 operators.
.IP "\(bu" 4
Charlie Hills, for spotting a lot of important issues I haven't thought.
.IP "\(bu" 4
Jonas Kramer, for fixing MySQL quoted strings and treating dot as punctuation character correctly.
.IP "\(bu" 4
Emanuele Zeppieri, for asking to fix SQL::Tokenizer to support dollars as well.
.IP "\(bu" 4
Nigel Metheringham, for extending the dollar signal support.
.IP "\(bu" 4
Devin Withers, for making it not choke on \s-1CR+LF\s0 in comments.
.IP "\(bu" 4
Luc Lanthier, for simplifying the regex and make it not choke on backslashes.
.SH "AUTHOR"
.IX Header "AUTHOR"
Copyright (c) 2007, 2008, 2009, 2010, 2011 Igor Sutton Lopes \*(L"<IZUT@cpan.org>\*(R". All rights
reserved.
.PP
Copyright (c) 2021 Veesh Goldman \*(L"<veesh@cpan.org>\*(R"
.PP
This module is free software; you can redistribute it and/or modify it under
the same terms as Perl itself.