1 .TH PCRE2TEST 1 "21 July 2018" "PCRE 10.32"
3 pcre2test - a program for testing Perl-compatible regular expressions.
7 .B pcre2test "[options] [input file [output file]]"
9 \fBpcre2test\fP is a test program for the PCRE2 regular expression libraries,
10 but it can also be used for experimenting with regular expressions. This
11 document describes the features of the test program; for details of the regular
12 expressions themselves, see the
16 documentation. For details of the PCRE2 library function calls and their
23 The input for \fBpcre2test\fP is a sequence of regular expression patterns and
24 subject strings to be matched. There are also command lines for setting
25 defaults and controlling some special actions. The output shows the result of
26 each match attempt. Modifiers on external or internal command lines, the
27 patterns, and the subject lines specify PCRE2 function options, control how the
28 subject is processed, and what output is produced.
30 As the original fairly simple PCRE library evolved, it acquired many different
31 features, and as a result, the original \fBpcretest\fP program ended up with a
32 lot of options in a messy, arcane syntax for testing all the features. The
33 move to the new PCRE2 API provided an opportunity to re-implement the test
34 program as \fBpcre2test\fP, with a cleaner modifier syntax. Nevertheless, there
35 are still many obscure modifiers, some of which are specifically designed for
36 use in conjunction with the test script and data files that are distributed as
37 part of PCRE2. All the modifiers are documented here, some without much
38 justification, but many of them are unlikely to be of use except when testing
42 .SH "PCRE2's 8-BIT, 16-BIT AND 32-BIT LIBRARIES"
45 Different versions of the PCRE2 library can be built to support character
46 strings that are encoded in 8-bit, 16-bit, or 32-bit code units. One, two, or
47 all three of these libraries may be simultaneously installed. The
48 \fBpcre2test\fP program can be used to test all the libraries. However, its own
49 input and output are always in 8-bit format. When testing the 16-bit or 32-bit
50 libraries, patterns and subject strings are converted to 16-bit or 32-bit
51 format before being passed to the library functions. Results are converted back
52 to 8-bit code units for output.
54 In the rest of this document, the names of library functions and structures
55 are given in generic form, for example, \fBpcre_compile()\fP. The actual
56 names used in the libraries have a suffix _8, _16, or _32, as appropriate.
59 .\" HTML <a name="inputencoding"></a>
63 Input to \fBpcre2test\fP is processed line by line, either by calling the C
64 library's \fBfgets()\fP function, or via the \fBlibreadline\fP library. In some
65 Windows environments character 26 (hex 1A) causes an immediate end of file, and
66 no further data is read, so this character should be avoided unless you really
69 The input is processed using using C's string functions, so must not
70 contain binary zeros, even though in Unix-like environments, \fBfgets()\fP
71 treats any bytes other than newline as data characters. An error is generated
72 if a binary zero is encountered. By default subject lines are processed for
73 backslash escapes, which makes it possible to include any data value in strings
74 that are passed to the library for matching. For patterns, there is a facility
75 for specifying some or all of the 8-bit input characters as hexadecimal pairs,
76 which makes it possible to include binary zeros.
79 .SS "Input for the 16-bit and 32-bit libraries"
82 When testing the 16-bit or 32-bit libraries, there is a need to be able to
83 generate character code points greater than 255 in the strings that are passed
84 to the library. For subject lines, backslash escapes can be used. In addition,
85 when the \fButf\fP modifier (see
86 .\" HTML <a href="#optionmodifiers">
88 "Setting compilation options"
90 below) is set, the pattern and any following subject lines are interpreted as
91 UTF-8 strings and translated to UTF-16 or UTF-32 as appropriate.
93 For non-UTF testing of wide characters, the \fButf8_input\fP modifier can be
94 used. This is mutually exclusive with \fButf\fP, and is allowed only in 16-bit
95 or 32-bit mode. It causes the pattern and following subject lines to be treated
96 as UTF-8 according to the original definition (RFC 2279), which allows for
97 character values up to 0x7fffffff. Each character is placed in one 16-bit or
98 32-bit code unit (in the 16-bit case, values greater than 0xffff cause an error
101 UTF-8 (in its original definition) is not capable of encoding values greater
102 than 0x7fffffff, but such values can be handled by the 32-bit library. When
103 testing this library in non-UTF mode with \fButf8_input\fP set, if any
104 character is preceded by the byte 0xff (which is an invalid byte in UTF-8)
105 0x80000000 is added to the character's value. This is the only way of passing
106 such code points in a pattern string. For subject strings, using an escape
107 sequence is preferable.
110 .SH "COMMAND LINE OPTIONS"
114 If the 8-bit library has been built, this option causes it to be used (this is
115 the default). If the 8-bit library has not been built, this option causes an
119 If the 16-bit library has been built, this option causes it to be used. If only
120 the 16-bit library has been built, this is the default. If the 16-bit library
121 has not been built, this option causes an error.
124 If the 32-bit library has been built, this option causes it to be used. If only
125 the 32-bit library has been built, this is the default. If the 32-bit library
126 has not been built, this option causes an error.
129 Behave as if each pattern has the \fBauto_callout\fP modifier, that is, insert
130 automatic callouts into every pattern that is compiled.
133 As for \fB-ac\fP, but in addition behave as if each subject line has the
134 \fBcallout_extra\fP modifier, that is, show additional information from
138 Behave as if each pattern has the \fBfullbincode\fP modifier; the full
139 internal binary form of the pattern is output after compilation.
142 Output the version number of the PCRE2 library, and all available information
143 about the optional features that are included, and then exit with zero exit
144 code. All other options are ignored. If both -C and -LM are present, whichever
145 is first is recognized.
147 \fB-C\fP \fIoption\fP
148 Output information about a specific build-time option, then exit. This
149 functionality is intended for use in scripts such as \fBRunTest\fP. The
150 following options output the value and set the exit code as indicated:
152 ebcdic-nl the code for LF (= NL) in an EBCDIC environment:
154 0 if used in an ASCII environment
155 exit code is always 0
156 linksize the configured internal link size (2, 3, or 4)
157 exit code is set to the link size
158 newline the default newline setting:
159 CR, LF, CRLF, ANYCRLF, ANY, or NUL
160 exit code is always 0
161 bsr the default setting for what \eR matches:
163 exit code is always 0
165 The following options output 1 for true or 0 for false, and set the exit code
168 backslash-C \eC is supported (not locked out)
169 ebcdic compiled for an EBCDIC environment
170 jit just-in-time support is available
171 pcre2-16 the 16-bit library was built
172 pcre2-32 the 32-bit library was built
173 pcre2-8 the 8-bit library was built
174 unicode Unicode support is available
176 If an unknown option is given, an error message is output; the exit code is 0.
179 Behave as if each pattern has the \fBdebug\fP modifier; the internal
180 form and information about the compiled pattern is output after compilation;
181 \fB-d\fP is equivalent to \fB-b -i\fP.
184 Behave as if each subject line has the \fBdfa\fP modifier; matching is done
185 using the \fBpcre2_dfa_match()\fP function instead of the default
188 \fB-error\fP \fInumber[,number,...]\fP
189 Call \fBpcre2_get_error_message()\fP for each of the error numbers in the
190 comma-separated list, display the resulting messages on the standard output,
191 then exit with zero exit code. The numbers may be positive or negative. This is
192 a convenience facility for PCRE2 maintainers.
195 Output a brief summary these options and then exit.
198 Behave as if each pattern has the \fBinfo\fP modifier; information about the
199 compiled pattern is given after compilation.
202 Behave as if each pattern line has the \fBjit\fP modifier; after successful
203 compilation, each pattern is passed to the just-in-time compiler, if available.
206 Behave as if each pattern line has the \fBjitverify\fP modifier; after
207 successful compilation, each pattern is passed to the just-in-time compiler, if
208 available, and the use of JIT is verified.
211 List modifiers: write a list of available pattern and subject modifiers to the
212 standard output, then exit with zero exit code. All other options are ignored.
213 If both -C and -LM are present, whichever is first is recognized.
215 \fB-pattern\fB \fImodifier-list\fP
216 Behave as if each pattern line contains the given modifiers.
219 Do not output the version number of \fBpcre2test\fP at the start of execution.
222 On Unix-like systems, set the size of the run-time stack to \fIsize\fP
223 mebibytes (units of 1024*1024 bytes).
225 \fB-subject\fP \fImodifier-list\fP
226 Behave as if each subject line contains the given modifiers.
229 Run each compile and match many times with a timer, and output the resulting
230 times per compile or match. When JIT is used, separate times are given for the
231 initial compile and the JIT compile. You can control the number of iterations
232 that are used for timing by following \fB-t\fP with a number (as a separate
233 item on the command line). For example, "-t 1000" iterates 1000 times. The
234 default is to iterate 500,000 times.
237 This is like \fB-t\fP except that it times only the matching phase, not the
241 These behave like \fB-t\fP and \fB-tm\fP, but in addition, at the end of a run,
242 the total times for all compiles and matches are output.
245 Output the PCRE2 version number and then exit.
251 If \fBpcre2test\fP is given two filename arguments, it reads from the first and
252 writes to the second. If the first name is "-", input is taken from the
253 standard input. If \fBpcre2test\fP is given only one argument, it reads from
254 that file and writes to stdout. Otherwise, it reads from stdin and writes to
257 When \fBpcre2test\fP is built, a configuration option can specify that it
258 should be linked with the \fBlibreadline\fP or \fBlibedit\fP library. When this
259 is done, if the input is from a terminal, it is read using the \fBreadline()\fP
260 function. This provides line-editing and history facilities. The output from
261 the \fB-help\fP option states whether or not \fBreadline()\fP will be used.
263 The program handles any number of tests, each of which consists of a set of
264 input lines. Each set starts with a regular expression pattern, followed by any
265 number of subject lines to be matched against that pattern. In between sets of
266 test data, command lines that begin with # may appear. This file format, with
267 some restrictions, can also be processed by the \fBperltest.sh\fP script that
268 is distributed with PCRE2 as a means of checking that the behaviour of PCRE2
269 and Perl is the same. For a specification of \fBperltest.sh\fP, see the
270 comments near its beginning.
272 When the input is a terminal, \fBpcre2test\fP prompts for each line of input,
273 using "re>" to prompt for regular expression patterns, and "data>" to prompt
274 for subject lines. Command lines starting with # can be entered only in
275 response to the "re>" prompt.
277 Each subject line is matched separately and independently. If you want to do
278 multi-line matches, you have to use the \en escape sequence (or \er or \er\en,
279 etc., depending on the newline setting) in a single line of input to encode the
280 newline sequences. There is no limit on the length of subject lines; the input
281 buffer is automatically extended if it is too small. There are replication
282 features that makes it possible to generate long repetitive pattern or subject
283 lines without having to supply them explicitly.
285 An empty line or the end of the file signals the end of the subject lines for a
286 test, at which point a new pattern or command line is expected if there is
287 still input to be read.
293 In between sets of test data, a line that begins with # is interpreted as a
294 command line. If the first character is followed by white space or an
295 exclamation mark, the line is treated as a comment, and ignored. Otherwise, the
296 following commands are recognized:
300 Subsequent patterns automatically have the PCRE2_NEVER_UTF and PCRE2_NEVER_UCP
301 options set, which locks out the use of the PCRE2_UTF and PCRE2_UCP options and
302 the use of (*UTF) and (*UCP) at the start of patterns. This command also forces
303 an error if a subsequent pattern contains any occurrences of \eP, \ep, or \eX,
304 which are still supported when PCRE2_UTF is not set, but which require Unicode
305 property support to be included in the library.
307 This is a trigger guard that is used in test files to ensure that UTF or
308 Unicode property tests are not accidentally added to files that are used when
309 Unicode support is not included in the library. Setting PCRE2_NEVER_UTF and
310 PCRE2_NEVER_UCP as a default can also be obtained by the use of \fB#pattern\fP;
311 the difference is that \fB#forbid_utf\fP cannot be unset, and the automatic
312 options are not displayed in pattern information, to avoid cluttering up test
317 This command is used to load a set of precompiled patterns from a file, as
318 described in the section entitled "Saving and restoring compiled patterns"
319 .\" HTML <a href="#saverestore">
324 #newline_default [<newline-list>]
326 When PCRE2 is built, a default newline convention can be specified. This
327 determines which characters and/or character pairs are recognized as indicating
328 a newline in a pattern or subject string. The default can be overridden when a
329 pattern is compiled. The standard test files contain tests of various newline
330 conventions, but the majority of the tests expect a single linefeed to be
331 recognized as a newline by default. Without special action the tests would fail
332 when PCRE2 is compiled with either CR or CRLF as the default newline.
334 The #newline_default command specifies a list of newline types that are
335 acceptable as the default. The types must be one of CR, LF, CRLF, ANYCRLF,
336 ANY, or NUL (in upper or lower case), for example:
338 #newline_default LF Any anyCRLF
340 If the default newline is in the list, this command has no effect. Otherwise,
341 except when testing the POSIX API, a \fBnewline\fP modifier that specifies the
342 first newline convention in the list (LF in the above example) is added to any
343 pattern that does not already have a \fBnewline\fP modifier. If the newline
344 list is empty, the feature is turned off. This command is present in a number
345 of the standard test input files.
347 When the POSIX API is being tested there is no way to override the default
348 newline convention, though it is possible to set the newline convention from
349 within the pattern. A warning is given if the \fBposix\fP or \fBposix_nosub\fP
350 modifier is used when \fB#newline_default\fP would set a default for the
353 #pattern <modifier-list>
355 This command sets a default modifier list that applies to all subsequent
356 patterns. Modifiers on a pattern can change these settings.
360 The appearance of this line causes all subsequent modifier settings to be
361 checked for compatibility with the \fBperltest.sh\fP script, which is used to
362 confirm that Perl gives the same results as PCRE2. Also, apart from comment
363 lines, #pattern commands, and #subject commands that set or unset "mark", no
364 command lines are permitted, because they and many of the modifiers are
365 specific to \fBpcre2test\fP, and should not be used in test files that are also
366 processed by \fBperltest.sh\fP. The \fB#perltest\fP command helps detect tests
367 that are accidentally put in the wrong file.
370 #popcopy [<modifiers>]
372 These commands are used to manipulate the stack of compiled patterns, as
373 described in the section entitled "Saving and restoring compiled patterns"
374 .\" HTML <a href="#saverestore">
381 This command is used to save a set of compiled patterns to a file, as described
382 in the section entitled "Saving and restoring compiled patterns"
383 .\" HTML <a href="#saverestore">
388 #subject <modifier-list>
390 This command sets a default modifier list that applies to all subsequent
391 subject lines. Modifiers on a subject line can change these settings.
394 .SH "MODIFIER SYNTAX"
397 Modifier lists are used with both pattern and subject lines. Items in a list
398 are separated by commas followed by optional white space. Trailing whitespace
399 in a modifier list is ignored. Some modifiers may be given for both patterns
400 and subject lines, whereas others are valid only for one or the other. Each
401 modifier has a long name, for example "anchored", and some of them must be
402 followed by an equals sign and a value, for example, "offset=12". Values cannot
403 contain comma characters, but may contain spaces. Modifiers that do not take
404 values may be preceded by a minus sign to turn off a previous setting.
406 A few of the more common modifiers can also be specified as single letters, for
407 example "i" for "caseless". In documentation, following the Perl convention,
408 these are written with a slash ("the /i modifier") for clarity. Abbreviated
409 modifiers must all be concatenated in the first item of a modifier list. If the
410 first item is not recognized as a long modifier name, it is interpreted as a
411 sequence of these abbreviations. For example:
413 /abc/ig,newline=cr,jit=3
415 This is a pattern line whose modifier list starts with two one-letter modifiers
416 (/i and /g). The lower-case abbreviated modifiers are the same as used in Perl.
422 A pattern line must start with one of the following characters (common symbols,
423 excluding pattern meta-characters):
425 / ! " ' ` - = _ : ; , % & @ ~
427 This is interpreted as the pattern's delimiter. A regular expression may be
428 continued over several input lines, in which case the newline characters are
429 included within it. It is possible to include the delimiter within the pattern
430 by escaping it with a backslash, for example
434 If you do this, the escape and the delimiter form part of the pattern, but
435 since the delimiters are all non-alphanumeric, this does not affect its
436 interpretation. If the terminating delimiter is immediately followed by a
437 backslash, for example,
441 then a backslash is added to the end of the pattern. This is done to provide a
442 way of testing the error condition that arises if a pattern finishes with a
447 is interpreted as the first line of a pattern that starts with "abc/", causing
448 pcre2test to read the next line as a continuation of the regular expression.
450 A pattern can be followed by a modifier list (details below).
453 .SH "SUBJECT LINE SYNTAX"
456 Before each subject line is passed to \fBpcre2_match()\fP or
457 \fBpcre2_dfa_match()\fP, leading and trailing white space is removed, and the
458 line is scanned for backslash escapes, unless the \fBsubject_literal\fP
459 modifier was set for the pattern. The following provide a means of encoding
460 non-printing characters in a visible way:
462 \ea alarm (BEL, \ex07)
463 \eb backspace (\ex08)
465 \ef form feed (\ex0c)
467 \er carriage return (\ex0d)
469 \ev vertical tab (\ex0b)
470 \ennn octal character (up to 3 octal digits); always
471 a byte unless > 255 in UTF-8 or 16-bit or 32-bit mode
472 \eo{dd...} octal character (any number of octal digits}
473 \exhh hexadecimal byte (up to 2 hex digits)
474 \ex{hh...} hexadecimal character (any number of hex digits)
476 The use of \ex{hh...} is not dependent on the use of the \fButf\fP modifier on
477 the pattern. It is recognized always. There may be any number of hexadecimal
478 digits inside the braces; invalid values provoke error messages.
480 Note that \exhh specifies one byte rather than one character in UTF-8 mode;
481 this makes it possible to construct invalid UTF-8 sequences for testing
482 purposes. On the other hand, \ex{hh} is interpreted as a UTF-8 character in
483 UTF-8 mode, generating more than one byte if the value is greater than 127.
484 When testing the 8-bit library not in UTF-8 mode, \ex{hh} generates one byte
485 for values less than 256, and causes an error for greater values.
487 In UTF-16 mode, all 4-digit \ex{hhhh} values are accepted. This makes it
488 possible to construct invalid UTF-16 sequences for testing purposes.
490 In UTF-32 mode, all 4- to 8-digit \ex{...} values are accepted. This makes it
491 possible to construct invalid UTF-32 sequences for testing purposes.
493 There is a special backslash sequence that specifies replication of one or more
496 \e[<characters>]{<count>}
498 This makes it possible to test long strings without having to provide them as
499 part of the file. For example:
503 is converted to "abcabcabcabc". This feature does not support nesting. To
504 include a closing square bracket in the characters, code it as \ex5D.
506 A backslash followed by an equals sign marks the end of the subject string and
507 the start of a modifier list. For example:
509 abc\e=notbol,notempty
511 If the subject string is empty and \e= is followed by whitespace, the line is
512 treated as a comment line, and is not used for matching. For example:
514 \e= This is a comment.
515 abc\e= This is an invalid modifier list.
517 A backslash followed by any other non-alphanumeric character just escapes that
518 character. A backslash followed by anything else causes an error. However, if
519 the very last character in the line is a backslash (and there is no modifier
520 list), it is ignored. This gives a way of passing an empty line as data, since
521 a real empty line terminates the data input.
523 If the \fBsubject_literal\fP modifier is set for a pattern, all subject lines
524 that follow are treated as literals, with no special treatment of backslashes.
525 No replication is possible, and any subject modifiers must be set as defaults
526 by a \fB#subject\fP command.
529 .SH "PATTERN MODIFIERS"
532 There are several types of modifier that can appear in pattern lines. Except
533 where noted below, they may also be used in \fB#pattern\fP commands. A
534 pattern's modifier list can add to or override default modifiers that were set
535 by a previous \fB#pattern\fP command.
538 .\" HTML <a name="optionmodifiers"></a>
539 .SS "Setting compilation options"
542 The following modifiers set options for \fBpcre2_compile()\fP. Most of them set
543 bits in the options argument of that function, but those whose names start with
544 PCRE2_EXTRA are additional options that are set in the compile context. For the
545 main options, there are some single-letter abbreviations that are the same as
546 Perl options. There is special handling for /x: if a second x is present,
547 PCRE2_EXTENDED is converted into PCRE2_EXTENDED_MORE as in Perl. A third
548 appearance adds PCRE2_EXTENDED as well, though this makes no difference to the
549 way \fBpcre2_compile()\fP behaves. See
553 for a description of the effects of these options.
555 allow_empty_class set PCRE2_ALLOW_EMPTY_CLASS
556 allow_surrogate_escapes set PCRE2_EXTRA_ALLOW_SURROGATE_ESCAPES
557 alt_bsux set PCRE2_ALT_BSUX
558 alt_circumflex set PCRE2_ALT_CIRCUMFLEX
559 alt_verbnames set PCRE2_ALT_VERBNAMES
560 anchored set PCRE2_ANCHORED
561 auto_callout set PCRE2_AUTO_CALLOUT
562 bad_escape_is_literal set PCRE2_EXTRA_BAD_ESCAPE_IS_LITERAL
563 /i caseless set PCRE2_CASELESS
564 dollar_endonly set PCRE2_DOLLAR_ENDONLY
565 /s dotall set PCRE2_DOTALL
566 dupnames set PCRE2_DUPNAMES
567 endanchored set PCRE2_ENDANCHORED
568 /x extended set PCRE2_EXTENDED
569 /xx extended_more set PCRE2_EXTENDED_MORE
570 firstline set PCRE2_FIRSTLINE
571 literal set PCRE2_LITERAL
572 match_line set PCRE2_EXTRA_MATCH_LINE
573 match_unset_backref set PCRE2_MATCH_UNSET_BACKREF
574 match_word set PCRE2_EXTRA_MATCH_WORD
575 /m multiline set PCRE2_MULTILINE
576 never_backslash_c set PCRE2_NEVER_BACKSLASH_C
577 never_ucp set PCRE2_NEVER_UCP
578 never_utf set PCRE2_NEVER_UTF
579 /n no_auto_capture set PCRE2_NO_AUTO_CAPTURE
580 no_auto_possess set PCRE2_NO_AUTO_POSSESS
581 no_dotstar_anchor set PCRE2_NO_DOTSTAR_ANCHOR
582 no_start_optimize set PCRE2_NO_START_OPTIMIZE
583 no_utf_check set PCRE2_NO_UTF_CHECK
585 ungreedy set PCRE2_UNGREEDY
586 use_offset_limit set PCRE2_USE_OFFSET_LIMIT
589 As well as turning on the PCRE2_UTF option, the \fButf\fP modifier causes all
590 non-printing characters in output strings to be printed using the \ex{hh...}
591 notation. Otherwise, those less than 0x100 are output in hex without the curly
592 brackets. Setting \fButf\fP in 16-bit or 32-bit mode also causes pattern and
593 subject strings to be translated to UTF-16 or UTF-32, respectively, before
594 being passed to library functions.
597 .\" HTML <a name="controlmodifiers"></a>
598 .SS "Setting compilation controls"
601 The following modifiers affect the compilation process or request information
602 about the pattern. There are single-letter abbreviations for some that are
603 heavily used in the test files.
605 bsr=[anycrlf|unicode] specify \eR handling
606 /B bincode show binary code without lengths
607 callout_info show callout information
608 convert=<options> request foreign pattern conversion
609 convert_glob_escape=c set glob escape character
610 convert_glob_separator=c set glob separator character
611 convert_length set convert buffer length
612 debug same as info,fullbincode
613 framesize show matching frame size
614 fullbincode show binary code with lengths
615 /I info show info about compiled pattern
616 hex unquoted characters are hexadecimal
617 jit[=<number>] use JIT
618 jitfast use JIT fast path
619 jitverify verify JIT use
620 locale=<name> use this locale
621 max_pattern_length=<n> set the maximum pattern length
622 memory show memory used
623 newline=<type> set newline type
624 null_context compile with a NULL context
625 parens_nest_limit=<n> set maximum parentheses depth
626 posix use the POSIX API
627 posix_nosub use the POSIX API with REG_NOSUB
628 push push compiled pattern onto the stack
629 pushcopy push a copy onto the stack
630 stackguard=<number> test the stackguard feature
631 subject_literal treat all subject lines as literal
632 tables=[0|1|2] select internal tables
633 use_length do not zero-terminate the pattern
634 utf8_input treat input as UTF-8
636 The effects of these modifiers are described in the following sections.
639 .SS "Newline and \eR handling"
642 The \fBbsr\fP modifier specifies what \eR in a pattern should match. If it is
643 set to "anycrlf", \eR matches CR, LF, or CRLF only. If it is set to "unicode",
644 \eR matches any Unicode newline sequence. The default can be specified when
645 PCRE2 is built; if it is not, the default is set to Unicode.
647 The \fBnewline\fP modifier specifies which characters are to be interpreted as
648 newlines, both in the pattern and in subject lines. The type must be one of CR,
649 LF, CRLF, ANYCRLF, ANY, or NUL (in upper or lower case).
652 .SS "Information about a pattern"
655 The \fBdebug\fP modifier is a shorthand for \fBinfo,fullbincode\fP, requesting
656 all available information.
658 The \fBbincode\fP modifier causes a representation of the compiled code to be
659 output after compilation. This information does not contain length and offset
660 values, which ensures that the same output is generated for different internal
661 link sizes and different code unit widths. By using \fBbincode\fP, the same
662 regression tests can be used in different environments.
664 The \fBfullbincode\fP modifier, by contrast, \fIdoes\fP include length and
665 offset values. This is used in a few special tests that run only for specific
666 code unit widths and link sizes, and is also useful for one-off tests.
668 The \fBinfo\fP modifier requests information about the compiled pattern
669 (whether it is anchored, has a fixed first character, and so on). The
670 information is obtained from the \fBpcre2_pattern_info()\fP function. Here are
671 some typical examples:
673 re> /(?i)(^a|^b)/m,info
674 Capturing subpattern count = 1
675 Compile options: multiline
676 Overall options: caseless multiline
677 First code unit at start or follows newline
678 Subject length lower bound = 1
681 Capturing subpattern count = 0
682 Compile options: <none>
683 Overall options: caseless
684 First code unit = 'a' (caseless)
685 Last code unit = 'c' (caseless)
686 Subject length lower bound = 3
688 "Compile options" are those specified by modifiers; "overall options" have
689 added options that are taken or deduced from the pattern. If both sets of
690 options are the same, just a single "options" line is output; if there are no
691 options, the line is omitted. "First code unit" is where any match must start;
692 if there is more than one they are listed as "starting code units". "Last code
693 unit" is the last literal code unit that must be present in any match. This is
694 not necessarily the last character. These lines are omitted if no starting or
695 ending code units are recorded.
697 The \fBframesize\fP modifier shows the size, in bytes, of the storage frames
698 used by \fBpcre2_match()\fP for handling backtracking. The size depends on the
699 number of capturing parentheses in the pattern.
701 The \fBcallout_info\fP modifier requests information about all the callouts in
702 the pattern. A list of them is output at the end of any other information that
703 is requested. For each callout, either its number or string is given, followed
704 by the item that follows it in the pattern.
707 .SS "Passing a NULL context"
710 Normally, \fBpcre2test\fP passes a context block to \fBpcre2_compile()\fP. If
711 the \fBnull_context\fP modifier is set, however, NULL is passed. This is for
712 testing that \fBpcre2_compile()\fP behaves correctly in this case (it uses
716 .SS "Specifying pattern characters in hexadecimal"
719 The \fBhex\fP modifier specifies that the characters of the pattern, except for
720 substrings enclosed in single or double quotes, are to be interpreted as pairs
721 of hexadecimal digits. This feature is provided as a way of creating patterns
722 that contain binary zeros and other non-printing characters. White space is
723 permitted between pairs of digits. For example, this pattern contains three
728 Parts of such a pattern are taken literally if quoted. This pattern contains
729 nine characters, only two of which are specified in hexadecimal:
733 Either single or double quotes may be used. There is no way of including
734 the delimiter within a substring. The \fBhex\fP and \fBexpand\fP modifiers are
738 .SS "Specifying the pattern's length"
741 By default, patterns are passed to the compiling functions as zero-terminated
742 strings but can be passed by length instead of being zero-terminated. The
743 \fBuse_length\fP modifier causes this to happen. Using a length happens
744 automatically (whether or not \fBuse_length\fP is set) when \fBhex\fP is set,
745 because patterns specified in hexadecimal may contain binary zeros.
747 If \fBhex\fP or \fBuse_length\fP is used with the POSIX wrapper API (see
748 .\" HTML <a href="#posixwrapper">
750 "Using the POSIX wrapper API"
752 below), the REG_PEND extension is used to pass the pattern's length.
755 .SS "Specifying wide characters in 16-bit and 32-bit modes"
758 In 16-bit and 32-bit modes, all input is automatically treated as UTF-8 and
759 translated to UTF-16 or UTF-32 when the \fButf\fP modifier is set. For testing
760 the 16-bit and 32-bit libraries in non-UTF mode, the \fButf8_input\fP modifier
761 can be used. It is mutually exclusive with \fButf\fP. Input lines are
762 interpreted as UTF-8 as a means of specifying wide characters. More details are
764 .\" HTML <a href="#inputencoding">
771 .SS "Generating long repetitive patterns"
774 Some tests use long patterns that are very repetitive. Instead of creating a
775 very long input line for such a pattern, you can use a special repetition
776 feature, similar to the one described for subject lines above. If the
777 \fBexpand\fP modifier is present on a pattern, parts of the pattern that have
780 \e[<characters>]{<count>}
782 are expanded before the pattern is passed to \fBpcre2_compile()\fP. For
783 example, \e[AB]{6000} is expanded to "ABAB..." 6000 times. This construction
784 cannot be nested. An initial "\e[" sequence is recognized only if "]{" followed
785 by decimal digits and "}" is found later in the pattern. If not, the characters
786 remain in the pattern unaltered. The \fBexpand\fP and \fBhex\fP modifiers are
789 If part of an expanded pattern looks like an expansion, but is really part of
790 the actual pattern, unwanted expansion can be avoided by giving two values in
791 the quantifier. For example, \e[AB]{6000,6000} is not recognized as an
794 If the \fBinfo\fP modifier is set on an expanded pattern, the result of the
795 expansion is included in the information that is output.
798 .SS "JIT compilation"
801 Just-in-time (JIT) compiling is a heavyweight optimization that can greatly
802 speed up pattern matching. See the
806 documentation for details. JIT compiling happens, optionally, after a pattern
807 has been successfully compiled into an internal form. The JIT compiler converts
808 this to optimized machine code. It needs to know whether the match-time options
809 PCRE2_PARTIAL_HARD and PCRE2_PARTIAL_SOFT are going to be used, because
810 different code is generated for the different cases. See the \fBpartial\fP
811 modifier in "Subject Modifiers"
812 .\" HTML <a href="#subjectmodifiers">
816 for details of how these options are specified for each match attempt.
818 JIT compilation is requested by the \fBjit\fP pattern modifier, which may
819 optionally be followed by an equals sign and a number in the range 0 to 7.
820 The three bits that make up the number specify which of the three JIT operating
821 modes are to be compiled:
823 1 compile JIT code for non-partial matching
824 2 compile JIT code for soft partial matching
825 4 compile JIT code for hard partial matching
827 The possible values for the \fBjit\fP modifier are therefore:
830 1 normal matching only
831 2 soft partial matching only
832 3 normal and soft partial matching
833 4 hard partial matching only
834 6 soft and hard partial matching only
837 If no number is given, 7 is assumed. The phrase "partial matching" means a call
838 to \fBpcre2_match()\fP with either the PCRE2_PARTIAL_SOFT or the
839 PCRE2_PARTIAL_HARD option set. Note that such a call may return a complete
840 match; the options enable the possibility of a partial match, but do not
841 require it. Note also that if you request JIT compilation only for partial
842 matching (for example, jit=2) but do not set the \fBpartial\fP modifier on a
843 subject line, that match will not use JIT code because none was compiled for
844 non-partial matching.
846 If JIT compilation is successful, the compiled JIT code will automatically be
847 used when an appropriate type of match is run, except when incompatible
848 run-time options are specified. For more details, see the
852 documentation. See also the \fBjitstack\fP modifier below for a way of
853 setting the size of the JIT stack.
855 If the \fBjitfast\fP modifier is specified, matching is done using the JIT
856 "fast path" interface, \fBpcre2_jit_match()\fP, which skips some of the sanity
857 checks that are done by \fBpcre2_match()\fP, and of course does not work when
858 JIT is not supported. If \fBjitfast\fP is specified without \fBjit\fP, jit=7 is
861 If the \fBjitverify\fP modifier is specified, information about the compiled
862 pattern shows whether JIT compilation was or was not successful. If
863 \fBjitverify\fP is specified without \fBjit\fP, jit=7 is assumed. If JIT
864 compilation is successful when \fBjitverify\fP is set, the text "(JIT)" is
865 added to the first output line after a match or non match when JIT-compiled
866 code was actually used in the match.
869 .SS "Setting a locale"
872 The \fBlocale\fP modifier must specify the name of a locale, for example:
874 /pattern/locale=fr_FR
876 The given locale is set, \fBpcre2_maketables()\fP is called to build a set of
877 character tables for the locale, and this is then passed to
878 \fBpcre2_compile()\fP when compiling the regular expression. The same tables
879 are used when matching the following subject lines. The \fBlocale\fP modifier
880 applies only to the pattern on which it appears, but can be given in a
881 \fB#pattern\fP command if a default is needed. Setting a locale and alternate
882 character tables are mutually exclusive.
885 .SS "Showing pattern memory"
888 The \fBmemory\fP modifier causes the size in bytes of the memory used to hold
889 the compiled pattern to be output. This does not include the size of the
890 \fBpcre2_code\fP block; it is just the actual compiled data. If the pattern is
891 subsequently passed to the JIT compiler, the size of the JIT compiled code is
892 also output. Here is an example:
894 re> /a(b)c/jit,memory
895 Memory allocation (code space): 21
896 Memory allocation (JIT code): 1910
900 .SS "Limiting nested parentheses"
903 The \fBparens_nest_limit\fP modifier sets a limit on the depth of nested
904 parentheses in a pattern. Breaching the limit causes a compilation error.
905 The default for the library is set when PCRE2 is built, but \fBpcre2test\fP
906 sets its own default of 220, which is required for running the standard test
910 .SS "Limiting the pattern length"
913 The \fBmax_pattern_length\fP modifier sets a limit, in code units, to the
914 length of pattern that \fBpcre2_compile()\fP will accept. Breaching the limit
915 causes a compilation error. The default is the largest number a PCRE2_SIZE
916 variable can hold (essentially unlimited).
919 .\" HTML <a name="posixwrapper"></a>
920 .SS "Using the POSIX wrapper API"
923 The \fBposix\fP and \fBposix_nosub\fP modifiers cause \fBpcre2test\fP to call
924 PCRE2 via the POSIX wrapper API rather than its native API. When
925 \fBposix_nosub\fP is used, the POSIX option REG_NOSUB is passed to
926 \fBregcomp()\fP. The POSIX wrapper supports only the 8-bit library. Note that
927 it does not imply POSIX matching semantics; for more detail see the
931 documentation. The following pattern modifiers set options for the
932 \fBregcomp()\fP function:
935 multiline REG_NEWLINE
937 ungreedy REG_UNGREEDY ) These options are not part of
938 ucp REG_UCP ) the POSIX standard
941 The \fBregerror_buffsize\fP modifier specifies a size for the error buffer that
942 is passed to \fBregerror()\fP in the event of a compilation error. For example:
944 /abc/posix,regerror_buffsize=20
946 This provides a means of testing the behaviour of \fBregerror()\fP when the
947 buffer is too small for the error message. If this modifier has not been set, a
948 large buffer is used.
950 The \fBaftertext\fP and \fBallaftertext\fP subject modifiers work as described
951 below. All other modifiers are either ignored, with a warning message, or cause
954 The pattern is passed to \fBregcomp()\fP as a zero-terminated string by
955 default, but if the \fBuse_length\fP or \fBhex\fP modifiers are set, the
956 REG_PEND extension is used to pass it by length.
959 .SS "Testing the stack guard feature"
962 The \fBstackguard\fP modifier is used to test the use of
963 \fBpcre2_set_compile_recursion_guard()\fP, a function that is provided to
964 enable stack availability to be checked during compilation (see the
968 documentation for details). If the number specified by the modifier is greater
969 than zero, \fBpcre2_set_compile_recursion_guard()\fP is called to set up
970 callback from \fBpcre2_compile()\fP to a local function. The argument it
971 receives is the current nesting parenthesis depth; if this is greater than the
972 value given by the modifier, non-zero is returned, causing the compilation to
976 .SS "Using alternative character tables"
979 The value specified for the \fBtables\fP modifier must be one of the digits 0,
980 1, or 2. It causes a specific set of built-in character tables to be passed to
981 \fBpcre2_compile()\fP. This is used in the PCRE2 tests to check behaviour with
982 different character tables. The digit specifies the tables as follows:
984 0 do not pass any special character tables
985 1 the default ASCII tables, as distributed in
986 pcre2_chartables.c.dist
987 2 a set of tables defining ISO 8859 characters
989 In table 2, some characters whose codes are greater than 128 are identified as
990 letters, digits, spaces, etc. Setting alternate character tables and a locale
991 are mutually exclusive.
994 .SS "Setting certain match controls"
997 The following modifiers are really subject modifiers, and are described under
998 "Subject Modifiers" below. However, they may be included in a pattern's
999 modifier list, in which case they are applied to every subject line that is
1000 processed with that pattern. These modifiers do not affect the compilation
1003 aftertext show text after match
1004 allaftertext show text after captures
1005 allcaptures show all captures
1006 allusedtext show all consulted text
1007 altglobal alternative global matching
1008 /g global global matching
1009 jitstack=<n> set size of JIT stack
1010 mark show mark values
1011 replace=<string> specify a replacement string
1012 startchar show starting character when relevant
1013 substitute_extended use PCRE2_SUBSTITUTE_EXTENDED
1014 substitute_overflow_length use PCRE2_SUBSTITUTE_OVERFLOW_LENGTH
1015 substitute_unknown_unset use PCRE2_SUBSTITUTE_UNKNOWN_UNSET
1016 substitute_unset_empty use PCRE2_SUBSTITUTE_UNSET_EMPTY
1018 These modifiers may not appear in a \fB#pattern\fP command. If you want them as
1019 defaults, set them in a \fB#subject\fP command.
1022 .SS "Specifying literal subject lines"
1025 If the \fBsubject_literal\fP modifier is present on a pattern, all the subject
1026 lines that it matches are taken as literal strings, with no interpretation of
1027 backslashes. It is not possible to set subject modifiers on such lines, but any
1028 that are set as defaults by a \fB#subject\fP command are recognized.
1031 .SS "Saving a compiled pattern"
1034 When a pattern with the \fBpush\fP modifier is successfully compiled, it is
1035 pushed onto a stack of compiled patterns, and \fBpcre2test\fP expects the next
1036 line to contain a new pattern (or a command) instead of a subject line. This
1037 facility is used when saving compiled patterns to a file, as described in the
1038 section entitled "Saving and restoring compiled patterns"
1039 .\" HTML <a href="#saverestore">
1043 If \fBpushcopy\fP is used instead of \fBpush\fP, a copy of the compiled
1044 pattern is stacked, leaving the original as current, ready to match the
1045 following input lines. This provides a way of testing the
1046 \fBpcre2_code_copy()\fP function.
1048 The \fBpush\fP and \fBpushcopy \fP modifiers are incompatible with compilation
1049 modifiers such as \fBglobal\fP that act at match time. Any that are specified
1050 are ignored (for the stacked copy), with a warning message, except for
1051 \fBreplace\fP, which causes an error. Note that \fBjitverify\fP, which is
1052 allowed, does not carry through to any subsequent matching that uses a stacked
1056 .SS "Testing foreign pattern conversion"
1059 The experimental foreign pattern conversion functions in PCRE2 can be tested by
1060 setting the \fBconvert\fP modifier. Its argument is a colon-separated list of
1061 options, which set the equivalent option for the \fBpcre2_pattern_convert()\fP
1064 glob PCRE2_CONVERT_GLOB
1065 glob_no_starstar PCRE2_CONVERT_GLOB_NO_STARSTAR
1066 glob_no_wild_separator PCRE2_CONVERT_GLOB_NO_WILD_SEPARATOR
1067 posix_basic PCRE2_CONVERT_POSIX_BASIC
1068 posix_extended PCRE2_CONVERT_POSIX_EXTENDED
1069 unset Unset all options
1071 The "unset" value is useful for turning off a default that has been set by a
1072 \fB#pattern\fP command. When one of these options is set, the input pattern is
1073 passed to \fBpcre2_pattern_convert()\fP. If the conversion is successful, the
1074 result is reflected in the output and then passed to \fBpcre2_compile()\fP. The
1075 normal \fButf\fP and \fBno_utf_check\fP options, if set, cause the
1076 PCRE2_CONVERT_UTF and PCRE2_CONVERT_NO_UTF_CHECK options to be passed to
1077 \fBpcre2_pattern_convert()\fP.
1079 By default, the conversion function is allowed to allocate a buffer for its
1080 output. However, if the \fBconvert_length\fP modifier is set to a value greater
1081 than zero, \fBpcre2test\fP passes a buffer of the given length. This makes it
1082 possible to test the length check.
1084 The \fBconvert_glob_escape\fP and \fBconvert_glob_separator\fP modifiers can be
1085 used to specify the escape and separator characters for glob processing,
1086 overriding the defaults, which are operating-system dependent.
1089 .\" HTML <a name="subjectmodifiers"></a>
1090 .SH "SUBJECT MODIFIERS"
1093 The modifiers that can appear in subject lines and the \fB#subject\fP
1094 command are of two types.
1097 .SS "Setting match options"
1100 The following modifiers set options for \fBpcre2_match()\fP or
1101 \fBpcre2_dfa_match()\fP. See
1105 for a description of their effects.
1107 anchored set PCRE2_ANCHORED
1108 endanchored set PCRE2_ENDANCHORED
1109 dfa_restart set PCRE2_DFA_RESTART
1110 dfa_shortest set PCRE2_DFA_SHORTEST
1111 no_jit set PCRE2_NO_JIT
1112 no_utf_check set PCRE2_NO_UTF_CHECK
1113 notbol set PCRE2_NOTBOL
1114 notempty set PCRE2_NOTEMPTY
1115 notempty_atstart set PCRE2_NOTEMPTY_ATSTART
1116 noteol set PCRE2_NOTEOL
1117 partial_hard (or ph) set PCRE2_PARTIAL_HARD
1118 partial_soft (or ps) set PCRE2_PARTIAL_SOFT
1120 The partial matching modifiers are provided with abbreviations because they
1121 appear frequently in tests.
1123 If the \fBposix\fP or \fBposix_nosub\fP modifier was present on the pattern,
1124 causing the POSIX wrapper API to be used, the only option-setting modifiers
1125 that have any effect are \fBnotbol\fP, \fBnotempty\fP, and \fBnoteol\fP,
1126 causing REG_NOTBOL, REG_NOTEMPTY, and REG_NOTEOL, respectively, to be passed to
1127 \fBregexec()\fP. The other modifiers are ignored, with a warning message.
1129 There is one additional modifier that can be used with the POSIX wrapper. It is
1130 ignored (with a warning) if used for non-POSIX matching.
1132 posix_startend=<n>[:<m>]
1134 This causes the subject string to be passed to \fBregexec()\fP using the
1135 REG_STARTEND option, which uses offsets to specify which part of the string is
1136 searched. If only one number is given, the end offset is passed as the end of
1137 the subject string. For more detail of REG_STARTEND, see the
1141 documentation. If the subject string contains binary zeros (coded as escapes
1142 such as \ex{00} because \fBpcre2test\fP does not support actual binary zeros in
1143 its input), you must use \fBposix_startend\fP to specify its length.
1146 .SS "Setting match controls"
1149 The following modifiers affect the matching process or request additional
1150 information. Some of them may also be specified on a pattern line (see above),
1151 in which case they apply to every subject line that is matched against that
1154 aftertext show text after match
1155 allaftertext show text after captures
1156 allcaptures show all captures
1157 allusedtext show all consulted text (non-JIT only)
1158 altglobal alternative global matching
1159 callout_capture show captures at callout time
1160 callout_data=<n> set a value to pass via callouts
1161 callout_error=<n>[:<m>] control callout error
1162 callout_extra show extra callout information
1163 callout_fail=<n>[:<m>] control callout failure
1164 callout_no_where do not show position of a callout
1165 callout_none do not supply a callout function
1166 copy=<number or name> copy captured substring
1167 depth_limit=<n> set a depth limit
1168 dfa use \fBpcre2_dfa_match()\fP
1169 find_limits find match and depth limits
1170 get=<number or name> extract captured substring
1171 getall extract all captured substrings
1172 /g global global matching
1173 heap_limit=<n> set a limit on heap memory (Kbytes)
1174 jitstack=<n> set size of JIT stack
1175 mark show mark values
1176 match_limit=<n> set a match limit
1177 memory show heap memory usage
1178 null_context match with a NULL context
1179 offset=<n> set starting offset
1180 offset_limit=<n> set offset limit
1181 ovector=<n> set size of output vector
1182 recursion_limit=<n> obsolete synonym for depth_limit
1183 replace=<string> specify a replacement string
1184 startchar show startchar when relevant
1185 startoffset=<n> same as offset=<n>
1186 substitute_extedded use PCRE2_SUBSTITUTE_EXTENDED
1187 substitute_overflow_length use PCRE2_SUBSTITUTE_OVERFLOW_LENGTH
1188 substitute_unknown_unset use PCRE2_SUBSTITUTE_UNKNOWN_UNSET
1189 substitute_unset_empty use PCRE2_SUBSTITUTE_UNSET_EMPTY
1190 zero_terminate pass the subject as zero-terminated
1192 The effects of these modifiers are described in the following sections. When
1193 matching via the POSIX wrapper API, the \fBaftertext\fP, \fBallaftertext\fP,
1194 and \fBovector\fP subject modifiers work as described below. All other
1195 modifiers are either ignored, with a warning message, or cause an error.
1198 .SS "Showing more text"
1201 The \fBaftertext\fP modifier requests that as well as outputting the part of
1202 the subject string that matched the entire pattern, \fBpcre2test\fP should in
1203 addition output the remainder of the subject string. This is useful for tests
1204 where the subject contains multiple copies of the same substring. The
1205 \fBallaftertext\fP modifier requests the same action for captured substrings as
1206 well as the main matched substring. In each case the remainder is output on the
1207 following line with a plus character following the capture number.
1209 The \fBallusedtext\fP modifier requests that all the text that was consulted
1210 during a successful pattern match by the interpreter should be shown. This
1211 feature is not supported for JIT matching, and if requested with JIT it is
1212 ignored (with a warning message). Setting this modifier affects the output if
1213 there is a lookbehind at the start of a match, or a lookahead at the end, or if
1214 \eK is used in the pattern. Characters that precede or follow the start and end
1215 of the actual match are indicated in the output by '<' or '>' characters
1216 underneath them. Here is an example:
1218 re> /(?<=pqr)abc(?=xyz)/
1219 data> 123pqrabcxyz456\e=allusedtext
1223 This shows that the matched string is "abc", with the preceding and following
1224 strings "pqr" and "xyz" having been consulted during the match (when processing
1227 The \fBstartchar\fP modifier requests that the starting character for the match
1228 be indicated, if it is different to the start of the matched string. The only
1229 time when this occurs is when \eK has been processed as part of the match. In
1230 this situation, the output for the matched string is displayed from the
1231 starting character instead of from the match point, with circumflex characters
1232 under the earlier characters. For example:
1235 data> abcxyz\e=startchar
1239 Unlike \fBallusedtext\fP, the \fBstartchar\fP modifier can be used with JIT.
1240 However, these two modifiers are mutually exclusive.
1243 .SS "Showing the value of all capture groups"
1246 The \fBallcaptures\fP modifier requests that the values of all potential
1247 captured parentheses be output after a match. By default, only those up to the
1248 highest one actually used in the match are output (corresponding to the return
1249 code from \fBpcre2_match()\fP). Groups that did not take part in the match
1250 are output as "<unset>". This modifier is not relevant for DFA matching (which
1251 does no capturing); it is ignored, with a warning message, if present.
1254 .SS "Testing callouts"
1257 A callout function is supplied when \fBpcre2test\fP calls the library matching
1258 functions, unless \fBcallout_none\fP is specified. Its behaviour can be
1259 controlled by various modifiers listed above whose names begin with
1260 \fBcallout_\fP. Details are given in the section entitled "Callouts"
1261 .\" HTML <a href="#callouts">
1267 .SS "Finding all matches in a string"
1270 Searching for all possible matches within a subject can be requested by the
1271 \fBglobal\fP or \fBaltglobal\fP modifier. After finding a match, the matching
1272 function is called again to search the remainder of the subject. The difference
1273 between \fBglobal\fP and \fBaltglobal\fP is that the former uses the
1274 \fIstart_offset\fP argument to \fBpcre2_match()\fP or \fBpcre2_dfa_match()\fP
1275 to start searching at a new point within the entire string (which is what Perl
1276 does), whereas the latter passes over a shortened subject. This makes a
1277 difference to the matching process if the pattern begins with a lookbehind
1278 assertion (including \eb or \eB).
1280 If an empty string is matched, the next match is done with the
1281 PCRE2_NOTEMPTY_ATSTART and PCRE2_ANCHORED flags set, in order to search for
1282 another, non-empty, match at the same point in the subject. If this match
1283 fails, the start offset is advanced, and the normal match is retried. This
1284 imitates the way Perl handles such cases when using the \fB/g\fP modifier or
1285 the \fBsplit()\fP function. Normally, the start offset is advanced by one
1286 character, but if the newline convention recognizes CRLF as a newline, and the
1287 current character is CR followed by LF, an advance of two characters occurs.
1290 .SS "Testing substring extraction functions"
1293 The \fBcopy\fP and \fBget\fP modifiers can be used to test the
1294 \fBpcre2_substring_copy_xxx()\fP and \fBpcre2_substring_get_xxx()\fP functions.
1295 They can be given more than once, and each can specify a group name or number,
1298 abcd\e=copy=1,copy=3,get=G1
1300 If the \fB#subject\fP command is used to set default copy and/or get lists,
1301 these can be unset by specifying a negative number to cancel all numbered
1302 groups and an empty name to cancel all named groups.
1304 The \fBgetall\fP modifier tests \fBpcre2_substring_list_get()\fP, which
1305 extracts all captured substrings.
1307 If the subject line is successfully matched, the substrings extracted by the
1308 convenience functions are output with C, G, or L after the string number
1309 instead of a colon. This is in addition to the normal full list. The string
1310 length (that is, the return from the extraction function) is given in
1311 parentheses after each substring, followed by the name when the extraction was
1315 .SS "Testing the substitution function"
1318 If the \fBreplace\fP modifier is set, the \fBpcre2_substitute()\fP function is
1319 called instead of one of the matching functions. Note that replacement strings
1320 cannot contain commas, because a comma signifies the end of a modifier. This is
1321 not thought to be an issue in a test program.
1323 Unlike subject strings, \fBpcre2test\fP does not process replacement strings
1324 for escape sequences. In UTF mode, a replacement string is checked to see if it
1325 is a valid UTF-8 string. If so, it is correctly converted to a UTF string of
1326 the appropriate code unit width. If it is not a valid UTF-8 string, the
1327 individual code units are copied directly. This provides a means of passing an
1328 invalid UTF-8 string for testing purposes.
1330 The following modifiers set options (in additional to the normal match options)
1331 for \fBpcre2_substitute()\fP:
1333 global PCRE2_SUBSTITUTE_GLOBAL
1334 substitute_extended PCRE2_SUBSTITUTE_EXTENDED
1335 substitute_overflow_length PCRE2_SUBSTITUTE_OVERFLOW_LENGTH
1336 substitute_unknown_unset PCRE2_SUBSTITUTE_UNKNOWN_UNSET
1337 substitute_unset_empty PCRE2_SUBSTITUTE_UNSET_EMPTY
1340 After a successful substitution, the modified string is output, preceded by the
1341 number of replacements. This may be zero if there were no matches. Here is a
1342 simple example of a substitution test:
1350 Subject and replacement strings should be kept relatively short (fewer than 256
1351 characters) for substitution tests, as fixed-size buffers are used. To make it
1352 easy to test for buffer overflow, if the replacement string starts with a
1353 number in square brackets, that number is passed to \fBpcre2_substitute()\fP as
1354 the size of the output buffer, with the replacement string starting at the next
1355 character. Here is an example that tests the edge case:
1358 123abc123\e=replace=[10]XYZ
1360 123abc123\e=replace=[9]XYZ
1361 Failed: error -47: no more memory
1363 The default action of \fBpcre2_substitute()\fP is to return
1364 PCRE2_ERROR_NOMEMORY when the output buffer is too small. However, if the
1365 PCRE2_SUBSTITUTE_OVERFLOW_LENGTH option is set (by using the
1366 \fBsubstitute_overflow_length\fP modifier), \fBpcre2_substitute()\fP continues
1367 to go through the motions of matching and substituting, in order to compute the
1368 size of buffer that is required. When this happens, \fBpcre2test\fP shows the
1369 required buffer length (which includes space for the trailing zero) as part of
1370 the error message. For example:
1372 /abc/substitute_overflow_length
1373 123abc123\e=replace=[9]XYZ
1374 Failed: error -47: no more memory: 10 code units are needed
1376 A replacement string is ignored with POSIX and DFA matching. Specifying partial
1377 matching provokes an error return ("bad option value") from
1378 \fBpcre2_substitute()\fP.
1381 .SS "Setting the JIT stack size"
1384 The \fBjitstack\fP modifier provides a way of setting the maximum stack size
1385 that is used by the just-in-time optimization code. It is ignored if JIT
1386 optimization is not being used. The value is a number of kibibytes (units of
1387 1024 bytes). Setting zero reverts to the default of 32KiB. Providing a stack
1388 that is larger than the default is necessary only for very complicated
1389 patterns. If \fBjitstack\fP is set non-zero on a subject line it overrides any
1390 value that was set on the pattern.
1393 .SS "Setting heap, match, and depth limits"
1396 The \fBheap_limit\fP, \fBmatch_limit\fP, and \fBdepth_limit\fP modifiers set
1397 the appropriate limits in the match context. These values are ignored when the
1398 \fBfind_limits\fP modifier is specified.
1401 .SS "Finding minimum limits"
1404 If the \fBfind_limits\fP modifier is present on a subject line, \fBpcre2test\fP
1405 calls the relevant matching function several times, setting different values in
1406 the match context via \fBpcre2_set_heap_limit()\fP,
1407 \fBpcre2_set_match_limit()\fP, or \fBpcre2_set_depth_limit()\fP until it finds
1408 the minimum values for each parameter that allows the match to complete without
1409 error. If JIT is being used, only the match limit is relevant.
1411 When using this modifier, the pattern should not contain any limit settings
1412 such as (*LIMIT_MATCH=...) within it. If such a setting is present and is
1413 lower than the minimum matching value, the minimum value cannot be found
1414 because \fBpcre2_set_match_limit()\fP etc. are only able to reduce the value of
1415 an in-pattern limit; they cannot increase it.
1417 For non-DFA matching, the minimum \fIdepth_limit\fP number is a measure of how
1418 much nested backtracking happens (that is, how deeply the pattern's tree is
1419 searched). In the case of DFA matching, \fIdepth_limit\fP controls the depth of
1420 recursive calls of the internal function that is used for handling pattern
1421 recursion, lookaround assertions, and atomic groups.
1423 For non-DFA matching, the \fImatch_limit\fP number is a measure of the amount
1424 of backtracking that takes place, and learning the minimum value can be
1425 instructive. For most simple matches, the number is quite small, but for
1426 patterns with very large numbers of matching possibilities, it can become large
1427 very quickly with increasing length of subject string. In the case of DFA
1428 matching, \fImatch_limit\fP controls the total number of calls, both recursive
1429 and non-recursive, to the internal matching function, thus controlling the
1430 overall amount of computing resource that is used.
1432 For both kinds of matching, the \fIheap_limit\fP number, which is in kibibytes
1433 (units of 1024 bytes), limits the amount of heap memory used for matching. A
1434 value of zero disables the use of any heap memory; many simple pattern matches
1435 can be done without using the heap, so zero is not an unreasonable setting.
1438 .SS "Showing MARK names"
1442 The \fBmark\fP modifier causes the names from backtracking control verbs that
1443 are returned from calls to \fBpcre2_match()\fP to be displayed. If a mark is
1444 returned for a match, non-match, or partial match, \fBpcre2test\fP shows it.
1445 For a match, it is on a line by itself, tagged with "MK:". Otherwise, it
1446 is added to the non-match message.
1449 .SS "Showing memory usage"
1452 The \fBmemory\fP modifier causes \fBpcre2test\fP to log the sizes of all heap
1453 memory allocation and freeing calls that occur during a call to
1454 \fBpcre2_match()\fP or \fBpcre2_dfa_match()\fP. These occur only when a match
1455 requires a bigger vector than the default for remembering backtracking points
1456 (\fBpcre2_match()\fP) or for internal workspace (\fBpcre2_dfa_match()\fP). In
1457 many cases there will be no heap memory used and therefore no additional
1458 output. No heap memory is allocated during matching with JIT, so in that case
1459 the \fBmemory\fP modifier never has any effect. For this modifier to work, the
1460 \fBnull_context\fP modifier must not be set on both the pattern and the
1461 subject, though it can be set on one or the other.
1464 .SS "Setting a starting offset"
1467 The \fBoffset\fP modifier sets an offset in the subject string at which
1468 matching starts. Its value is a number of code units, not characters.
1471 .SS "Setting an offset limit"
1474 The \fBoffset_limit\fP modifier sets a limit for unanchored matches. If a match
1475 cannot be found starting at or before this offset in the subject, a "no match"
1476 return is given. The data value is a number of code units, not characters. When
1477 this modifier is used, the \fBuse_offset_limit\fP modifier must have been set
1478 for the pattern; if not, an error is generated.
1481 .SS "Setting the size of the output vector"
1484 The \fBovector\fP modifier applies only to the subject line in which it
1485 appears, though of course it can also be used to set a default in a
1486 \fB#subject\fP command. It specifies the number of pairs of offsets that are
1487 available for storing matching information. The default is 15.
1489 A value of zero is useful when testing the POSIX API because it causes
1490 \fBregexec()\fP to be called with a NULL capture vector. When not testing the
1491 POSIX API, a value of zero is used to cause
1492 \fBpcre2_match_data_create_from_pattern()\fP to be called, in order to create a
1493 match block of exactly the right size for the pattern. (It is not possible to
1494 create a match block with a zero-length ovector; there is always at least one
1498 .SS "Passing the subject as zero-terminated"
1501 By default, the subject string is passed to a native API matching function with
1502 its correct length. In order to test the facility for passing a zero-terminated
1503 string, the \fBzero_terminate\fP modifier is provided. It causes the length to
1504 be passed as PCRE2_ZERO_TERMINATED. When matching via the POSIX interface,
1505 this modifier is ignored, with a warning.
1507 When testing \fBpcre2_substitute()\fP, this modifier also has the effect of
1508 passing the replacement string as zero-terminated.
1511 .SS "Passing a NULL context"
1514 Normally, \fBpcre2test\fP passes a context block to \fBpcre2_match()\fP,
1515 \fBpcre2_dfa_match()\fP or \fBpcre2_jit_match()\fP. If the \fBnull_context\fP
1516 modifier is set, however, NULL is passed. This is for testing that the matching
1517 functions behave correctly in this case (they use default values). This
1518 modifier cannot be used with the \fBfind_limits\fP modifier or when testing the
1519 substitution function.
1522 .SH "THE ALTERNATIVE MATCHING FUNCTION"
1525 By default, \fBpcre2test\fP uses the standard PCRE2 matching function,
1526 \fBpcre2_match()\fP to match each subject line. PCRE2 also supports an
1527 alternative matching function, \fBpcre2_dfa_match()\fP, which operates in a
1528 different way, and has some restrictions. The differences between the two
1529 functions are described in the
1535 If the \fBdfa\fP modifier is set, the alternative matching function is used.
1536 This function finds all possible matches at a given point in the subject. If,
1537 however, the \fBdfa_shortest\fP modifier is set, processing stops after the
1538 first match is found. This is always the shortest possible match.
1541 .SH "DEFAULT OUTPUT FROM pcre2test"
1544 This section describes the output when the normal matching function,
1545 \fBpcre2_match()\fP, is being used.
1547 When a match succeeds, \fBpcre2test\fP outputs the list of captured substrings,
1548 starting with number 0 for the string that matched the whole pattern.
1549 Otherwise, it outputs "No match" when the return is PCRE2_ERROR_NOMATCH, or
1550 "Partial match:" followed by the partially matching substring when the
1551 return is PCRE2_ERROR_PARTIAL. (Note that this is the
1552 entire substring that was inspected during the partial match; it may include
1553 characters before the actual match start if a lookbehind assertion, \eK, \eb,
1554 or \eB was involved.)
1556 For any other return, \fBpcre2test\fP outputs the PCRE2 negative error number
1557 and a short descriptive phrase. If the error is a failed UTF string check, the
1558 code unit offset of the start of the failing character is also output. Here is
1559 an example of an interactive \fBpcre2test\fP run.
1562 PCRE2 version 10.22 2016-07-29
1571 Unset capturing substrings that are not followed by one that is set are not
1572 shown by \fBpcre2test\fP unless the \fBallcaptures\fP modifier is specified. In
1573 the following example, there are two capturing substrings, but when the first
1574 data line is matched, the second, unset substring is not shown. An "internal"
1575 unset substring is shown as "<unset>", as for the second data line.
1586 If the strings contain any non-printing characters, they are output as \exhh
1587 escapes if the value is less than 256 and UTF mode is not set. Otherwise they
1588 are output as \ex{hh...} escapes. See below for the definition of non-printing
1589 characters. If the \fBaftertext\fP modifier is set, the output for substring
1590 0 is followed by the the rest of the subject string, identified by "0+" like
1598 If global matching is requested, the results of successive matching attempts
1599 are output in sequence, like this:
1610 "No match" is output only if the first match attempt fails. Here is an example
1611 of a failure message (the offset 4 that is specified by the \fBoffset\fP
1612 modifier is past the end of the subject string):
1615 data> xyz\e=offset=4
1616 Error -24 (bad offset value)
1618 Note that whereas patterns can be continued over several lines (a plain ">"
1619 prompt is used for continuations), subject lines may not. However newlines can
1620 be included in a subject by means of the \en escape (or \er, \er\en, etc.,
1621 depending on the newline sequence setting).
1625 .SH "OUTPUT FROM THE ALTERNATIVE MATCHING FUNCTION"
1628 When the alternative matching function, \fBpcre2_dfa_match()\fP, is used, the
1629 output consists of a list of all the matches that start at the first point in
1630 the subject where there is at least one match. For example:
1632 re> /(tang|tangerine|tan)/
1633 data> yellow tangerine\e=dfa
1638 Using the normal matching function on this data finds only "tang". The
1639 longest matching string is always given first (and numbered zero). After a
1640 PCRE2_ERROR_PARTIAL return, the output is "Partial match:", followed by the
1641 partially matching substring. Note that this is the entire substring that was
1642 inspected during the partial match; it may include characters before the actual
1643 match start if a lookbehind assertion, \eb, or \eB was involved. (\eK is not
1644 supported for DFA matching.)
1646 If global matching is requested, the search for further matches resumes
1647 at the end of the longest match. For example:
1649 re> /(tang|tangerine|tan)/g
1650 data> yellow tangerine and tangy sultana\e=dfa
1658 The alternative matching function does not support substring capture, so the
1659 modifiers that are concerned with captured substrings are not relevant.
1662 .SH "RESTARTING AFTER A PARTIAL MATCH"
1665 When the alternative matching function has given the PCRE2_ERROR_PARTIAL
1666 return, indicating that the subject partially matched the pattern, you can
1667 restart the match with additional subject data by means of the
1668 \fBdfa_restart\fP modifier. For example:
1670 re> /^\ed?\ed(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\ed\ed$/
1673 data> n05\e=dfa,dfa_restart
1676 For further information about partial matching, see the
1683 .\" HTML <a name="callouts"></a>
1687 If the pattern contains any callout requests, \fBpcre2test\fP's callout
1688 function is called during matching unless \fBcallout_none\fP is specified. This
1689 works with both matching functions, and with JIT, though there are some
1690 differences in behaviour. The output for callouts with numerical arguments and
1691 those with string arguments is slightly different.
1694 .SS "Callouts with numerical arguments"
1697 By default, the callout function displays the callout number, the start and
1698 current positions in the subject text at the callout time, and the next pattern
1699 item to be tested. For example:
1704 This output indicates that callout number 0 occurred for a match attempt
1705 starting at the fourth character of the subject string, when the pointer was at
1706 the seventh character, and when the next pattern item was \ed. Just
1707 one circumflex is output if the start and current positions are the same, or if
1708 the current position precedes the start position, which can happen if the
1709 callout is in a lookbehind assertion.
1711 Callouts numbered 255 are assumed to be automatic callouts, inserted as a
1712 result of the \fBauto_callout\fP pattern modifier. In this case, instead of
1713 showing the callout number, the offset in the pattern, preceded by a plus, is
1714 output. For example:
1716 re> /\ed?[A-E]\e*/auto_callout
1725 If a pattern contains (*MARK) items, an additional line is output whenever
1726 a change of latest mark is passed to the callout function. For example:
1728 re> /a(*MARK:X)bc/auto_callout
1739 The mark changes between matching "a" and "b", but stays the same for the rest
1740 of the match, so nothing more is output. If, as a result of backtracking, the
1741 mark reverts to being unset, the text "<unset>" is output.
1744 .SS "Callouts with string arguments"
1747 The output for a callout with a string argument is similar, except that instead
1748 of outputting a callout number before the position indicators, the callout
1749 string and its offset in the pattern string are output before the reflection of
1750 the subject string, and the subject string is reflected for each callout. For
1753 re> /^ab(?C'first')cd(?C"second")ef/
1755 Callout (7): 'first'
1758 Callout (20): "second"
1765 .SS "Callout modifiers"
1768 The callout function in \fBpcre2test\fP returns zero (carry on matching) by
1769 default, but you can use a \fBcallout_fail\fP modifier in a subject line to
1770 change this and other parameters of the callout (see below).
1772 If the \fBcallout_capture\fP modifier is set, the current captured groups are
1773 output when a callout occurs. This is useful only for non-DFA matching, as
1774 \fBpcre2_dfa_match()\fP does not support capturing, so no captures are ever
1777 The normal callout output, showing the callout number or pattern offset (as
1778 described above) is suppressed if the \fBcallout_no_where\fP modifier is set.
1780 When using the interpretive matching function \fBpcre2_match()\fP without JIT,
1781 setting the \fBcallout_extra\fP modifier causes additional output from
1782 \fBpcre2test\fP's callout function to be generated. For the first callout in a
1783 match attempt at a new starting position in the subject, "New match attempt" is
1784 output. If there has been a backtrack since the last callout (or start of
1785 matching if this is the first callout), "Backtrack" is output, followed by "No
1786 other matching paths" if the backtrack ended the previous match attempt. For
1789 re> /(a+)b/auto_callout,no_start_optimize,no_auto_possess
1790 data> aac\e=callout_extra
1802 No other matching paths
1810 No other matching paths
1816 No other matching paths
1823 Notice that various optimizations must be turned off if you want all possible
1824 matching paths to be scanned. If \fBno_start_optimize\fP is not used, there is
1825 an immediate "no match", without any callouts, because the starting
1826 optimization fails to find "b" in the subject, which it knows must be present
1827 for any match. If \fBno_auto_possess\fP is not used, the "a+" item is turned
1828 into "a++", which reduces the number of backtracks.
1830 The \fBcallout_extra\fP modifier has no effect if used with the DFA matching
1831 function, or with JIT.
1834 .SS "Return values from callouts"
1837 The default return from the callout function is zero, which allows matching to
1838 continue. The \fBcallout_fail\fP modifier can be given one or two numbers. If
1839 there is only one number, 1 is returned instead of 0 (causing matching to
1840 backtrack) when a callout of that number is reached. If two numbers (<n>:<m>)
1841 are given, 1 is returned when callout <n> is reached and there have been at
1842 least <m> callouts. The \fBcallout_error\fP modifier is similar, except that
1843 PCRE2_ERROR_CALLOUT is returned, causing the entire matching process to be
1844 aborted. If both these modifiers are set for the same callout number,
1845 \fBcallout_error\fP takes precedence. Note that callouts with string arguments
1846 are always given the number zero.
1848 The \fBcallout_data\fP modifier can be given an unsigned or a negative number.
1849 This is set as the "user data" that is passed to the matching function, and
1850 passed back when the callout function is invoked. Any value other than zero is
1851 used as a return from \fBpcre2test\fP's callout function.
1853 Inserting callouts can be helpful when using \fBpcre2test\fP to check
1854 complicated regular expressions. For further information about callouts, see
1863 .SH "NON-PRINTING CHARACTERS"
1866 When \fBpcre2test\fP is outputting text in the compiled version of a pattern,
1867 bytes other than 32-126 are always treated as non-printing characters and are
1868 therefore shown as hex escapes.
1870 When \fBpcre2test\fP is outputting text that is a matched part of a subject
1871 string, it behaves in the same way, unless a different locale has been set for
1872 the pattern (using the \fBlocale\fP modifier). In this case, the
1873 \fBisprint()\fP function is used to distinguish printing and non-printing
1878 .\" HTML <a name="saverestore"></a>
1879 .SH "SAVING AND RESTORING COMPILED PATTERNS"
1882 It is possible to save compiled patterns on disc or elsewhere, and reload them
1883 later, subject to a number of restrictions. JIT data cannot be saved. The host
1884 on which the patterns are reloaded must be running the same version of PCRE2,
1885 with the same code unit width, and must also have the same endianness, pointer
1886 width and PCRE2_SIZE type. Before compiled patterns can be saved they must be
1887 serialized, that is, converted to a stream of bytes. A single byte stream may
1888 contain any number of compiled patterns, but they must all use the same
1889 character tables. A single copy of the tables is included in the byte stream
1890 (its size is 1088 bytes).
1892 The functions whose names begin with \fBpcre2_serialize_\fP are used
1893 for serializing and de-serializing. They are described in the
1895 \fBpcre2serialize\fP
1897 documentation. In this section we describe the features of \fBpcre2test\fP that
1898 can be used to test these functions.
1900 Note that "serialization" in PCRE2 does not convert compiled patterns to an
1901 abstract format like Java or .NET. It just makes a reloadable byte code stream.
1902 Hence the restrictions on reloading mentioned above.
1904 In \fBpcre2test\fP, when a pattern with \fBpush\fP modifier is successfully
1905 compiled, it is pushed onto a stack of compiled patterns, and \fBpcre2test\fP
1906 expects the next line to contain a new pattern (or command) instead of a
1907 subject line. By contrast, the \fBpushcopy\fP modifier causes a copy of the
1908 compiled pattern to be stacked, leaving the original available for immediate
1909 matching. By using \fBpush\fP and/or \fBpushcopy\fP, a number of patterns can
1910 be compiled and retained. These modifiers are incompatible with \fBposix\fP,
1911 and control modifiers that act at match time are ignored (with a message) for
1912 the stacked patterns. The \fBjitverify\fP modifier applies only at compile
1919 causes all the stacked patterns to be serialized and the result written to the
1920 named file. Afterwards, all the stacked patterns are freed. The command
1924 reads the data in the file, and then arranges for it to be de-serialized, with
1925 the resulting compiled patterns added to the pattern stack. The pattern on the
1926 top of the stack can be retrieved by the #pop command, which must be followed
1927 by lines of subjects that are to be matched with the pattern, terminated as
1928 usual by an empty line or end of file. This command may be followed by a
1929 modifier list containing only
1930 .\" HTML <a href="#controlmodifiers">
1934 that act after a pattern has been compiled. In particular, \fBhex\fP,
1935 \fBposix\fP, \fBposix_nosub\fP, \fBpush\fP, and \fBpushcopy\fP are not allowed,
1937 .\" HTML <a href="#optionmodifiers">
1939 option-setting modifiers.
1941 The JIT modifiers are, however permitted. Here is an example that saves and
1942 reloads two patterns.
1954 If \fBjitverify\fP is used with #pop, it does not automatically imply
1955 \fBjit\fP, which is different behaviour from when it is used on a pattern.
1957 The #popcopy command is analagous to the \fBpushcopy\fP modifier in that it
1958 makes current a copy of the topmost stack pattern, leaving the original still
1966 \fBpcre2\fP(3), \fBpcre2api\fP(3), \fBpcre2callout\fP(3),
1967 \fBpcre2jit\fP, \fBpcre2matching\fP(3), \fBpcre2partial\fP(d),
1968 \fBpcre2pattern\fP(3), \fBpcre2serialize\fP(3).
1976 University Computing Service
1985 Last updated: 21 July 2018
1986 Copyright (c) 1997-2018 University of Cambridge.
projects / ossec-hids.git / blob