5 Version 10.32-RC1 10-September-2018
6 -----------------------------------
8 1. When matching using the the REG_STARTEND feature of the POSIX API with a
9 non-zero starting offset, unset capturing groups with lower numbers than a
10 group that did capture something were not being correctly returned as "unset"
11 (that is, with offset values of -1).
13 2. When matching using the POSIX API, pcre2test used to omit listing unset
14 groups altogether. Now it shows those that come before any actual captures as
15 "<unset>", as happens for non-POSIX matching.
17 3. Running "pcre2test -C" always stated "\R matches CR, LF, or CRLF only",
18 whatever the build configuration was. It now correctly says "\R matches all
19 Unicode newlines" in the default case when --enable-bsr-anycrlf has not been
20 specified. Similarly, running "pcre2test -C bsr" never produced the result
23 4. Matching the pattern /(*UTF)\C[^\v]+\x80/ against an 8-bit string containing
24 multi-code-unit characters caused bad behaviour and possibly a crash. This
25 issue was fixed for other kinds of repeat in release 10.20 by change 19, but
26 repeating character classes were overlooked.
28 5. pcre2grep now supports the inclusion of binary zeros in patterns that are
29 read from files via the -f option.
31 6. A small fix to pcre2grep to avoid compiler warnings for -Wformat-overflow=2.
33 7. Added --enable-jit=auto support to configure.ac.
35 8. Added some dummy variables to the heapframe structure in 16-bit and 32-bit
36 modes for the benefit of m68k, where pointers can be 16-bit aligned. The
37 dummies force 32-bit alignment and this ensures that the structure is a
38 multiple of PCRE2_SIZE, a requirement that is tested at compile time. In other
39 architectures, alignment requirements take care of this automatically.
41 9. When returning an error from pcre2_pattern_convert(), ensure the error
42 offset is set zero for early errors.
44 10. A number of patches for Windows support from Daniel Richard G:
46 (a) List of error numbers in Runtest.bat corrected (it was not the same as in
49 (b) pcre2grep snprintf() workaround as used elsewhere in the tree.
51 (c) Support for non-C99 snprintf() that returns -1 in the overflow case.
53 11. Minor tidy of pcre2_dfa_match() code.
55 12. Refactored pcre2_dfa_match() so that the internal recursive calls no longer
56 use the stack for local workspace and local ovectors. Instead, an initial block
57 of stack is reserved, but if this is insufficient, heap memory is used. The
58 heap limit parameter now applies to pcre2_dfa_match().
60 13. If a "find limits" test of DFA matching in pcre2test resulted in too many
61 matches for the ovector, no matches were displayed.
63 14. Removed an occurrence of ctrl/Z from test 6 because Windows treats it as
64 EOF. The test looks to have come from a fuzzer.
66 15. If PCRE2 was built with a default match limit a lot greater than the
67 default default of 10 000 000, some JIT tests of the match limit no longer
68 failed. All such tests now set 10 000 000 as the upper limit.
70 16. Another Windows related patch for pcregrep to ensure that WIN32 is
71 undefined under Cygwin.
73 17. Test for the presence of stdint.h and inttypes.h in configure and CMake and
74 include whichever exists (stdint preferred) instead of unconditionally
75 including stdint. This makes life easier for old and non-standard systems.
77 18. Further changes to improve portability, especially to old and or non-
80 (a) Put all printf arguments in RunGrepTest into single, not double, quotes,
81 and use \0 not \x00 for binary zero.
83 (b) Avoid the use of C++ (i.e. BCPL) // comments.
85 (c) Parameterize the use of %zu in pcre2test to make it like %td. For both of
86 these now, if using MSVC or a standard C before C99, %lu is used with a
89 19. Applied a contributed patch to CMakeLists.txt to increase the stack size
90 when linking pcre2test with MSVC. This gets rid of a stack overflow error in
91 the standard set of tests.
93 20. Output a warning in pcre2test when ignoring the "altglobal" modifier when
94 it is given with the "replace" modifier.
96 21. In both pcre2test and pcre2_substitute(), with global matching, a pattern
97 that matched an empty string, but never at the starting match offset, was not
98 handled in a Perl-compatible way. The pattern /(<?=\G.)/ is an example of such
99 a pattern. Because \G is in a lookbehind assertion, there has to be a
100 "bumpalong" before there can be a match. The automatic "advance by one
101 character after an empty string match" rule is therefore inappropriate. A more
102 complicated algorithm has now been implemented.
104 22. When checking to see if a lookbehind is of fixed length, lookaheads were
105 correctly ignored, but qualifiers on lookaheads were not being ignored, leading
106 to an incorrect "lookbehind assertion is not fixed length" error.
108 23. The VERSION condition test was reading fractional PCRE2 version numbers
109 such as the 04 in 10.04 incorrectly and hence giving wrong results.
111 24. Updated to Unicode version 11.0.0. As well as the usual addition of new
112 scripts and characters, this involved re-jigging the grapheme break property
113 algorithm because Unicode has changed the way emojis are handled.
115 25. Fixed an obscure bug that struck when there were two atomic groups not
116 separated by something with a backtracking point. There could be an incorrect
117 backtrack into the first of the atomic groups. A complicated example is
118 /(?>a(*:1))(?>b)(*SKIP:1)x|.*/ matched against "abc", where the *SKIP
119 shouldn't find a MARK (because is in an atomic group), but it did.
121 26. Upgraded the perltest.sh script: (1) #pattern lines can now be used to set
122 a list of modifiers for all subsequent patterns - only those that the script
123 recognizes are meaningful; (2) #subject lines can be used to set or unset a
124 default "mark" modifier; (3) Unsupported #command lines give a warning when
125 they are ignored; (4) Mark data is output only if the "mark" modifier is
128 27. (*ACCEPT:ARG), (*FAIL:ARG), and (*COMMIT:ARG) are now supported.
130 28. A (*MARK) name was not being passed back for positive assertions that were
131 terminated by (*ACCEPT).
133 29. Add support for \N{U+dddd}, but only in Unicode mode.
135 30. Add support for (?^) for unsetting all imnsx options.
137 31. The PCRE2_EXTENDED (/x) option only ever discarded space characters whose
138 code point was less than 256 and that were recognized by the lookup table
139 generated by pcre2_maketables(), which uses isspace() to identify white space.
140 Now, when Unicode support is compiled, PCRE2_EXTENDED also discards U+0085,
141 U+200E, U+200F, U+2028, and U+2029, which are additional characters defined by
142 Unicode as "Pattern White Space". This makes PCRE2 compatible with Perl.
144 32. In certain circumstances, option settings within patterns were not being
145 correctly processed. For example, the pattern /((?i)A)(?m)B/ incorrectly
146 matched "ab". (The (?m) setting lost the fact that (?i) should be reset at the
147 end of its group during the parse process, but without another setting such as
148 (?m) the compile phase got it right.) This bug was introduced by the
149 refactoring in release 10.23.
151 33. PCRE2 uses bcopy() if available when memmove() is not, and it used just to
152 define memmove() as function call to bcopy(). This hasn't been tested for a
153 long time because in pcre2test the result of memmove() was being used, whereas
154 bcopy() doesn't return a result. This feature is now refactored always to call
155 an emulation function when there is no memmove(). The emulation makes use of
156 bcopy() when available.
158 34. When serializing a pattern, set the memctl, executable_jit, and tables
159 fields (that is, all the fields that contain pointers) to zeros so that the
160 result of serializing is always the same. These fields are re-set when the
161 pattern is deserialized.
163 35. In a pattern such as /[^\x{100}-\x{ffff}]*[\x80-\xff]/ which has a repeated
164 negative class with no characters less than 0x100 followed by a positive class
165 with only characters less than 0x100, the first class was incorrectly being
166 auto-possessified, causing incorrect match failures.
168 36. Removed the character type bit ctype_meta, which dates from PCRE1 and is
171 37. Tidied up unnecessarily complicated macros used in the escapes table.
173 38. Since 10.21, the new testoutput8-16-4 file has accidentally been omitted
174 from distribution tarballs, owing to a typo in Makefile.am which had
175 testoutput8-16-3 twice. Now fixed.
177 39. If the only branch in a conditional subpattern was anchored, the whole
178 subpattern was treated as anchored, when it should not have been, since the
179 assumed empty second branch cannot be anchored. Demonstrated by test patterns
180 such as /(?(1)^())b/ or /(?(?=^))b/.
182 40. A repeated conditional subpattern that could match an empty string was
183 always assumed to be unanchored. Now it it checked just like any other
184 repeated conditional subpattern, and can be found to be anchored if the minimum
185 quantifier is one or more. I can't see much use for a repeated anchored
186 pattern, but the behaviour is now consistent.
188 41. Minor addition to pcre2_jit_compile.c to avoid static analyzer complaint
189 (for an event that could never occur but you had to have external information
192 42. If before the first match in a file that was being searched by pcre2grep
193 there was a line that was sufficiently long to cause the input buffer to be
194 expanded, the variable holding the location of the end of the previous match
195 was being adjusted incorrectly, and could cause an overflow warning from a code
196 sanitizer. However, as the value is used only to print pending "after" lines
197 when the next match is reached (and there are no such lines in this case) this
198 bug could do no damage.
201 Version 10.31 12-February-2018
202 ------------------------------
204 1. Fix typo (missing ]) in VMS code in pcre2test.c.
206 2. Replace the replicated code for matching extended Unicode grapheme sequences
207 (which got a lot more complicated by change 10.30/49) by a single subroutine
208 that is called by both pcre2_match() and pcre2_dfa_match().
210 3. Add idempotent guard to pcre2_internal.h.
212 4. Add new pcre2_config() options: PCRE2_CONFIG_NEVER_BACKSLASH_C and
213 PCRE2_CONFIG_COMPILED_WIDTHS.
215 5. Cut out \C tests in the JIT regression tests when NEVER_BACKSLASH_C is
216 defined (e.g. by --enable-never-backslash-C).
218 6. Defined public names for all the pcre2_compile() error numbers, and used
219 the public names in pcre2_convert.c.
221 7. Fixed a small memory leak in pcre2test (convert contexts).
223 8. Added two casts to compile.c and one to match.c to avoid compiler warnings.
225 9. Added code to pcre2grep when compiled under VMS to set the symbol
226 PCRE2GREP_RC to the exit status, because VMS does not distinguish between
229 10. Added the -LM (list modifiers) option to pcre2test. Also made -C complain
230 about a bad option only if the following argument item does not start with a
233 11. pcre2grep was truncating components of file names to 128 characters when
234 processing files with the -r option, and also (some very odd code) truncating
235 path names to 512 characters. There is now a check on the absolute length of
236 full path file names, which may be up to 2047 characters long.
238 12. When an assertion contained (*ACCEPT) it caused all open capturing groups
239 to be closed (as for a non-assertion ACCEPT), which was wrong and could lead to
240 misbehaviour for subsequent references to groups that started outside the
241 assertion. ACCEPT in an assertion now closes only those groups that were
242 started within that assertion. Fixes oss-fuzz issues 3852 and 3891.
244 13. Multiline matching in pcre2grep was misbehaving if the pattern matched
245 within a line, and then matched again at the end of the line and over into
246 subsequent lines. Behaviour was different with and without colouring, and
247 sometimes context lines were incorrectly printed and/or line endings were lost.
248 All these issues should now be fixed.
250 14. If --line-buffered was specified for pcre2grep when input was from a
251 compressed file (.gz or .bz2) a segfault occurred. (Line buffering should be
252 ignored for compressed files.)
254 15. Although pcre2_jit_match checks whether the pattern is compiled
255 in a given mode, it was also expected that at least one mode is available.
256 This is fixed and pcre2_jit_match returns with PCRE2_ERROR_JIT_BADOPTION
257 when the pattern is not optimized by JIT at all.
259 16. The line number and related variables such as match counts in pcre2grep
260 were all int variables, causing overflow when files with more than 2147483647
261 lines were processed (assuming 32-bit ints). They have all been changed to
264 17. If a backreference with a minimum repeat count of zero was first in a
265 pattern, apart from assertions, an incorrect first matching character could be
266 recorded. For example, for the pattern /(?=(a))\1?b/, "b" was incorrectly set
267 as the first character of a match.
269 18. Characters in a leading positive assertion are considered for recording a
270 first character of a match when the rest of the pattern does not provide one.
271 However, a character in a non-assertive group within a leading assertion such
272 as in the pattern /(?=(a))\1?b/ caused this process to fail. This was an
273 infelicity rather than an outright bug, because it did not affect the result of
274 a match, just its speed. (In fact, in this case, the starting 'a' was
275 subsequently picked up in the study.)
277 19. A minor tidy in pcre2_match(): making all PCRE2_ERROR_ returns use "return"
278 instead of "RRETURN" saves unwinding the backtracks in these cases (only one
281 20. Allocate a single callout block on the stack at the start of pcre2_match()
282 and set its never-changing fields once only. Do the same for pcre2_dfa_match().
284 21. Save the extra compile options (set in the compile context) with the
285 compiled pattern (they were not previously saved), add PCRE2_INFO_EXTRAOPTIONS
286 to retrieve them, and update pcre2test to show them.
288 22. Added PCRE2_CALLOUT_STARTMATCH and PCRE2_CALLOUT_BACKTRACK bits to a new
289 field callout_flags in callout blocks. The bits are set by pcre2_match(), but
290 not by JIT or pcre2_dfa_match(). Their settings are shown in pcre2test callouts
291 if the callout_extra subject modifier is set. These bits are provided to help
292 with tracking how a backtracking match is proceeding.
294 23. Updated the pcre2demo.c demonstration program, which was missing the extra
295 code for -g that handles the case when \K in an assertion causes the match to
296 end at the original start point. Also arranged for it to detect when \K causes
297 the end of a match to be before its start.
299 24. Similar to 23 above, strange things (including loops) could happen in
300 pcre2grep when \K was used in an assertion when --colour was used or in
301 multiline mode. The "end at original start point" bug is fixed, and if the end
302 point is found to be before the start point, they are swapped.
304 25. When PCRE2_FIRSTLINE without PCRE2_NO_START_OPTIMIZE was used in non-JIT
305 matching (both pcre2_match() and pcre2_dfa_match()) and the matched string
306 started with the first code unit of a newline sequence, matching failed because
307 it was not tried at the newline.
309 26. Code for giving up a non-partial match after failing to find a starting
310 code unit anywhere in the subject was missing when searching for one of a
311 number of code units (the bitmap case) in both pcre2_match() and
312 pcre2_dfa_match(). This was a missing optimization rather than a bug.
314 27. Tidied up the ACROSSCHAR macro to be like FORWARDCHAR and BACKCHAR, using a
315 pointer argument rather than a code unit value. This should not have affected
318 28. The JIT compiler has been updated.
320 29. Avoid pointer overflow for unset captures in pcre2_substring_list_get().
321 This could not actually cause a crash because it was always used in a memcpy()
322 call with zero length.
324 30. Some internal structures have a variable-length ovector[] as their last
325 element. Their actual memory is obtained dynamically, giving an ovector of
326 appropriate length. However, they are defined in the structure as
327 ovector[NUMBER], where NUMBER is large so that array bound checkers don't
328 grumble. The value of NUMBER was 10000, but a fuzzer exceeded 5000 capturing
329 groups, making the ovector larger than this. The number has been increased to
330 131072, which allows for the maximum number of captures (65535) plus the
331 overall match. This fixes oss-fuzz issue 5415.
333 31. Auto-possessification at the end of a capturing group was dependent on what
334 follows the group (e.g. /(a+)b/ would auto-possessify the a+) but this caused
335 incorrect behaviour when the group was called recursively from elsewhere in the
336 pattern where something different might follow. This bug is an unforseen
337 consequence of change #1 for 10.30 - the implementation of backtracking into
338 recursions. Iterators at the ends of capturing groups are no longer considered
339 for auto-possessification if the pattern contains any recursions. Fixes
343 Version 10.30 14-August-2017
344 ----------------------------
346 1. The main interpreter, pcre2_match(), has been refactored into a new version
347 that does not use recursive function calls (and therefore the stack) for
348 remembering backtracking positions. This makes --disable-stack-for-recursion a
349 NOOP. The new implementation allows backtracking into recursive group calls in
350 patterns, making it more compatible with Perl, and also fixes some other
351 hard-to-do issues such as #1887 in Bugzilla. The code is also cleaner because
352 the old code had a number of fudges to try to reduce stack usage. It seems to
353 run no slower than the old code.
355 A number of bugs in the refactored code were subsequently fixed during testing
356 before release, but after the code was made available in the repository. These
357 bugs were never in fully released code, but are noted here for the record.
359 (a) If a pattern had fewer capturing parentheses than the ovector supplied in
360 the match data block, a memory error (detectable by ASAN) occurred after
361 a match, because the external block was being set from non-existent
362 internal ovector fields. Fixes oss-fuzz issue 781.
364 (b) A pattern with very many capturing parentheses (when the internal frame
365 size was greater than the initial frame vector on the stack) caused a
366 crash. A vector on the heap is now set up at the start of matching if the
367 vector on the stack is not big enough to handle at least 10 frames.
368 Fixes oss-fuzz issue 783.
370 (c) Handling of (*VERB)s in recursions was wrong in some cases.
372 (d) Captures in negative assertions that were used as conditions were not
373 happening if the assertion matched via (*ACCEPT).
375 (e) Mark values were not being passed out of recursions.
377 (f) Refactor some code in do_callout() to avoid picky compiler warnings about
378 negative indices. Fixes oss-fuzz issue 1454.
380 (g) Similarly refactor the way the variable length ovector is addressed for
381 similar reasons. Fixes oss-fuzz issue 1465.
383 2. Now that pcre2_match() no longer uses recursive function calls (see above),
384 the "match limit recursion" value seems misnamed. It still exists, and limits
385 the depth of tree that is searched. To avoid future confusion, it has been
386 renamed as "depth limit" in all relevant places (--with-depth-limit,
387 (*LIMIT_DEPTH), pcre2_set_depth_limit(), etc) but the old names are still
388 available for backwards compatibility.
390 3. Hardened pcre2test so as to reduce the number of bugs reported by fuzzers:
392 (a) Check for malloc failures when getting memory for the ovector (POSIX) or
393 the match data block (non-POSIX).
395 4. In the 32-bit library in non-UTF mode, an attempt to find a Unicode property
396 for a character with a code point greater than 0x10ffff (the Unicode maximum)
399 5. If a lookbehind assertion that contained a back reference to a group
400 appearing later in the pattern was compiled with the PCRE2_ANCHORED option,
401 undefined actions (often a segmentation fault) could occur, depending on what
402 other options were set. An example assertion is (?<!\1(abc)) where the
403 reference \1 precedes the group (abc). This fixes oss-fuzz issue 865.
405 6. Added the PCRE2_INFO_FRAMESIZE item to pcre2_pattern_info() and arranged for
406 pcre2test to use it to output the frame size when the "framesize" modifier is
409 7. Reworked the recursive pattern matching in the JIT compiler to follow the
412 8. When the zero_terminate modifier was specified on a pcre2test subject line
413 for global matching, unpredictable things could happen. For example, in UTF-8
414 mode, the pattern //g,zero_terminate read random memory when matched against an
415 empty string with zero_terminate. This was a bug in pcre2test, not the library.
417 9. Moved some Windows-specific code in pcre2grep (introduced in 10.23/13) out
418 of the section that is compiled when Unix-style directory scanning is
419 available, and into a new section that is always compiled for Windows.
421 10. In pcre2test, explicitly close the file after an error during serialization
422 or deserialization (the "load" or "save" commands).
424 11. Fix memory leak in pcre2_serialize_decode() when the input is invalid.
426 12. Fix potential NULL dereference in pcre2_callout_enumerate() if called with
427 a NULL pattern pointer when Unicode support is available.
429 13. When the 32-bit library was being tested by pcre2test, error messages that
430 were longer than 64 code units could cause a buffer overflow. This was a bug in
433 14. The alternative matching function, pcre2_dfa_match() misbehaved if it
434 encountered a character class with a possessive repeat, for example [a-f]{3}+.
436 15. The depth (formerly recursion) limit now applies to DFA matching (as
437 of 10.23/36); pcre2test has been upgraded so that \=find_limits works with DFA
438 matching to find the minimum value for this limit.
440 16. Since 10.21, if pcre2_match() was called with a null context, default
441 memory allocation functions were used instead of whatever was used when the
442 pattern was compiled.
444 17. Changes to the pcre2test "memory" modifier on a subject line. These apply
445 only to pcre2_match():
447 (a) Warn if null_context is set on both pattern and subject, because the
448 memory details cannot then be shown.
450 (b) Remember (up to a certain number of) memory allocations and their
451 lengths, and list only the lengths, so as to be system-independent.
452 (In practice, the new interpreter never has more than 2 blocks allocated
455 18. Make pcre2test detect an error return from pcre2_get_error_message(), give
456 a message, and abandon the run (this would have detected #13 above).
458 19. Implemented PCRE2_ENDANCHORED.
460 20. Applied Jason Hood's patches (slightly modified) to pcre2grep, to implement
461 the --output=text (-O) option and the inbuilt callout echo.
463 21. Extend auto-anchoring etc. to ignore groups with a zero qualifier and
464 single-branch conditions with a false condition (e.g. DEFINE) at the start of a
465 branch. For example, /(?(DEFINE)...)^A/ and /(...){0}^B/ are now flagged as
468 22. Added an explicit limit on the amount of heap used by pcre2_match(), set by
469 pcre2_set_heap_limit() or (*LIMIT_HEAP=xxx). Upgraded pcre2test to show the
470 heap limit along with other pattern information, and to find the minimum when
471 the find_limits modifier is set.
473 23. Write to the last 8 bytes of the pcre2_real_code structure when a compiled
474 pattern is set up so as to initialize any padding the compiler might have
475 included. This avoids valgrind warnings when a compiled pattern is copied, in
476 particular when it is serialized.
478 24. Remove a redundant line of code left in accidentally a long time ago.
480 25. Remove a duplication typo in pcre2_tables.c
482 26. Correct an incorrect cast in pcre2_valid_utf.c
484 27. Update pcre2test, remove some unused code in pcre2_match(), and upgrade the
485 tests to improve coverage.
487 28. Some fixes/tidies as a result of looking at Coverity Scan output:
489 (a) Typo: ">" should be ">=" in opcode check in pcre2_auto_possess.c.
490 (b) Added some casts to avoid "suspicious implicit sign extension".
491 (c) Resource leaks in pcre2test in rare error cases.
492 (d) Avoid warning for never-use case OP_TABLE_LENGTH which is just a fudge
493 for checking at compile time that tables are the right size.
494 (e) Add missing "fall through" comment.
496 29. Implemented PCRE2_EXTENDED_MORE and related /xx and (?xx) features.
498 30. Implement (?n: for PCRE2_NO_AUTO_CAPTURE, because Perl now has this.
500 31. If more than one of "push", "pushcopy", or "pushtablescopy" were set in
501 pcre2test, a crash could occur.
503 32. Make -bigstack in RunTest allocate a 64MiB stack (instead of 16MiB) so
504 that all the tests can run with clang's sanitizing options.
506 33. Implement extra compile options in the compile context and add the first
507 one: PCRE2_EXTRA_ALLOW_SURROGATE_ESCAPES.
509 34. Implement newline type PCRE2_NEWLINE_NUL.
511 35. A lookbehind assertion that had a zero-length branch caused undefined
512 behaviour when processed by pcre2_dfa_match(). This is oss-fuzz issue 1859.
514 36. The match limit value now also applies to pcre2_dfa_match() as there are
515 patterns that can use up a lot of resources without necessarily recursing very
516 deeply. (Compare item 10.23/36.) This should fix oss-fuzz #1761.
518 37. Implement PCRE2_EXTRA_BAD_ESCAPE_IS_LITERAL.
520 38. Fix returned offsets from regexec() when REG_STARTEND is used with a
521 starting offset greater than zero.
523 39. Implement REG_PEND (GNU extension) for the POSIX wrapper.
525 40. Implement the subject_literal modifier in pcre2test, and allow jitstack on
528 41. Implement PCRE2_LITERAL and use it to support REG_NOSPEC.
530 42. Implement PCRE2_EXTRA_MATCH_LINE and PCRE2_EXTRA_MATCH_WORD for the benefit
533 43. Re-implement pcre2grep's -F, -w, and -x options using PCRE2_LITERAL,
534 PCRE2_EXTRA_MATCH_WORD, and PCRE2_EXTRA_MATCH_LINE. This fixes two bugs:
536 (a) The -F option did not work for fixed strings containing \E.
537 (b) The -w option did not work for patterns with multiple branches.
539 44. Added configuration options for the SELinux compatible execmem allocator in
542 45. Increased the limit for searching for a "must be present" code unit in
543 subjects from 1000 to 2000 for 8-bit searches, since they use memchr() and are
546 46. Arrange for anchored patterns to record and use "first code unit" data,
547 because this can give a fast "no match" without searching for a "required code
548 unit". Previously only non-anchored patterns did this.
550 47. Upgraded the Unicode tables from Unicode 8.0.0 to Unicode 10.0.0.
552 48. Add the callout_no_where modifier to pcre2test.
554 49. Update extended grapheme breaking rules to the latest set that are in
555 Unicode Standard Annex #29.
557 50. Added experimental foreign pattern conversion facilities
558 (pcre2_pattern_convert() and friends).
560 51. Change the macro FWRITE, used in pcre2grep, to FWRITE_IGNORE because FWRITE
561 is defined in a system header in cygwin. Also modified some of the #ifdefs in
562 pcre2grep related to Windows and Cygwin support.
564 52. Change 3(g) for 10.23 was a bit too zealous. If a hyphen that follows a
565 character class is the last character in the class, Perl does not give a
566 warning. PCRE2 now also treats this as a literal.
568 53. Related to 52, though PCRE2 was throwing an error for [[:digit:]-X] it was
569 not doing so for [\d-X] (and similar escapes), as is documented.
571 54. Fixed a MIPS issue in the JIT compiler reported by Joshua Kinard.
573 55. Fixed a "maybe uninitialized" warning for class_uchardata in \p handling in
574 pcre2_compile() which could never actually trigger (code should have been cut
575 out when Unicode support is disabled).
578 Version 10.23 14-February-2017
579 ------------------------------
581 1. Extended pcre2test with the utf8_input modifier so that it is able to
582 generate all possible 16-bit and 32-bit code unit values in non-UTF modes.
584 2. In any wide-character mode (8-bit UTF or any 16-bit or 32-bit mode), without
585 PCRE2_UCP set, a negative character type such as \D in a positive class should
586 cause all characters greater than 255 to match, whatever else is in the class.
587 There was a bug that caused this not to happen if a Unicode property item was
588 added to such a class, for example [\D\P{Nd}] or [\W\pL].
590 3. There has been a major re-factoring of the pcre2_compile.c file. Most syntax
591 checking is now done in the pre-pass that identifies capturing groups. This has
592 reduced the amount of duplication and made the code tidier. While doing this,
593 some minor bugs and Perl incompatibilities were fixed, including:
595 (a) \Q\E in the middle of a quantifier such as A+\Q\E+ is now ignored instead
596 of giving an invalid quantifier error.
598 (b) {0} can now be used after a group in a lookbehind assertion; previously
599 this caused an "assertion is not fixed length" error.
601 (c) Perl always treats (?(DEFINE) as a "define" group, even if a group with
602 the name "DEFINE" exists. PCRE2 now does likewise.
604 (d) A recursion condition test such as (?(R2)...) must now refer to an
607 (e) A conditional recursion test such as (?(R)...) misbehaved if there was a
608 group whose name began with "R".
610 (f) When testing zero-terminated patterns under valgrind, the terminating
611 zero is now marked "no access". This catches bugs that would otherwise
612 show up only with non-zero-terminated patterns.
614 (g) A hyphen appearing immediately after a POSIX character class (for example
615 /[[:ascii:]-z]/) now generates an error. Perl does accept this as a
616 literal, but gives a warning, so it seems best to fail it in PCRE.
618 (h) An empty \Q\E sequence may appear after a callout that precedes an
619 assertion condition (it is, of course, ignored).
621 One effect of the refactoring is that some error numbers and messages have
622 changed, and the pattern offset given for compiling errors is not always the
623 right-most character that has been read. In particular, for a variable-length
624 lookbehind assertion it now points to the start of the assertion. Another
625 change is that when a callout appears before a group, the "length of next
626 pattern item" that is passed now just gives the length of the opening
627 parenthesis item, not the length of the whole group. A length of zero is now
628 given only for a callout at the end of the pattern. Automatic callouts are no
629 longer inserted before and after explicit callouts in the pattern.
631 A number of bugs in the refactored code were subsequently fixed during testing
632 before release, but after the code was made available in the repository. Many
633 of the bugs were discovered by fuzzing testing. Several of them were related to
634 the change from assuming a zero-terminated pattern (which previously had
635 required non-zero terminated strings to be copied). These bugs were never in
636 fully released code, but are noted here for the record.
638 (a) An overall recursion such as (?0) inside a lookbehind assertion was not
639 being diagnosed as an error.
641 (b) In utf mode, the length of a *MARK (or other verb) name was being checked
642 in characters instead of code units, which could lead to bad code being
643 compiled, leading to unpredictable behaviour.
645 (c) In extended /x mode, characters whose code was greater than 255 caused
646 a lookup outside one of the global tables. A similar bug existed for wide
647 characters in *VERB names.
649 (d) The amount of memory needed for a compiled pattern was miscalculated if a
650 lookbehind contained more than one toplevel branch and the first branch
653 (e) In UTF-8 or UTF-16 modes with PCRE2_EXTENDED (/x) set and a non-zero-
654 terminated pattern, if a # comment ran on to the end of the pattern, one
655 or more code units past the end were being read.
657 (f) An unterminated repeat at the end of a non-zero-terminated pattern (e.g.
658 "{2,2") could cause reading beyond the pattern.
660 (g) When reading a callout string, if the end delimiter was at the end of the
661 pattern one further code unit was read.
663 (h) An unterminated number after \g' could cause reading beyond the pattern.
665 (i) An insufficient memory size was being computed for compiling with
668 (j) A conditional group with an assertion condition used more memory than was
669 allowed for it during parsing, so too many of them could therefore
672 (k) If parsing a pattern exactly filled the buffer, the internal test for
673 overrun did not check when the final META_END item was added.
675 (l) If a lookbehind contained a subroutine call, and the called group
676 contained an option setting such as (?s), and the PCRE2_ANCHORED option
677 was set, unpredictable behaviour could occur. The underlying bug was
678 incorrect code and insufficient checking while searching for the end of
679 the called subroutine in the parsed pattern.
681 (m) Quantifiers following (*VERB)s were not being diagnosed as errors.
683 (n) The use of \Q...\E in a (*VERB) name when PCRE2_ALT_VERBNAMES and
684 PCRE2_AUTO_CALLOUT were both specified caused undetermined behaviour.
686 (o) If \Q was preceded by a quantified item, and the following \E was
687 followed by '?' or '+', and there was at least one literal character
688 between them, an internal error "unexpected repeat" occurred (example:
691 (p) A buffer overflow could occur while sorting the names in the group name
692 list (depending on the order in which the names were seen).
694 (q) A conditional group that started with a callout was not doing the right
695 check for a following assertion, leading to compiling bad code. Example:
698 (r) If a character whose code point was greater than 0xffff appeared within
699 a lookbehind that was within another lookbehind, the calculation of the
700 lookbehind length went wrong and could provoke an internal error.
702 (t) The sequence \E- or \Q\E- after a POSIX class in a character class caused
703 an internal error. Now the hyphen is treated as a literal.
705 4. Back references are now permitted in lookbehind assertions when there are
706 no duplicated group numbers (that is, (?| has not been used), and, if the
707 reference is by name, there is only one group of that name. The referenced
708 group must, of course be of fixed length.
710 5. pcre2test has been upgraded so that, when run under valgrind with valgrind
711 support enabled, reading past the end of the pattern is detected, both when
712 compiling and during callout processing.
714 6. \g{+<number>} (e.g. \g{+2} ) is now supported. It is a "forward back
715 reference" and can be useful in repetitions (compare \g{-<number>} ). Perl does
716 not recognize this syntax.
718 7. Automatic callouts are no longer generated before and after callouts in the
721 8. When pcre2test was outputing information from a callout, the caret indicator
722 for the current position in the subject line was incorrect if it was after an
723 escape sequence for a character whose code point was greater than \x{ff}.
725 9. Change 19 for 10.22 had a typo (PCRE_STATIC_RUNTIME should be
726 PCRE2_STATIC_RUNTIME). Fix from David Gaussmann.
728 10. Added --max-buffer-size to pcre2grep, to allow for automatic buffer
729 expansion when long lines are encountered. Original patch by Dmitry
732 11. If pcre2grep was compiled with JIT support, but the library was compiled
733 without it (something that neither ./configure nor CMake allow, but it can be
734 done by editing config.h), pcre2grep was giving a JIT error. Now it detects
735 this situation and does not try to use JIT.
737 12. Added some "const" qualifiers to variables in pcre2grep.
739 13. Added Dmitry Cherniachenko's patch for colouring output in Windows
740 (untested by me). Also, look for GREP_COLOUR or GREP_COLOR if the environment
741 variables PCRE2GREP_COLOUR and PCRE2GREP_COLOR are not found.
743 14. Add the -t (grand total) option to pcre2grep.
745 15. A number of bugs have been mended relating to match start-up optimizations
746 when the first thing in a pattern is a positive lookahead. These all applied
747 only when PCRE2_NO_START_OPTIMIZE was *not* set:
749 (a) A pattern such as (?=.*X)X$ was incorrectly optimized as if it needed
750 both an initial 'X' and a following 'X'.
751 (b) Some patterns starting with an assertion that started with .* were
752 incorrectly optimized as having to match at the start of the subject or
753 after a newline. There are cases where this is not true, for example,
754 (?=.*[A-Z])(?=.{8,16})(?!.*[\s]) matches after the start in lines that
755 start with spaces. Starting .* in an assertion is no longer taken as an
756 indication of matching at the start (or after a newline).
758 16. The "offset" modifier in pcre2test was not being ignored (as documented)
759 when the POSIX API was in use.
761 17. Added --enable-fuzz-support to "configure", causing an non-installed
762 library containing a test function that can be called by fuzzers to be
763 compiled. A non-installed binary to run the test function locally, called
764 pcre2fuzzcheck is also compiled.
766 18. A pattern with PCRE2_DOTALL (/s) set but not PCRE2_NO_DOTSTAR_ANCHOR, and
767 which started with .* inside a positive lookahead was incorrectly being
768 compiled as implicitly anchored.
770 19. Removed all instances of "register" declarations, as they are considered
771 obsolete these days and in any case had become very haphazard.
773 20. Add strerror() to pcre2test for failed file opening.
775 21. Make pcre2test -C list valgrind support when it is enabled.
777 22. Add the use_length modifier to pcre2test.
779 23. Fix an off-by-one bug in pcre2test for the list of names for 'get' and
782 24. Add PCRE2_CALL_CONVENTION into the prototype declarations in pcre2.h as it
783 is apparently needed there as well as in the function definitions. (Why did
784 nobody ask for this in PCRE1?)
786 25. Change the _PCRE2_H and _PCRE2_UCP_H guard macros in the header files to
787 PCRE2_H_IDEMPOTENT_GUARD and PCRE2_UCP_H_IDEMPOTENT_GUARD to be more standard
788 compliant and unique.
790 26. pcre2-config --libs-posix was listing -lpcre2posix instead of
791 -lpcre2-posix. Also, the CMake build process was building the library with the
794 27. In pcre2test, give some offset information for errors in hex patterns.
795 This uses the C99 formatting sequence %td, except for MSVC which doesn't
796 support it - %lu is used instead.
798 28. Implemented pcre2_code_copy_with_tables(), and added pushtablescopy to
799 pcre2test for testing it.
801 29. Fix small memory leak in pcre2test.
803 30. Fix out-of-bounds read for partial matching of /./ against an empty string
804 when the newline type is CRLF.
806 31. Fix a bug in pcre2test that caused a crash when a locale was set either in
807 the current pattern or a previous one and a wide character was matched.
809 32. The appearance of \p, \P, or \X in a substitution string when
810 PCRE2_SUBSTITUTE_EXTENDED was set caused a segmentation fault (NULL
813 33. If the starting offset was specified as greater than the subject length in
814 a call to pcre2_substitute() an out-of-bounds memory reference could occur.
816 34. When PCRE2 was compiled to use the heap instead of the stack for recursive
817 calls to match(), a repeated minimizing caseless back reference, or a
818 maximizing one where the two cases had different numbers of code units,
819 followed by a caseful back reference, could lose the caselessness of the first
820 repeated back reference (example: /(Z)(a)\2{1,2}?(?-i)\1X/i should match ZaAAZX
823 35. When a pattern is too complicated, PCRE2 gives up trying to find a minimum
824 matching length and just records zero. Typically this happens when there are
825 too many nested or recursive back references. If the limit was reached in
826 certain recursive cases it failed to be triggered and an internal error could
829 36. The pcre2_dfa_match() function now takes note of the recursion limit for
830 the internal recursive calls that are used for lookrounds and recursions within
833 37. More refactoring has got rid of the internal could_be_empty_branch()
834 function (around 400 lines of code, including comments) by keeping track of
835 could-be-emptiness as the pattern is compiled instead of scanning compiled
836 groups. (This would have been much harder before the refactoring of #3 above.)
837 This lifts a restriction on the number of branches in a group (more than about
838 1100 would give "pattern is too complicated").
840 38. Add the "-ac" command line option to pcre2test as a synonym for "-pattern
843 39. In a library with Unicode support, incorrect data was compiled for a
844 pattern with PCRE2_UCP set without PCRE2_UTF if a class required all wide
845 characters to match (for example, /[\s[:^ascii:]]/).
847 40. The callout_error modifier has been added to pcre2test to make it possible
848 to return PCRE2_ERROR_CALLOUT from a callout.
850 41. A minor change to pcre2grep: colour reset is now "<esc>[0m" instead of
853 42. The limit in the auto-possessification code that was intended to catch
854 overly-complicated patterns and not spend too much time auto-possessifying was
855 being reset too often, resulting in very long compile times for some patterns.
856 Now such patterns are no longer completely auto-possessified.
858 43. Applied Jason Hood's revised patch for RunTest.bat.
860 44. Added a new Windows script RunGrepTest.bat, courtesy of Jason Hood.
862 45. Minor cosmetic fix to pcre2test: move a variable that is not used under
863 Windows into the "not Windows" code.
865 46. Applied Jason Hood's patches to upgrade pcre2grep under Windows and tidy
868 * normalised the Windows condition by ensuring WIN32 is defined;
869 * enables the callout feature under Windows;
870 * adds globbing (Microsoft's implementation expands quoted args),
871 using a tweaked opendirectory;
872 * implements the is_*_tty functions for Windows;
873 * --color=always will write the ANSI sequences to file;
874 * add sequences 4 (underline works on Win10) and 5 (blink as bright
875 background, relatively standard on DOS/Win);
876 * remove the (char *) casts for the now-const strings;
877 * remove GREP_COLOUR (grep's command line allowed the 'u', but not
878 the environment), parsing GREP_COLORS instead;
879 * uses the current colour if not set, rather than black;
880 * add print_match for the undefined case;
883 In addition, colour settings containing anything other than digits and
884 semicolon are ignored, and the colour controls are no longer output for empty
887 47. Detecting patterns that are too large inside the length-measuring loop
888 saves processing ridiculously long patterns to their end.
890 48. Ignore PCRE2_CASELESS when processing \h, \H, \v, and \V in classes as it
891 just wastes time. In the UTF case it can also produce redundant entries in
892 XCLASS lists caused by characters with multiple other cases and pairs of
893 characters in the same "not-x" sublists.
895 49. A pattern such as /(?=(a\K))/ can report the end of the match being before
896 its start; pcre2test was not handling this correctly when using the POSIX
897 interface (it was OK with the native interface).
899 50. In pcre2grep, ignore all JIT compile errors. This means that pcre2grep will
900 continue to work, falling back to interpretation if anything goes wrong with
903 51. Applied patches from Christian Persch to configure.ac to make use of the
904 AC_USE_SYSTEM_EXTENSIONS macro and to test for functions used by the JIT
907 52. Minor fixes to pcre2grep from Jason Hood:
908 * fixed some spacing;
909 * Windows doesn't usually use single quotes, so I've added a define
910 to use appropriate quotes [in an example];
911 * LC_ALL was displayed as "LCC_ALL";
912 * numbers 11, 12 & 13 should end in "th";
913 * use double quotes in usage message.
915 53. When autopossessifying, skip empty branches without recursion, to reduce
916 stack usage for the benefit of clang with -fsanitize-address, which uses huge
917 stack frames. Example pattern: /X?(R||){3335}/. Fixes oss-fuzz issue 553.
919 54. A pattern with very many explicit back references to a group that is a long
920 way from the start of the pattern could take a long time to compile because
921 searching for the referenced group in order to find the minimum length was
922 being done repeatedly. Now up to 128 group minimum lengths are cached and the
923 attempt to find a minimum length is abandoned if there is a back reference to a
924 group whose number is greater than 128. (In that case, the pattern is so
925 complicated that this optimization probably isn't worth it.) This fixes
928 55. Issue 32 for 10.22 below was not correctly fixed. If pcre2grep in multiline
929 mode with --only-matching matched several lines, it restarted scanning at the
930 next line instead of moving on to the end of the matched string, which can be
931 several lines after the start.
933 56. Applied Jason Hood's new patch for RunGrepTest.bat that updates it in line
934 with updates to the non-Windows version.
938 Version 10.22 29-July-2016
939 --------------------------
941 1. Applied Jason Hood's patches to RunTest.bat and testdata/wintestoutput3
942 to fix problems with running the tests under Windows.
944 2. Implemented a facility for quoting literal characters within hexadecimal
945 patterns in pcre2test, to make it easier to create patterns with just a few
946 non-printing characters.
948 3. Binary zeros are not supported in pcre2test input files. It now detects them
951 4. Updated the valgrind parameters in RunTest: (a) changed smc-check=all to
952 smc-check=all-non-file; (b) changed obj:* in the suppression file to obj:??? so
953 that it matches only unknown objects.
955 5. Updated the maintenance script maint/ManyConfigTests to make it easier to
956 select individual groups of tests.
958 6. When the POSIX wrapper function regcomp() is called, the REG_NOSUB option
959 used to set PCRE2_NO_AUTO_CAPTURE when calling pcre2_compile(). However, this
960 disables the use of back references (and subroutine calls), which are supported
961 by other implementations of regcomp() with RE_NOSUB. Therefore, REG_NOSUB no
962 longer causes PCRE2_NO_AUTO_CAPTURE to be set, though it still ignores nmatch
963 and pmatch when regexec() is called.
965 7. Because of 6 above, pcre2test has been modified with a new modifier called
966 posix_nosub, to call regcomp() with REG_NOSUB. Previously the no_auto_capture
967 modifier had this effect. That option is now ignored when the POSIX API is in
970 8. Minor tidies to the pcre2demo.c sample program, including more comments
971 about its 8-bit-ness.
973 9. Detect unmatched closing parentheses and give the error in the pre-scan
974 instead of later. Previously the pre-scan carried on and could give a
975 misleading incorrect error message. For example, /(?J)(?'a'))(?'a')/ gave a
976 message about invalid duplicate group names.
978 10. It has happened that pcre2test was accidentally linked with another POSIX
979 regex library instead of libpcre2-posix. In this situation, a call to regcomp()
980 (in the other library) may succeed, returning zero, but of course putting its
981 own data into the regex_t block. In one example the re_pcre2_code field was
982 left as NULL, which made pcre2test think it had not got a compiled POSIX regex,
983 so it treated the next line as another pattern line, resulting in a confusing
984 error message. A check has been added to pcre2test to see if the data returned
985 from a successful call of regcomp() are valid for PCRE2's regcomp(). If they
986 are not, an error message is output and the pcre2test run is abandoned. The
987 message points out the possibility of a mis-linking. Hopefully this will avoid
988 some head-scratching the next time this happens.
990 11. A pattern such as /(?<=((?C)0))/, which has a callout inside a lookbehind
991 assertion, caused pcre2test to output a very large number of spaces when the
992 callout was taken, making the program appearing to loop.
994 12. A pattern that included (*ACCEPT) in the middle of a sufficiently deeply
995 nested set of parentheses of sufficient size caused an overflow of the
996 compiling workspace (which was diagnosed, but of course is not desirable).
998 13. Detect missing closing parentheses during the pre-pass for group
1001 14. Changed some integer variable types and put in a number of casts, following
1002 a report of compiler warnings from Visual Studio 2013 and a few tests with
1003 gcc's -Wconversion (which still throws up a lot).
1005 15. Implemented pcre2_code_copy(), and added pushcopy and #popcopy to pcre2test
1008 16. Change 66 for 10.21 introduced the use of snprintf() in PCRE2's version of
1009 regerror(). When the error buffer is too small, my version of snprintf() puts a
1010 binary zero in the final byte. Bug #1801 seems to show that other versions do
1011 not do this, leading to bad output from pcre2test when it was checking for
1012 buffer overflow. It no longer assumes a binary zero at the end of a too-small
1015 17. Fixed typo ("&&" for "&") in pcre2_study(). Fortunately, this could not
1016 actually affect anything, by sheer luck.
1018 18. Two minor fixes for MSVC compilation: (a) removal of apparently incorrect
1019 "const" qualifiers in pcre2test and (b) defining snprintf as _snprintf for
1020 older MSVC compilers. This has been done both in src/pcre2_internal.h for most
1021 of the library, and also in src/pcre2posix.c, which no longer includes
1022 pcre2_internal.h (see 24 below).
1024 19. Applied Chris Wilson's patch (Bugzilla #1681) to CMakeLists.txt for MSVC
1025 static compilation. Subsequently applied Chris Wilson's second patch, putting
1026 the first patch under a new option instead of being unconditional when
1029 20. Updated pcre2grep to set stdout as binary when run under Windows, so as not
1030 to convert \r\n at the ends of reflected lines into \r\r\n. This required
1031 ensuring that other output that is written to stdout (e.g. file names) uses the
1032 appropriate line terminator: \r\n for Windows, \n otherwise.
1034 21. When a line is too long for pcre2grep's internal buffer, show the maximum
1035 length in the error message.
1037 22. Added support for string callouts to pcre2grep (Zoltan's patch with PH
1040 23. RunTest.bat was missing a "set type" line for test 22.
1042 24. The pcre2posix.c file was including pcre2_internal.h, and using some
1043 "private" knowledge of the data structures. This is unnecessary; the code has
1044 been re-factored and no longer includes pcre2_internal.h.
1046 25. A racing condition is fixed in JIT reported by Mozilla.
1048 26. Minor code refactor to avoid "array subscript is below array bounds"
1051 27. Minor code refactor to avoid "left shift of negative number" warning.
1053 28. Add a bit more sanity checking to pcre2_serialize_decode() and document
1054 that it expects trusted data.
1056 29. Fix typo in pcre2_jit_test.c
1058 30. Due to an oversight, pcre2grep was not making use of JIT when available.
1061 31. The RunGrepTest script is updated to use the valgrind suppressions file
1062 when testing with JIT under valgrind (compare 10.21/51 below). The suppressions
1063 file is updated so that is now the same as for PCRE1: it suppresses the
1064 Memcheck warnings Addr16 and Cond in unknown objects (that is, JIT-compiled
1065 code). Also changed smc-check=all to smc-check=all-non-file as was done for
1066 RunTest (see 4 above).
1068 32. Implemented the PCRE2_NO_JIT option for pcre2_match().
1070 33. Fix typo that gave a compiler error when JIT not supported.
1072 34. Fix comment describing the returns from find_fixedlength().
1074 35. Fix potential negative index in pcre2test.
1076 36. Calls to pcre2_get_error_message() with error numbers that are never
1077 returned by PCRE2 functions were returning empty strings. Now the error code
1078 PCRE2_ERROR_BADDATA is returned. A facility has been added to pcre2test to
1079 show the texts for given error numbers (i.e. to call pcre2_get_error_message()
1080 and display what it returns) and a few representative error codes are now
1083 37. Added "&& !defined(__INTEL_COMPILER)" to the test for __GNUC__ in
1084 pcre2_match.c, in anticipation that this is needed for the same reason it was
1085 recently added to pcrecpp.cc in PCRE1.
1087 38. Using -o with -M in pcre2grep could cause unnecessary repeated output when
1088 the match extended over a line boundary, as it tried to find more matches "on
1089 the same line" - but it was already over the end.
1091 39. Allow \C in lookbehinds and DFA matching in UTF-32 mode (by converting it
1092 to the same code as '.' when PCRE2_DOTALL is set).
1094 40. Fix two clang compiler warnings in pcre2test when only one code unit width
1097 41. Upgrade RunTest to automatically re-run test 2 with a large (64MiB) stack
1098 if it fails when running the interpreter with a 16MiB stack (and if changing
1099 the stack size via pcre2test is possible). This avoids having to manually set a
1100 large stack size when testing with clang.
1102 42. Fix register overwite in JIT when SSE2 acceleration is enabled.
1104 43. Detect integer overflow in pcre2test pattern and data repetition counts.
1106 44. In pcre2test, ignore "allcaptures" after DFA matching.
1108 45. Fix unaligned accesses on x86. Patch by Marc Mutz.
1110 46. Fix some more clang compiler warnings.
1113 Version 10.21 12-January-2016
1114 -----------------------------
1116 1. Improve matching speed of patterns starting with + or * in JIT.
1118 2. Use memchr() to find the first character in an unanchored match in 8-bit
1119 mode in the interpreter. This gives a significant speed improvement.
1121 3. Removed a redundant copy of the opcode_possessify table in the
1122 pcre2_auto_possessify.c source.
1124 4. Fix typos in dftables.c for z/OS.
1126 5. Change 36 for 10.20 broke the handling of [[:>:]] and [[:<:]] in that
1127 processing them could involve a buffer overflow if the following character was
1128 an opening parenthesis.
1130 6. Change 36 for 10.20 also introduced a bug in processing this pattern:
1131 /((?x)(*:0))#(?'/. Specifically: if a setting of (?x) was followed by a (*MARK)
1132 setting (which (*:0) is), then (?x) did not get unset at the end of its group
1133 during the scan for named groups, and hence the external # was incorrectly
1134 treated as a comment and the invalid (?' at the end of the pattern was not
1135 diagnosed. This caused a buffer overflow during the real compile. This bug was
1136 discovered by Karl Skomski with the LLVM fuzzer.
1138 7. Moved the pcre2_find_bracket() function from src/pcre2_compile.c into its
1139 own source module to avoid a circular dependency between src/pcre2_compile.c
1140 and src/pcre2_study.c
1142 8. A callout with a string argument containing an opening square bracket, for
1143 example /(?C$[$)(?<]/, was incorrectly processed and could provoke a buffer
1144 overflow. This bug was discovered by Karl Skomski with the LLVM fuzzer.
1146 9. The handling of callouts during the pre-pass for named group identification
1147 has been tightened up.
1149 10. The quantifier {1} can be ignored, whether greedy, non-greedy, or
1150 possessive. This is a very minor optimization.
1152 11. A possessively repeated conditional group that could match an empty string,
1153 for example, /(?(R))*+/, was incorrectly compiled.
1155 12. The Unicode tables have been updated to Unicode 8.0.0 (thanks to Christian
1158 13. An empty comment (?#) in a pattern was incorrectly processed and could
1159 provoke a buffer overflow. This bug was discovered by Karl Skomski with the
1162 14. Fix infinite recursion in the JIT compiler when certain patterns such as
1163 /(?:|a|){100}x/ are analysed.
1165 15. Some patterns with character classes involving [: and \\ were incorrectly
1166 compiled and could cause reading from uninitialized memory or an incorrect
1167 error diagnosis. Examples are: /[[:\\](?<[::]/ and /[[:\\](?'abc')[a:]. The
1168 first of these bugs was discovered by Karl Skomski with the LLVM fuzzer.
1170 16. Pathological patterns containing many nested occurrences of [: caused
1171 pcre2_compile() to run for a very long time. This bug was found by the LLVM
1174 17. A missing closing parenthesis for a callout with a string argument was not
1175 being diagnosed, possibly leading to a buffer overflow. This bug was found by
1178 18. A conditional group with only one branch has an implicit empty alternative
1179 branch and must therefore be treated as potentially matching an empty string.
1181 19. If (?R was followed by - or + incorrect behaviour happened instead of a
1182 diagnostic. This bug was discovered by Karl Skomski with the LLVM fuzzer.
1184 20. Another bug that was introduced by change 36 for 10.20: conditional groups
1185 whose condition was an assertion preceded by an explicit callout with a string
1186 argument might be incorrectly processed, especially if the string contained \Q.
1187 This bug was discovered by Karl Skomski with the LLVM fuzzer.
1189 21. Compiling PCRE2 with the sanitize options of clang showed up a number of
1190 very pedantic coding infelicities and a buffer overflow while checking a UTF-8
1191 string if the final multi-byte UTF-8 character was truncated.
1193 22. For Perl compatibility in EBCDIC environments, ranges such as a-z in a
1194 class, where both values are literal letters in the same case, omit the
1195 non-letter EBCDIC code points within the range.
1197 23. Finding the minimum matching length of complex patterns with back
1198 references and/or recursions can take a long time. There is now a cut-off that
1199 gives up trying to find a minimum length when things get too complex.
1201 24. An optimization has been added that speeds up finding the minimum matching
1202 length for patterns containing repeated capturing groups or recursions.
1204 25. If a pattern contained a back reference to a group whose number was
1205 duplicated as a result of appearing in a (?|...) group, the computation of the
1206 minimum matching length gave a wrong result, which could cause incorrect "no
1207 match" errors. For such patterns, a minimum matching length cannot at present
1210 26. Added a check for integer overflow in conditions (?(<digits>) and
1211 (?(R<digits>). This omission was discovered by Karl Skomski with the LLVM
1214 27. Fixed an issue when \p{Any} inside an xclass did not read the current
1217 28. If pcre2grep was given the -q option with -c or -l, or when handling a
1218 binary file, it incorrectly wrote output to stdout.
1220 29. The JIT compiler did not restore the control verb head in case of *THEN
1221 control verbs. This issue was found by Karl Skomski with a custom LLVM fuzzer.
1223 30. The way recursive references such as (?3) are compiled has been re-written
1224 because the old way was the cause of many issues. Now, conversion of the group
1225 number into a pattern offset does not happen until the pattern has been
1226 completely compiled. This does mean that detection of all infinitely looping
1227 recursions is postponed till match time. In the past, some easy ones were
1228 detected at compile time. This re-writing was done in response to yet another
1229 bug found by the LLVM fuzzer.
1231 31. A test for a back reference to a non-existent group was missing for items
1232 such as \987. This caused incorrect code to be compiled. This issue was found
1233 by Karl Skomski with a custom LLVM fuzzer.
1235 32. Error messages for syntax errors following \g and \k were giving inaccurate
1236 offsets in the pattern.
1238 33. Improve the performance of starting single character repetitions in JIT.
1240 34. (*LIMIT_MATCH=) now gives an error instead of setting the value to 0.
1242 35. Error messages for syntax errors in *LIMIT_MATCH and *LIMIT_RECURSION now
1243 give the right offset instead of zero.
1245 36. The JIT compiler should not check repeats after a {0,1} repeat byte code.
1246 This issue was found by Karl Skomski with a custom LLVM fuzzer.
1248 37. The JIT compiler should restore the control chain for empty possessive
1249 repeats. This issue was found by Karl Skomski with a custom LLVM fuzzer.
1251 38. A bug which was introduced by the single character repetition optimization
1254 39. Match limit check added to recursion. This issue was found by Karl Skomski
1255 with a custom LLVM fuzzer.
1257 40. Arrange for the UTF check in pcre2_match() and pcre2_dfa_match() to look
1258 only at the part of the subject that is relevant when the starting offset is
1261 41. Improve first character match in JIT with SSE2 on x86.
1263 42. Fix two assertion fails in JIT. These issues were found by Karl Skomski
1264 with a custom LLVM fuzzer.
1266 43. Correct the setting of CMAKE_C_FLAGS in CMakeLists.txt (patch from Roy Ivy
1269 44. Fix bug in RunTest.bat for new test 14, and adjust the script for the added
1270 test (there are now 20 in total).
1272 45. Fixed a corner case of range optimization in JIT.
1274 46. Add the ${*MARK} facility to pcre2_substitute().
1276 47. Modifier lists in pcre2test were splitting at spaces without the required
1279 48. Implemented PCRE2_ALT_VERBNAMES.
1281 49. Fixed two issues in JIT. These were found by Karl Skomski with a custom
1284 50. The pcre2test program has been extended by adding the #newline_default
1285 command. This has made it possible to run the standard tests when PCRE2 is
1286 compiled with either CR or CRLF as the default newline convention. As part of
1287 this work, the new command was added to several test files and the testing
1288 scripts were modified. The pcre2grep tests can now also be run when there is no
1289 LF in the default newline convention.
1291 51. The RunTest script has been modified so that, when JIT is used and valgrind
1292 is specified, a valgrind suppressions file is set up to ignore "Invalid read of
1293 size 16" errors because these are false positives when the hardware supports
1294 the SSE2 instruction set.
1296 52. It is now possible to have comment lines amid the subject strings in
1297 pcre2test (and perltest.sh) input.
1299 53. Implemented PCRE2_USE_OFFSET_LIMIT and pcre2_set_offset_limit().
1301 54. Add the null_context modifier to pcre2test so that calling pcre2_compile()
1302 and the matching functions with NULL contexts can be tested.
1304 55. Implemented PCRE2_SUBSTITUTE_EXTENDED.
1306 56. In a character class such as [\W\p{Any}] where both a negative-type escape
1307 ("not a word character") and a property escape were present, the property
1308 escape was being ignored.
1310 57. Fixed integer overflow for patterns whose minimum matching length is very,
1313 58. Implemented --never-backslash-C.
1315 59. Change 55 above introduced a bug by which certain patterns provoked the
1316 erroneous error "\ at end of pattern".
1318 60. The special sequences [[:<:]] and [[:>:]] gave rise to incorrect compiling
1319 errors or other strange effects if compiled in UCP mode. Found with libFuzzer
1320 and AddressSanitizer.
1322 61. Whitespace at the end of a pcre2test pattern line caused a spurious error
1323 message if there were only single-character modifiers. It should be ignored.
1325 62. The use of PCRE2_NO_AUTO_CAPTURE could cause incorrect compilation results
1326 or segmentation errors for some patterns. Found with libFuzzer and
1329 63. Very long names in (*MARK) or (*THEN) etc. items could provoke a buffer
1332 64. Improve error message for overly-complicated patterns.
1334 65. Implemented an optional replication feature for patterns in pcre2test, to
1335 make it easier to test long repetitive patterns. The tests for 63 above are
1336 converted to use the new feature.
1338 66. In the POSIX wrapper, if regerror() was given too small a buffer, it could
1341 67. In pcre2_substitute() in UTF mode, the UTF validity check on the
1342 replacement string was happening before the length setting when the replacement
1343 string was zero-terminated.
1345 68. In pcre2_substitute() in UTF mode, PCRE2_NO_UTF_CHECK can be set for the
1346 second and subsequent calls to pcre2_match().
1348 69. There was no check for integer overflow for a replacement group number in
1349 pcre2_substitute(). An added check for a number greater than the largest group
1350 number in the pattern means this is not now needed.
1352 70. The PCRE2-specific VERSION condition didn't work correctly if only one
1353 digit was given after the decimal point, or if more than two digits were given.
1354 It now works with one or two digits, and gives a compile time error if more are
1357 71. In pcre2_substitute() there was the possibility of reading one code unit
1358 beyond the end of the replacement string.
1360 72. The code for checking a subject's UTF-32 validity for a pattern with a
1361 lookbehind involved an out-of-bounds pointer, which could potentially cause
1362 trouble in some environments.
1364 73. The maximum lookbehind length was incorrectly calculated for patterns such
1365 as /(?<=(a)(?-1))x/ which have a recursion within a backreference.
1367 74. Give an error if a lookbehind assertion is longer than 65535 code units.
1369 75. Give an error in pcre2_substitute() if a match ends before it starts (as a
1370 result of the use of \K).
1372 76. Check the length of subpattern names and the names in (*MARK:xx) etc.
1373 dynamically to avoid the possibility of integer overflow.
1375 77. Implement pcre2_set_max_pattern_length() so that programs can restrict the
1376 size of patterns that they are prepared to handle.
1378 78. (*NO_AUTO_POSSESS) was not working.
1380 79. Adding group information caching improves the speed of compiling when
1381 checking whether a group has a fixed length and/or could match an empty string,
1382 especially when recursion or subroutine calls are involved. However, this
1383 cannot be used when (?| is present in the pattern because the same number may
1384 be used for groups of different sizes. To catch runaway patterns in this
1385 situation, counts have been introduced to the functions that scan for empty
1386 branches or compute fixed lengths.
1388 80. Allow for the possibility of the size of the nest_save structure not being
1389 a factor of the size of the compiling workspace (it currently is).
1391 81. Check for integer overflow in minimum length calculation and cap it at
1394 82. Small optimizations in code for finding the minimum matching length.
1396 83. Lock out configuring for EBCDIC with non-8-bit libraries.
1398 84. Test for error code <= 0 in regerror().
1400 85. Check for too many replacements (more than INT_MAX) in pcre2_substitute().
1402 86. Avoid the possibility of computing with an out-of-bounds pointer (though
1403 not dereferencing it) while handling lookbehind assertions.
1405 87. Failure to get memory for the match data in regcomp() is now given as a
1406 regcomp() error instead of waiting for regexec() to pick it up.
1408 88. In pcre2_substitute(), ensure that CRLF is not split when it is a valid
1411 89. Paranoid check in regcomp() for bad error code from pcre2_compile().
1413 90. Run test 8 (internal offsets and code sizes) for link sizes 3 and 4 as well
1416 91. Document that JIT has a limit on pattern size, and give more information
1417 about JIT compile failures in pcre2test.
1419 92. Implement PCRE2_INFO_HASBACKSLASHC.
1421 93. Re-arrange valgrind support code in pcre2test to avoid spurious reports
1422 with JIT (possibly caused by SSE2?).
1424 94. Support offset_limit in JIT.
1426 95. A sequence such as [[:punct:]b] that is, a POSIX character class followed
1427 by a single ASCII character in a class item, was incorrectly compiled in UCP
1428 mode. The POSIX class got lost, but only if the single character followed it.
1430 96. [:punct:] in UCP mode was matching some characters in the range 128-255
1431 that should not have been matched.
1433 97. If [:^ascii:] or [:^xdigit:] are present in a non-negated class, all
1434 characters with code points greater than 255 are in the class. When a Unicode
1435 property was also in the class (if PCRE2_UCP is set, escapes such as \w are
1436 turned into Unicode properties), wide characters were not correctly handled,
1437 and could fail to match.
1439 98. In pcre2test, make the "startoffset" modifier a synonym of "offset",
1440 because it sets the "startoffset" parameter for pcre2_match().
1442 99. If PCRE2_AUTO_CALLOUT was set on a pattern that had a (?# comment between
1443 an item and its qualifier (for example, A(?#comment)?B) pcre2_compile()
1444 misbehaved. This bug was found by the LLVM fuzzer.
1446 100. The error for an invalid UTF pattern string always gave the code unit
1447 offset as zero instead of where the invalidity was found.
1449 101. Further to 97 above, negated classes such as [^[:^ascii:]\d] were also not
1450 working correctly in UCP mode.
1452 102. Similar to 99 above, if an isolated \E was present between an item and its
1453 qualifier when PCRE2_AUTO_CALLOUT was set, pcre2_compile() misbehaved. This bug
1454 was found by the LLVM fuzzer.
1456 103. The POSIX wrapper function regexec() crashed if the option REG_STARTEND
1457 was set when the pmatch argument was NULL. It now returns REG_INVARG.
1459 104. Allow for up to 32-bit numbers in the ordin() function in pcre2grep.
1461 105. An empty \Q\E sequence between an item and its qualifier caused
1462 pcre2_compile() to misbehave when auto callouts were enabled. This bug
1463 was found by the LLVM fuzzer.
1465 106. If both PCRE2_ALT_VERBNAMES and PCRE2_EXTENDED were set, and a (*MARK) or
1466 other verb "name" ended with whitespace immediately before the closing
1467 parenthesis, pcre2_compile() misbehaved. Example: /(*:abc )/, but only when
1468 both those options were set.
1470 107. In a number of places pcre2_compile() was not handling NULL characters
1471 correctly, and pcre2test with the "bincode" modifier was not always correctly
1472 displaying fields containing NULLS:
1474 (a) Within /x extended #-comments
1475 (b) Within the "name" part of (*MARK) and other *verbs
1476 (c) Within the text argument of a callout
1478 108. If a pattern that was compiled with PCRE2_EXTENDED started with white
1479 space or a #-type comment that was followed by (?-x), which turns off
1480 PCRE2_EXTENDED, and there was no subsequent (?x) to turn it on again,
1481 pcre2_compile() assumed that (?-x) applied to the whole pattern and
1482 consequently mis-compiled it. This bug was found by the LLVM fuzzer. The fix
1483 for this bug means that a setting of any of the (?imsxJU) options at the start
1484 of a pattern is no longer transferred to the options that are returned by
1485 PCRE2_INFO_ALLOPTIONS. In fact, this was an anachronism that should have
1486 changed when the effects of those options were all moved to compile time.
1488 109. An escaped closing parenthesis in the "name" part of a (*verb) when
1489 PCRE2_ALT_VERBNAMES was set caused pcre2_compile() to malfunction. This bug
1490 was found by the LLVM fuzzer.
1492 110. Implemented PCRE2_SUBSTITUTE_UNSET_EMPTY, and updated pcre2test to make it
1493 possible to test it.
1495 111. "Harden" pcre2test against ridiculously large values in modifiers and
1496 command line arguments.
1498 112. Implemented PCRE2_SUBSTITUTE_UNKNOWN_UNSET and PCRE2_SUBSTITUTE_OVERFLOW_
1501 113. Fix printing of *MARK names that contain binary zeroes in pcre2test.
1504 Version 10.20 30-June-2015
1505 --------------------------
1507 1. Callouts with string arguments have been added.
1509 2. Assertion code generator in JIT has been optimized.
1511 3. The invalid pattern (?(?C) has a missing assertion condition at the end. The
1512 pcre2_compile() function read past the end of the input before diagnosing an
1513 error. This bug was discovered by the LLVM fuzzer.
1515 4. Implemented pcre2_callout_enumerate().
1517 5. Fix JIT compilation of conditional blocks whose assertion is converted to
1518 (*FAIL). E.g: /(?(?!))/.
1520 6. The pattern /(?(?!)^)/ caused references to random memory. This bug was
1521 discovered by the LLVM fuzzer.
1523 7. The assertion (?!) is optimized to (*FAIL). This was not handled correctly
1524 when this assertion was used as a condition, for example (?(?!)a|b). In
1525 pcre2_match() it worked by luck; in pcre2_dfa_match() it gave an incorrect
1526 error about an unsupported item.
1528 8. For some types of pattern, for example /Z*(|d*){216}/, the auto-
1529 possessification code could take exponential time to complete. A recursion
1530 depth limit of 1000 has been imposed to limit the resources used by this
1531 optimization. This infelicity was discovered by the LLVM fuzzer.
1533 9. A pattern such as /(*UTF)[\S\V\H]/, which contains a negated special class
1534 such as \S in non-UCP mode, explicit wide characters (> 255) can be ignored
1535 because \S ensures they are all in the class. The code for doing this was
1536 interacting badly with the code for computing the amount of space needed to
1537 compile the pattern, leading to a buffer overflow. This bug was discovered by
1540 10. A pattern such as /((?2)+)((?1))/ which has mutual recursion nested inside
1541 other kinds of group caused stack overflow at compile time. This bug was
1542 discovered by the LLVM fuzzer.
1544 11. A pattern such as /(?1)(?#?'){8}(a)/ which had a parenthesized comment
1545 between a subroutine call and its quantifier was incorrectly compiled, leading
1546 to buffer overflow or other errors. This bug was discovered by the LLVM fuzzer.
1548 12. The illegal pattern /(?(?<E>.*!.*)?)/ was not being diagnosed as missing an
1549 assertion after (?(. The code was failing to check the character after (?(?<
1550 for the ! or = that would indicate a lookbehind assertion. This bug was
1551 discovered by the LLVM fuzzer.
1553 13. A pattern such as /X((?2)()*+){2}+/ which has a possessive quantifier with
1554 a fixed maximum following a group that contains a subroutine reference was
1555 incorrectly compiled and could trigger buffer overflow. This bug was discovered
1558 14. Negative relative recursive references such as (?-7) to non-existent
1559 subpatterns were not being diagnosed and could lead to unpredictable behaviour.
1560 This bug was discovered by the LLVM fuzzer.
1562 15. The bug fixed in 14 was due to an integer variable that was unsigned when
1563 it should have been signed. Some other "int" variables, having been checked,
1564 have either been changed to uint32_t or commented as "must be signed".
1566 16. A mutual recursion within a lookbehind assertion such as (?<=((?2))((?1)))
1567 caused a stack overflow instead of the diagnosis of a non-fixed length
1568 lookbehind assertion. This bug was discovered by the LLVM fuzzer.
1570 17. The use of \K in a positive lookbehind assertion in a non-anchored pattern
1571 (e.g. /(?<=\Ka)/) could make pcre2grep loop.
1573 18. There was a similar problem to 17 in pcre2test for global matches, though
1574 the code there did catch the loop.
1576 19. If a greedy quantified \X was preceded by \C in UTF mode (e.g. \C\X*),
1577 and a subsequent item in the pattern caused a non-match, backtracking over the
1578 repeated \X did not stop, but carried on past the start of the subject, causing
1579 reference to random memory and/or a segfault. There were also some other cases
1580 where backtracking after \C could crash. This set of bugs was discovered by the
1583 20. The function for finding the minimum length of a matching string could take
1584 a very long time if mutual recursion was present many times in a pattern, for
1585 example, /((?2){73}(?2))((?1))/. A better mutual recursion detection method has
1586 been implemented. This infelicity was discovered by the LLVM fuzzer.
1588 21. Implemented PCRE2_NEVER_BACKSLASH_C.
1590 22. The feature for string replication in pcre2test could read from freed
1591 memory if the replication required a buffer to be extended, and it was not
1592 working properly in 16-bit and 32-bit modes. This issue was discovered by a
1593 fuzzer: see http://lcamtuf.coredump.cx/afl/.
1595 23. Added the PCRE2_ALT_CIRCUMFLEX option.
1597 24. Adjust the treatment of \8 and \9 to be the same as the current Perl
1600 25. Static linking against the PCRE2 library using the pkg-config module was
1601 failing on missing pthread symbols.
1603 26. If a group that contained a recursive back reference also contained a
1604 forward reference subroutine call followed by a non-forward-reference
1605 subroutine call, for example /.((?2)(?R)\1)()/, pcre2_compile() failed to
1606 compile correct code, leading to undefined behaviour or an internally detected
1607 error. This bug was discovered by the LLVM fuzzer.
1609 27. Quantification of certain items (e.g. atomic back references) could cause
1610 incorrect code to be compiled when recursive forward references were involved.
1611 For example, in this pattern: /(?1)()((((((\1++))\x85)+)|))/. This bug was
1612 discovered by the LLVM fuzzer.
1614 28. A repeated conditional group whose condition was a reference by name caused
1615 a buffer overflow if there was more than one group with the given name. This
1616 bug was discovered by the LLVM fuzzer.
1618 29. A recursive back reference by name within a group that had the same name as
1619 another group caused a buffer overflow. For example: /(?J)(?'d'(?'d'\g{d}))/.
1620 This bug was discovered by the LLVM fuzzer.
1622 30. A forward reference by name to a group whose number is the same as the
1623 current group, for example in this pattern: /(?|(\k'Pm')|(?'Pm'))/, caused a
1624 buffer overflow at compile time. This bug was discovered by the LLVM fuzzer.
1626 31. Fix -fsanitize=undefined warnings for left shifts of 1 by 31 (it treats 1
1627 as an int; fixed by writing it as 1u).
1629 32. Fix pcre2grep compile when -std=c99 is used with gcc, though it still gives
1630 a warning for "fileno" unless -std=gnu99 us used.
1632 33. A lookbehind assertion within a set of mutually recursive subpatterns could
1633 provoke a buffer overflow. This bug was discovered by the LLVM fuzzer.
1635 34. Give an error for an empty subpattern name such as (?'').
1637 35. Make pcre2test give an error if a pattern that follows #forbud_utf contains
1640 36. The way named subpatterns are handled has been refactored. There is now a
1641 pre-pass over the regex which does nothing other than identify named
1642 subpatterns and count the total captures. This means that information about
1643 named patterns is known before the rest of the compile. In particular, it means
1644 that forward references can be checked as they are encountered. Previously, the
1645 code for handling forward references was contorted and led to several errors in
1646 computing the memory requirements for some patterns, leading to buffer
1649 37. There was no check for integer overflow in subroutine calls such as (?123).
1651 38. The table entry for \l in EBCDIC environments was incorrect, leading to its
1652 being treated as a literal 'l' instead of causing an error.
1654 39. If a non-capturing group containing a conditional group that could match
1655 an empty string was repeated, it was not identified as matching an empty string
1656 itself. For example: /^(?:(?(1)x|)+)+$()/.
1658 40. In an EBCDIC environment, pcretest was mishandling the escape sequences
1659 \a and \e in test subject lines.
1661 41. In an EBCDIC environment, \a in a pattern was converted to the ASCII
1662 instead of the EBCDIC value.
1664 42. The handling of \c in an EBCDIC environment has been revised so that it is
1665 now compatible with the specification in Perl's perlebcdic page.
1667 43. Single character repetition in JIT has been improved. 20-30% speedup
1668 was achieved on certain patterns.
1670 44. The EBCDIC character 0x41 is a non-breaking space, equivalent to 0xa0 in
1671 ASCII/Unicode. This has now been added to the list of characters that are
1672 recognized as white space in EBCDIC.
1674 45. When PCRE2 was compiled without Unicode support, the use of \p and \P gave
1675 an error (correctly) when used outside a class, but did not give an error
1678 46. \h within a class was incorrectly compiled in EBCDIC environments.
1680 47. JIT should return with error when the compiled pattern requires
1681 more stack space than the maximum.
1683 48. Fixed a memory leak in pcre2grep when a locale is set.
1686 Version 10.10 06-March-2015
1687 ---------------------------
1689 1. When a pattern is compiled, it remembers the highest back reference so that
1690 when matching, if the ovector is too small, extra memory can be obtained to
1691 use instead. A conditional subpattern whose condition is a check on a capture
1692 having happened, such as, for example in the pattern /^(?:(a)|b)(?(1)A|B)/, is
1693 another kind of back reference, but it was not setting the highest
1694 backreference number. This mattered only if pcre2_match() was called with an
1695 ovector that was too small to hold the capture, and there was no other kind of
1696 back reference (a situation which is probably quite rare). The effect of the
1697 bug was that the condition was always treated as FALSE when the capture could
1698 not be consulted, leading to a incorrect behaviour by pcre2_match(). This bug
1701 2. Functions for serialization and deserialization of sets of compiled patterns
1704 3. The value that is returned by PCRE2_INFO_SIZE has been corrected to remove
1705 excess code units at the end of the data block that may occasionally occur if
1706 the code for calculating the size over-estimates. This change stops the
1707 serialization code copying uninitialized data, to which valgrind objects. The
1708 documentation of PCRE2_INFO_SIZE was incorrect in stating that the size did not
1709 include the general overhead. This has been corrected.
1711 4. All code units in every slot in the table of group names are now set, again
1712 in order to avoid accessing uninitialized data when serializing.
1714 5. The (*NO_JIT) feature is implemented.
1716 6. If a bug that caused pcre2_compile() to use more memory than allocated was
1717 triggered when using valgrind, the code in (3) above passed a stupidly large
1718 value to valgrind. This caused a crash instead of an "internal error" return.
1720 7. A reference to a duplicated named group (either a back reference or a test
1721 for being set in a conditional) that occurred in a part of the pattern where
1722 PCRE2_DUPNAMES was not set caused the amount of memory needed for the pattern
1723 to be incorrectly calculated, leading to overwriting.
1725 8. A mutually recursive set of back references such as (\2)(\1) caused a
1726 segfault at compile time (while trying to find the minimum matching length).
1727 The infinite loop is now broken (with the minimum length unset, that is, zero).
1729 9. If an assertion that was used as a condition was quantified with a minimum
1730 of zero, matching went wrong. In particular, if the whole group had unlimited
1731 repetition and could match an empty string, a segfault was likely. The pattern
1732 (?(?=0)?)+ is an example that caused this. Perl allows assertions to be
1733 quantified, but not if they are being used as conditions, so the above pattern
1734 is faulted by Perl. PCRE2 has now been changed so that it also rejects such
1737 10. The error message for an invalid quantifier has been changed from "nothing
1738 to repeat" to "quantifier does not follow a repeatable item".
1740 11. If a bad UTF string is compiled with NO_UTF_CHECK, it may succeed, but
1741 scanning the compiled pattern in subsequent auto-possessification can get out
1742 of step and lead to an unknown opcode. Previously this could have caused an
1743 infinite loop. Now it generates an "internal error" error. This is a tidyup,
1744 not a bug fix; passing bad UTF with NO_UTF_CHECK is documented as having an
1747 12. A UTF pattern containing a "not" match of a non-ASCII character and a
1748 subroutine reference could loop at compile time. Example: /[^\xff]((?1))/.
1750 13. The locale test (RunTest 3) has been upgraded. It now checks that a locale
1751 that is found in the output of "locale -a" can actually be set by pcre2test
1752 before it is accepted. Previously, in an environment where a locale was listed
1753 but would not set (an example does exist), the test would "pass" without
1754 actually doing anything. Also the fr_CA locale has been added to the list of
1755 locales that can be used.
1757 14. Fixed a bug in pcre2_substitute(). If a replacement string ended in a
1758 capturing group number without parentheses, the last character was incorrectly
1759 literally included at the end of the replacement string.
1761 15. A possessive capturing group such as (a)*+ with a minimum repeat of zero
1762 failed to allow the zero-repeat case if pcre2_match() was called with an
1763 ovector too small to capture the group.
1765 16. Improved error message in pcre2test when setting the stack size (-S) fails.
1767 17. Fixed two bugs in CMakeLists.txt: (1) Some lines had got lost in the
1768 transfer from PCRE1, meaning that CMake configuration failed if "build tests"
1769 was selected. (2) The file src/pcre2_serialize.c had not been added to the list
1770 of PCRE2 sources, which caused a failure to build pcre2test.
1772 18. Fixed typo in pcre2_serialize.c (DECL instead of DEFN) that causes problems
1775 19. Use binary input when reading back saved serialized patterns in pcre2test.
1777 20. Added RunTest.bat for running the tests under Windows.
1779 21. "make distclean" was not removing config.h, a file that may be created for
1782 22. A pattern such as "((?2){0,1999}())?", which has a group containing a
1783 forward reference repeated a large (but limited) number of times within a
1784 repeated outer group that has a zero minimum quantifier, caused incorrect code
1785 to be compiled, leading to the error "internal error: previously-checked
1786 referenced subpattern not found" when an incorrect memory address was read.
1787 This bug was reported as "heap overflow", discovered by Kai Lu of Fortinet's
1788 FortiGuard Labs. (Added 24-March-2015: CVE-2015-2325 was given to this.)
1790 23. A pattern such as "((?+1)(\1))/" containing a forward reference subroutine
1791 call within a group that also contained a recursive back reference caused
1792 incorrect code to be compiled. This bug was reported as "heap overflow",
1793 discovered by Kai Lu of Fortinet's FortiGuard Labs. (Added 24-March-2015:
1794 CVE-2015-2326 was given to this.)
1796 24. Computing the size of the JIT read-only data in advance has been a source
1797 of various issues, and new ones are still appear unfortunately. To fix
1798 existing and future issues, size computation is eliminated from the code,
1799 and replaced by on-demand memory allocation.
1801 25. A pattern such as /(?i)[A-`]/, where characters in the other case are
1802 adjacent to the end of the range, and the range contained characters with more
1803 than one other case, caused incorrect behaviour when compiled in UTF mode. In
1804 that example, the range a-j was left out of the class.
1807 Version 10.00 05-January-2015
1808 -----------------------------
1810 Version 10.00 is the first release of PCRE2, a revised API for the PCRE
1811 library. Changes prior to 10.00 are logged in the ChangeLog file for the old
1812 API, up to item 20 for release 8.36.
1814 The code of the library was heavily revised as part of the new API
1815 implementation. Details of each and every modification were not individually
1816 logged. In addition to the API changes, the following changes were made. They
1817 are either new functionality, or bug fixes and other noticeable changes of
1818 behaviour that were implemented after the code had been forked.
1820 1. Including Unicode support at build time is now enabled by default, but it
1821 can optionally be disabled. It is not enabled by default at run time (no
1824 2. The test program, now called pcre2test, was re-specified and almost
1825 completely re-written. Its input is not compatible with input for pcretest.
1827 3. Patterns may start with (*NOTEMPTY) or (*NOTEMPTY_ATSTART) to set the
1828 PCRE2_NOTEMPTY or PCRE2_NOTEMPTY_ATSTART options for every subject line that is
1829 matched by that pattern.
1831 4. For the benefit of those who use PCRE2 via some other application, that is,
1832 not writing the function calls themselves, it is possible to check the PCRE2
1833 version by matching a pattern such as /(?(VERSION>=10)yes|no)/ against a
1834 string such as "yesno".
1836 5. There are case-equivalent Unicode characters whose encodings use different
1837 numbers of code units in UTF-8. U+023A and U+2C65 are one example. (It is
1838 theoretically possible for this to happen in UTF-16 too.) If a backreference to
1839 a group containing one of these characters was greedily repeated, and during
1840 the match a backtrack occurred, the subject might be backtracked by the wrong
1841 number of code units. For example, if /^(\x{23a})\1*(.)/ is matched caselessly
1842 (and in UTF-8 mode) against "\x{23a}\x{2c65}\x{2c65}\x{2c65}", group 2 should
1843 capture the final character, which is the three bytes E2, B1, and A5 in UTF-8.
1844 Incorrect backtracking meant that group 2 captured only the last two bytes.
1845 This bug has been fixed; the new code is slower, but it is used only when the
1846 strings matched by the repetition are not all the same length.
1848 6. A pattern such as /()a/ was not setting the "first character must be 'a'"
1849 information. This applied to any pattern with a group that matched no
1850 characters, for example: /(?:(?=.)|(?<!x))a/.
1852 7. When an (*ACCEPT) is triggered inside capturing parentheses, it arranges for
1853 those parentheses to be closed with whatever has been captured so far. However,
1854 it was failing to mark any other groups between the highest capture so far and
1855 the currrent group as "unset". Thus, the ovector for those groups contained
1856 whatever was previously there. An example is the pattern /(x)|((*ACCEPT))/ when
1857 matched against "abcd".
1859 8. The pcre2_substitute() function has been implemented.
1861 9. If an assertion used as a condition was quantified with a minimum of zero
1862 (an odd thing to do, but it happened), SIGSEGV or other misbehaviour could
1865 10. The PCRE2_NO_DOTSTAR_ANCHOR option has been implemented.