doc/modsecurity2-data-formats.xml

   1 <?xml version="1.0" encoding="UTF-8"?>
   2 <!DOCTYPE article PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN"
   3                          "http://www.docbook.org/xml/4.4/docbookx.dtd">
   4 <article>
   5     <title>ModSecurity 2 Data Formats</title>
   6     <articleinfo>
   7         <releaseinfo>Version 2.5.10-dev1 (March 24, 2009)</releaseinfo>
   8         <copyright>
   9             <year>2004-2009</year>
  10             <holder>Breach Security, Inc. (<ulink url="http://www.breach.com"
  11                     >http://www.breach.com</ulink>)</holder>
  12         </copyright>
  13     </articleinfo>
  14     <para>The purpose of this document is to describe the formats of the ModSecurity alert messages,
  15         transaction logs and communication protocols, which would not only allow for a better
  16         understanding what ModSecurity does but also for an easy integration with third-party tools
  17         and products.</para>
  18     <section>
  19         <title>Alerts</title>
  20         <para>As part of its operations ModSecurity will emit alerts, which are either
  21                 <emphasis>warnings</emphasis> (non-fatal) or <emphasis>errors</emphasis> (fatal,
  22             usually leading to the interception of the transaction in question). Below is an example
  23             of a ModSecurity alert entry:</para>
  24         <programlisting>Access denied with code 505 (phase 1). Match of "rx
  25   ^HTTP/(0\\\\.9|1\\\\.[01])$" against "REQUEST_PROTOCOL" required.
  26   [id "960034"] [msg "HTTP protocol version is not allowed by policy"]
  27   [severity "CRITICAL"] [uri "/"] [unique_id "PQaTTVBEUOkAAFwKXrYAAAAM"]</programlisting>
  28         <note>
  29             <para>Alerts will only ever contain one line of text but we've broken the above example
  30                 into multiple lines to make it fit into the page.</para>
  31         </note>
  32         <para>Each alert entry begins with the engine message, which describes what ModSecurity did
  33             and why. For
  34             example:<programlisting>Access denied with code 505 (phase 1). Match of "rx
  35   ^HTTP/(0\\\\.9|1\\\\.[01])$" against "REQUEST_PROTOCOL" required.</programlisting></para>
  36         <section>
  37             <title>Alert Action Description</title>
  38             <para>The first part of the engine message tells you whether ModSecurity acted to
  39                 interrupt transaction or rule processing:</para>
  40             <orderedlist>
  41                 <listitem>
  42                     <para>If the alert is only a warning, the first sentence will simply say
  43                             <emphasis>Warning</emphasis>.</para>
  44                 </listitem>
  45                 <listitem>
  46                     <para>If the transaction was intercepted, the first sentence will begin with
  47                             <emphasis>Access denied</emphasis>. What follows is the list of possible
  48                         messages related to transaction interception:</para>
  49                     <itemizedlist>
  50                         <listitem>
  51                             <para><emphasis>Access denied with code %0</emphasis> - a response with
  52                                 status code <literal>%0</literal> was sent.</para>
  53                         </listitem>
  54                         <listitem>
  55                             <para><emphasis>Access denied with connection close</emphasis> -
  56                                 connection was abruptly closed.</para>
  57                         </listitem>
  58                         <listitem>
  59                             <para><emphasis>Access denied with redirection to %0 using status
  60                                     %1</emphasis> - a redirection to URI <literal>%0</literal> was
  61                                 issued using status <literal>%1</literal>.</para>
  62                         </listitem>
  63                     </itemizedlist>
  64                 </listitem>
  65                 <listitem>
  66                     <para>There is also a special message that ModSecurity emits where an <literal
  67                             >allow</literal> action is executed. There are three variations of this
  68                         type of message:</para>
  69                     <itemizedlist>
  70                         <listitem>
  71                             <para><emphasis>Access allowed</emphasis> - rule engine stopped
  72                                 processing rules (transaction was unaffected).</para>
  73                         </listitem>
  74                         <listitem>
  75                             <para><emphasis>Access to phase allowed</emphasis> - rule engine stopped
  76                                 processing rules in the current phase only. Subsequent phases will
  77                                 be processed normally. Transaction was not affected by this rule but
  78                                 it may be affected by any of the rules in the subsequent
  79                                 phase.</para>
  80                         </listitem>
  81                         <listitem>
  82                             <para><emphasis>Access to request allowed</emphasis> - rule engine
  83                                 stopped processing rules in the current phase. Phases prior to
  84                                 request execution in the backend (currently phases 1 and 2) will not
  85                                 be processed. The response phases (currently phases 3 and 4) and
  86                                 others (currently phase 5) will be processed as normal. Transaction
  87                                 was not affected by this rule but it may be affected by any of the
  88                                 rules in the subsequent phase.</para>
  89                         </listitem>
  90                     </itemizedlist>
  91                 </listitem>
  92             </orderedlist>
  93         </section>
  94         <section>
  95             <title>Alert Justification Description</title>
  96             <para>The second part of the engine message explains <emphasis>why</emphasis> the alert
  97                 was generated. Since it is automatically generated from the rules it will be very
  98                 technical in nature, talking about operators and their parameters and give you
  99                 insight into what the rule looked like. But this message cannot give you insight
 100                 into the reasoning behind the rule. A well-written rule will always specify a
 101                 human-readable message (using the <literal>msg</literal> action) to provide further
 102                 information.</para>
 103             <para>The format of the second part of the engine message depends on whether it was
 104                 generated by the operator (which happens on a match) or by the rule processor (which
 105                 happens where there is not a match, but the negation was used):</para>
 106             <itemizedlist>
 107                 <listitem>
 108                     <para><literal>@beginsWith</literal> - <emphasis>String match %0 at
 109                             %1.</emphasis></para>
 110                 </listitem>
 111                 <listitem>
 112                     <para><literal>@contains</literal> - <emphasis>String match %0 at
 113                         %1.</emphasis></para>
 114                 </listitem>
 115                 <listitem>
 116                     <para><literal>@containsWord</literal> - <emphasis>String match %0 at
 117                             %1.</emphasis></para>
 118                 </listitem>
 119                 <listitem>
 120                     <para><literal>@endsWith</literal> - <emphasis>String match %0 at
 121                         %1.</emphasis></para>
 122                 </listitem>
 123                 <listitem>
 124                     <para><literal>@eq</literal> - <emphasis>Operator EQ matched %0 at
 125                             %1.</emphasis></para>
 126                 </listitem>
 127                 <listitem>
 128                     <para><literal>@ge</literal> - <emphasis>Operator GE matched %0 at
 129                             %1.</emphasis></para>
 130                 </listitem>
 131                 <listitem>
 132                     <para><literal>@geoLookup</literal> - <emphasis>Geo lookup for %0 succeeded at
 133                             %1.</emphasis></para>
 134                 </listitem>
 135                 <listitem>
 136                     <para><literal>@inspectFile</literal> - <emphasis>File %0 rejected by the
 137                             approver script %1: %2</emphasis></para>
 138                 </listitem>
 139                 <listitem>
 140                     <para><literal>@le</literal> - <emphasis>Operator LE matched %0 at
 141                             %1.</emphasis></para>
 142                 </listitem>
 143                 <listitem>
 144                     <para><literal>@lt</literal> - <emphasis>Operator LT matched %0 at
 145                             %1.</emphasis></para>
 146                 </listitem>
 147                 <listitem>
 148                     <para><literal>@rbl</literal> - <emphasis>RBL lookup of %0 succeeded at
 149                             %1.</emphasis></para>
 150                 </listitem>
 151                 <listitem>
 152                     <para><literal>@rx</literal> - <emphasis>Pattern match %0 at
 153                         %1.</emphasis></para>
 154                 </listitem>
 155                 <listitem>
 156                     <para><literal>@streq</literal> - <emphasis>String match %0 at
 157                         %1.</emphasis></para>
 158                 </listitem>
 159                 <listitem>
 160                     <para><literal>@validateByteRange</literal> - <emphasis>Found %0 byte(s) in %1
 161                             outside range: %2.</emphasis></para>
 162                 </listitem>
 163                 <listitem>
 164                     <para><literal>@validateDTD</literal> - <emphasis>XML: DTD validation
 165                             failed.</emphasis></para>
 166                 </listitem>
 167                 <listitem>
 168                     <para><literal>@validateSchema</literal> - <emphasis>XML: Schema validation
 169                             failed.</emphasis></para>
 170                 </listitem>
 171                 <listitem>
 172                     <para><literal>@validateUrlEncoding</literal></para>
 173                     <itemizedlist>
 174                         <listitem>
 175                             <para><emphasis>Invalid URL Encoding: Non-hexadecimal digits used at
 176                                     %0.</emphasis></para>
 177                         </listitem>
 178                         <listitem>
 179                             <para><emphasis>Invalid URL Encoding: Not enough characters at the end
 180                                     of input at %0.</emphasis></para>
 181                         </listitem>
 182                     </itemizedlist>
 183                 </listitem>
 184                 <listitem>
 185                     <para><literal>@validateUtf8Encoding</literal></para>
 186                     <itemizedlist>
 187                         <listitem>
 188                             <para><emphasis>Invalid UTF-8 encoding: not enough bytes in character at
 189                                     %0.</emphasis></para>
 190                         </listitem>
 191                         <listitem>
 192                             <para><emphasis>Invalid UTF-8 encoding: invalid byte value in character
 193                                     at %0.</emphasis></para>
 194                         </listitem>
 195                         <listitem>
 196                             <para><emphasis>Invalid UTF-8 encoding: overlong character detected at
 197                                     %0.</emphasis></para>
 198                         </listitem>
 199                         <listitem>
 200                             <para><emphasis>Invalid UTF-8 encoding: use of restricted character at
 201                                     %0.</emphasis></para>
 202                         </listitem>
 203                         <listitem>
 204                             <para><emphasis>Invalid UTF-8 encoding: decoding error at
 205                                 %0.</emphasis></para>
 206                         </listitem>
 207                     </itemizedlist>
 208                 </listitem>
 209                 <listitem>
 210                     <para><literal>@verifyCC</literal> - <emphasis>CC# match %0 at
 211                         %1.</emphasis></para>
 212                 </listitem>
 213             </itemizedlist>
 214             <para>Messages not related to operators:</para>
 215             <itemizedlist>
 216                 <listitem>
 217                     <para>When <literal>SecAction</literal> directive is processed -
 218                             <emphasis>Unconditional match in SecAction.</emphasis></para>
 219                 </listitem>
 220                 <listitem>
 221                     <para>When <literal>SecRule</literal> does not match but negation is used -
 222                             <emphasis>Match of %0 against %1 required.</emphasis></para>
 223                 </listitem>
 224             </itemizedlist>
 225             <note>
 226                 <para>The parameters to the operators <literal>@rx</literal> and <literal
 227                         >@pm</literal> (regular expression and text pattern, respectively) will be
 228                     truncated to 252 bytes if they are longer than this limit. In this case the
 229                     parameter in the alert message will be terminated with three dots.</para>
 230             </note>
 231         </section>
 232         <section>
 233             <title>Meta-data</title>
 234             <para>The metadata fields are always placed at the end of the alert entry. Each metadata
 235                 field is a text fragment that consists of an open bracket followed by the metadata
 236                 field name, followed by the value and the closing bracket. What follows is the text
 237                 fragment that makes up the <literal>id</literal> metadata field.</para>
 238             <programlisting>[id "960034"]</programlisting>
 239             <para>The following metadata fields are currently used:</para>
 240             <orderedlist>
 241                 <listitem>
 242                     <para><literal>offset</literal> - The byte offset where a match occured within
 243                         the target data. This is not always available.</para>
 244                 </listitem>
 245                 <listitem>
 246                     <para><literal>id</literal> - Unique rule ID, as specified by the <literal
 247                             >id</literal> action.</para>
 248                 </listitem>
 249                 <listitem>
 250                     <para><literal>rev</literal> - Rule revision, as specified by the <literal
 251                             >rev</literal> action.</para>
 252                 </listitem>
 253                 <listitem>
 254                     <para><literal>msg</literal> - Human-readable message, as specified by the
 255                             <literal>msg</literal> action.</para>
 256                 </listitem>
 257                 <listitem>
 258                     <para><literal>severity</literal> - Event severity as text, as specified by the
 259                             <literal>severity</literal> action. The possible values (with their
 260                         corresponding numberical values in brackets) are <literal
 261                             >EMERGENCY</literal> (0), <literal>ALERT</literal> (1), <literal
 262                             >CRITICAL</literal> (2), <literal>ERROR</literal> (3), <literal
 263                             >WARNING</literal> (4), <literal>NOTICE</literal> (5), <literal
 264                             >INFO</literal> (6) and <literal>DEBUG</literal> (7).</para>
 265                 </listitem>
 266                 <listitem>
 267                     <para><literal>unique_id</literal> - Unique event ID, generated
 268                         automatically.</para>
 269                 </listitem>
 270                 <listitem>
 271                     <para><literal>uri</literal> - Request URI.</para>
 272                 </listitem>
 273                 <listitem>
 274                     <para><literal>logdata</literal> - contains transaction data fragment, as
 275                         specified by the <literal>logdata</literal> action.</para>
 276                 </listitem>
 277             </orderedlist>
 278         </section>
 279         <section>
 280             <title>Escaping</title>
 281             <para>ModSecurity alerts will always contain text fragments that were taken from
 282                 configuration or the transaction. Such text fragments escaped before they are user
 283                 in messages, in order to sanitise the potentially dangerous characters. They are
 284                 also sometimes surrounded using double quotes. The escaping algorithm is as
 285                     follows:<orderedlist>
 286                     <listitem>
 287                         <para>Characters <literal>0x08</literal> (<literal>BACKSPACE</literal>),
 288                                 <literal>0x0a</literal> (<literal>NEWLINE</literal>), <literal
 289                                 >0x10</literal> (<literal>CARRIAGE RETURN</literal>), <literal
 290                                 >0x09</literal> (<literal>HORIZONTAL TAB</literal>) and <literal
 291                                 >0x0b</literal> (<literal>VERTICAL TAB</literal>) will be
 292                             represented as <literal>\b</literal>, <literal>\n</literal>, <literal
 293                                 >\r</literal>, <literal>\t</literal> and <literal>\v</literal>,
 294                             respectively.</para>
 295                     </listitem>
 296                     <listitem>
 297                         <para>Bytes from the ranges <literal>0-0x1f</literal> and <literal
 298                                 >0x7f-0xff</literal> (inclusive) will be represented as <literal
 299                                 >\xHH</literal>, where <literal>HH</literal> is the hexadecimal
 300                             value of the byte.</para>
 301                     </listitem>
 302                     <listitem>
 303                         <para>Backslash characters (<literal>\</literal>) will be represented as
 304                                 <literal>\\</literal>.</para>
 305                     </listitem>
 306                     <listitem>
 307                         <para>Each double quote character will be represented as <literal
 308                                 >\"</literal>, but only if the entire fragment is surrounded with
 309                             double quotes.</para>
 310                     </listitem>
 311                 </orderedlist></para>
 312         </section>
 313         <section>
 314             <title>Alerts in the Apache Error Log</title>
 315             <para>Every ModSecurity alert conforms to the following format when it appears in the
 316                 Apache error log:</para>
 317             <programlisting>[Sun Jun 24 10:19:58 2007] [error] [client 192.168.0.1]
 318             ModSecurity: ALERT_MESSAGE</programlisting>
 319             <para>The above is a standard Apache error log format. The <literal>ModSecurity:
 320                 </literal> prefix is specific to ModSecurity. It is used to allow quick
 321                 identification of ModSecurity alert messages when they appear in the same file next
 322                 to other Apache messages.</para>
 323             <para>The actual message (<literal>ALERT_MESSAGE</literal> in the example above) is in
 324                 the same format as described in the <emphasis>Alerts</emphasis> section.</para>
 325             <note>
 326                 <para>Apache further escapes ModSecurity alert messages before writing them to the
 327                     error log. This means that all backslash characters will be doubled in the error
 328                     log. In practice, since ModSecurity will already represent a single backslash
 329                     within an untrusted text fragment as two backslashes, the end result in the
 330                     Apache error log will be <emphasis>four</emphasis> backslashes. Thus, if you
 331                     need to interpret a ModSecurity message from the error log, you should decode
 332                     the message part after the <literal>ModSecurity:</literal> prefix first. This
 333                     step will peel the first encoding layer.</para>
 334             </note>
 335         </section>
 336         <section>
 337             <title>Alerts in Audit Logs</title>
 338             <para>Alerts are transported in the <literal>H</literal> section of the ModSecurity
 339                 Audit Log. Alerts will appear each on a separate line and in the order they were
 340                 generated by ModSecurity. Each line will be in the following format:</para>
 341             <programlisting>Message: ALERT_MESSAGE</programlisting>
 342             <para>Below is an example of an <literal>H</literal> section that contains two alert
 343                 messages:</para>
 344             <programlisting>--c7036611-H--
 345 Message: Warning. Match of "rx ^apache.*perl" against
 346   "REQUEST_HEADERS:User-Agent" required. [id "990011"] [msg "Request
 347   Indicates an automated program explored the site"] [severity "NOTICE"]
 348 Message: Warning. Pattern match "(?:\\b(?:(?:s(?:elect\\b(?:.{1,100}?\\b
 349   (?:(?:length|count|top)\\b.{1,100}?\\bfrom|from\\b.{1,100}?\\bwhere)
 350   |.*?\\b(?:d(?:ump\\b.*\\bfrom|ata_type)|(?:to_(?:numbe|cha)|inst)r))|p_
 351   (?:(?:addextendedpro|sqlexe)c|(?:oacreat|prepar)e|execute(?:sql)?|
 352   makewebt ..." at ARGS:c. [id "950001"] [msg "SQL Injection Attack.
 353   Matched signature: union select"] [severity "CRITICAL"]
 354 Stopwatch: 1199881676978327 2514 (396 2224 -)
 355 Producer: ModSecurity v2.x.x (Apache 2.x)
 356 Server: Apache/2.x.x
 357
 358 --c7036611-Z--</programlisting>
 359         </section>
 360     </section>
 361     <section>
 362         <title>Audit Log</title>
 363         <para>ModSecurity records one transaction in a single audit log file. Below is an
 364             example:</para>
 365         <programlisting>--c7036611-A--
 366 [09/Jan/2008:12:27:56 +0000] OSD4l1BEUOkAAHZ8Y3QAAAAH 209.90.77.54 64995
 367   80.68.80.233 80
 368 --c7036611-B--
 369 GET //EvilBoard_0.1a/index.php?c='/**/union/**/select/**/1,concat(username,
 370   char(77),password,char(77),email_address,char(77),info,char(77),user_level,
 371   char(77))/**/from/**/eb_members/**/where/**/userid=1/*http://kamloopstutor.
 372   com/images/banners/on.txt? HTTP/1.1
 373 TE: deflate,gzip;q=0.3
 374 Connection: TE, cslose
 375 Host: www.example.com
 376 User-Agent: libwww-perl/5.808
 377
 378 --c7036611-F--
 379 HTTP/1.1 404 Not Found
 380 Content-Length: 223
 381 Connection: close
 382 Content-Type: text/html; charset=iso-8859-1
 383
 384 --c7036611-H--
 385 Message: Warning. Match of "rx ^apache.*perl" against
 386   "REQUEST_HEADERS:User-Agent" required. [id "990011"] [msg "Request
 387   Indicates an automated program explored the site"] [severity "NOTICE"]
 388 Message: Warning. Pattern match "(?:\\b(?:(?:s(?:elect\\b(?:.{1,100}?\\b
 389   (?:(?:length|count|top)\\b.{1,100}?\\bfrom|from\\b.{1,100}?\\bwhere)
 390   |.*?\\b(?:d(?:ump\\b.*\\bfrom|ata_type)|(?:to_(?:numbe|cha)|inst)r))|p_
 391   (?:(?:addextendedpro|sqlexe)c|(?:oacreat|prepar)e|execute(?:sql)?|
 392   makewebt ..." at ARGS:c. [id "950001"] [msg "SQL Injection Attack.
 393   Matched signature: union select"] [severity "CRITICAL"]
 394 Stopwatch: 1199881676978327 2514 (396 2224 -)
 395 Producer: ModSecurity v2.x.x (Apache 2.x)
 396 Server: Apache/2.x.x
 397
 398 --c7036611-Z--
 399 </programlisting>
 400         <para>The file consist of multiple sections, each in different format. Separators are used
 401             to define sections:</para>
 402         <programlisting>--c7036611-A--</programlisting>
 403         <para>A separator always begins on a new line and conforms to the following format:</para>
 404         <orderedlist>
 405             <listitem>
 406                 <para>Two dashes</para>
 407             </listitem>
 408             <listitem>
 409                 <para>Unique boundary, which consists from several hexadecimal characters.</para>
 410             </listitem>
 411             <listitem>
 412                 <para>One dash character.</para>
 413             </listitem>
 414             <listitem>
 415                 <para>Section identifier, currently a single uppercase letter.</para>
 416             </listitem>
 417             <listitem>
 418                 <para>Two trailing dashes.</para>
 419             </listitem>
 420         </orderedlist>
 421         <para>Refer to the documentation for <literal>SecAuditLogParts</literal> for the explanation
 422             of each part.</para>
 423         <section>
 424             <title>Parts</title>
 425             <para>This section documents the audit log parts available in ModSecurity 2.x. They are: <itemizedlist>
 426                     <listitem>
 427                         <para><literal moreinfo="none">A</literal> - audit log header</para>
 428                     </listitem>
 429                     <listitem>
 430                         <para><literal moreinfo="none">B</literal> - request headers</para>
 431                     </listitem>
 432                     <listitem>
 433                         <para><literal moreinfo="none">C</literal> - request body</para>
 434                     </listitem>
 435                     <listitem>
 436                         <para><literal moreinfo="none">D</literal> - intended response headers (NOT
 437                             IMPLEMENTED)</para>
 438                     </listitem>
 439                     <listitem>
 440                         <para><literal moreinfo="none">E</literal> - intended response body</para>
 441                     </listitem>
 442                     <listitem>
 443                         <para><literal moreinfo="none">F</literal> - response headers</para>
 444                     </listitem>
 445                     <listitem>
 446                         <para><literal moreinfo="none">G</literal> - response body (NOT
 447                             IMPLEMENTED)</para>
 448                     </listitem>
 449                     <listitem>
 450                         <para><literal moreinfo="none">H</literal> - audit log trailer</para>
 451                     </listitem>
 452                     <listitem>
 453                         <para><literal moreinfo="none">I</literal> - reduced multipart request
 454                             body</para>
 455                     </listitem>
 456                     <listitem>
 457                         <para><literal moreinfo="none">J</literal> - multipart files information
 458                             (NOT IMPLEMENTED)</para>
 459                     </listitem>
 460                     <listitem>
 461                         <para><literal moreinfo="none">K</literal> - matched rules
 462                             information</para>
 463                     </listitem>
 464                     <listitem>
 465                         <para><literal moreinfo="none">Z</literal> - audit log footer</para>
 466                     </listitem>
 467                 </itemizedlist></para>
 468             <section>
 469                 <title>Audit Log Header (<literal>A</literal>)</title>
 470                 <para>ModSecurity 2.x audit log entries always begin with the header part. For
 471                     example:</para>
 472                 <programlisting>--c7036611-A--
 473 [09/Jan/2008:12:27:56 +0000] OSD4l1BEUOkAAHZ8Y3QAAAAH 209.90.77.54 64995
 474   80.68.80.233 80</programlisting>
 475                 <para>The header contains only one line, with the following information on
 476                     it:</para>
 477                 <orderedlist>
 478                     <listitem>
 479                         <para>Timestamp</para>
 480                     </listitem>
 481                     <listitem>
 482                         <para>Unique transaction ID</para>
 483                     </listitem>
 484                     <listitem>
 485                         <para>Source IP address (IPv4 or IPv6)</para>
 486                     </listitem>
 487                     <listitem>
 488                         <para>Source port</para>
 489                     </listitem>
 490                     <listitem>
 491                         <para>Destination IP address (IPv4 or IPv6)</para>
 492                     </listitem>
 493                     <listitem>
 494                         <para>Destination port</para>
 495                     </listitem>
 496                 </orderedlist>
 497             </section>
 498             <section>
 499                 <title>Request Headers (<literal>B</literal>)</title>
 500                 <para>The request headers part contains the request line and the request headers.
 501                     The information present in this part will not be identical to that sent by the
 502                     client responsible for the transaction. ModSecurity 2.x for Apache does not have
 503                     access to the raw data; it sees what Apache itself sees. While the end result
 504                     may be identical to the raw request, differences are possible in some
 505                     areas:</para>
 506                 <orderedlist>
 507                     <listitem>
 508                         <para>If any of the fields are <literal>NUL</literal>-terminated, Apache
 509                             will only see the content prior to the NUL.</para>
 510                     </listitem>
 511                     <listitem>
 512                         <para>Headers that span multiple lines (feature known as header folding)
 513                             will be collapsed into a single line.</para>
 514                     </listitem>
 515                     <listitem>
 516                         <para>Multiple headers with the same name will be combined into a single
 517                             header (as allowed by the HTTP RFC).</para>
 518                     </listitem>
 519                 </orderedlist>
 520             </section>
 521             <section>
 522                 <title>Request Body (<literal>C</literal>)</title>
 523                 <para>This part contains the request body of the transaction, after dechunking and
 524                     decompression (if applicable).</para>
 525             </section>
 526             <section>
 527                 <title>Intended Response Headers (<literal>D</literal>)</title>
 528                 <para>This part contains the status line and the request headers that would have
 529                     been delivered to the client had ModSecurity not intervened. Thus this part
 530                     makes sense only for transactions where ModSecurity altered the data flow. By
 531                     differentiating before the intended and the final response headers, we are able
 532                     to record what was internally ready for sending, but also what was actually
 533                     sent.</para>
 534                 <note>
 535                     <para>This part is reserved for future use. It is not implemented in ModSecurity
 536                         2.x.</para>
 537                 </note>
 538             </section>
 539             <section>
 540                 <title>Intended Response Body (<literal>E</literal>)</title>
 541                 <para>This part contains the transaction response body (before compression and
 542                     chunking, where used) that was either sent or would have been sent had
 543                     ModSecurity not intervened. You can find whether interception took place by
 544                     looking at the <literal>Action</literal> header of the part <literal
 545                     >H</literal>. If that header is present, and the interception took place in
 546                     phase 3 or 4 then the <literal>E</literal> part contains the intended response
 547                     body. Otherwise, it contains the actual response body.</para>
 548                 <note>
 549                     <para>Once the <literal>G</literal> (actual response body) part is implemented,
 550                         part <literal>E</literal> will be present only in audit logs that contain a
 551                         transaction that was intercepted, and there will be no need for further
 552                         analsys.</para>
 553                 </note>
 554             </section>
 555             <section>
 556                 <title>Response Headers (<literal>F</literal>)</title>
 557                 <para>This part contains the actual response headers sent to the client. Since
 558                     ModSecurity 2.x for Apache does not access the raw connection data, it
 559                     constructs part <literal>F</literal> out of the internal Apache data structures
 560                     that hold the response headers.</para>
 561                 <para>Some headers (the <literal>Date</literal> and <literal>Server</literal>
 562                     response headers) are generated just before they are sent and ModSecurity is not
 563                     able to record those. You should note than ModSecurity is working as part of a
 564                     reverse proxy, the backend web server will have generated these two servers, and
 565                     in that case they will be recorded. </para>
 566             </section>
 567             <section>
 568                 <title>Response Body (G)</title>
 569                 <para>When implemented, this part will contain the actual response body before
 570                     compression and chunking.</para>
 571                 <note>
 572                     <para>This part is reserved for future use. It is not implemented in ModSecurity
 573                         2.x.</para>
 574                 </note>
 575             </section>
 576             <section>
 577                 <title>Audit Log Trailer (H)</title>
 578                 <para>Part <literal>H</literal> contains additional transaction meta-data that was
 579                     obtained from the web server or from ModSecurity itself. The part contains a
 580                     number of trailer headers, which are similar to HTTP headers (without support
 581                     for header folding):<orderedlist>
 582                         <listitem>
 583                             <para>Action</para>
 584                         </listitem>
 585                         <listitem>
 586                             <para>Apache-Error</para>
 587                         </listitem>
 588                         <listitem>
 589                             <para>Message</para>
 590                         </listitem>
 591                         <listitem>
 592                             <para>Producer</para>
 593                         </listitem>
 594                         <listitem>
 595                             <para>Response-Body-Transformed</para>
 596                         </listitem>
 597                         <listitem>
 598                             <para>Sanitised-Args</para>
 599                         </listitem>
 600                         <listitem>
 601                             <para>Sanitised-Request-Headers</para>
 602                         </listitem>
 603                         <listitem>
 604                             <para>Sanitised-Response-Headers</para>
 605                         </listitem>
 606                         <listitem>
 607                             <para>Server</para>
 608                         </listitem>
 609                         <listitem>
 610                             <para>Stopwatch</para>
 611                         </listitem>
 612                         <listitem>
 613                             <para>WebApp-Info</para>
 614                         </listitem>
 615                     </orderedlist></para>
 616                 <section>
 617                     <title>Action</title>
 618                     <para>The <literal>Action</literal> header is present only for the transactions
 619                         that were intercepted:</para>
 620                     <programlisting>Action: Intercepted (phase 2)</programlisting>
 621                     <para>The phase information documents the phase in which the decision to
 622                         intercept took place.</para>
 623                 </section>
 624                 <section>
 625                     <title>Apache-Error</title>
 626                     <para>The Apache-Error header contains Apache error log messages observed by
 627                         ModSecurity, excluding those sent by ModSecurity itself. For example:</para>
 628                     <programlisting>Apache-Error: [file "/tmp/buildd/apache2-2.0.54/build-tree/apache2/server/
 629   core.c"] [line 3505] [level 3] File does not exist: /var/www/www.
 630   modsecurity.org/fst/documentation/modsecurity-apache/2.5.0-dev2</programlisting>
 631                 </section>
 632                 <section>
 633                     <title>Message</title>
 634                     <para>Zero or more <literal>Message</literal> headers can be present in any
 635                         trailer, and each such header will represent a single ModSecurity warning or
 636                         error, displayed in the order they were raised.</para>
 637                     <para>The example below was broken into multiple lines to make it fit this
 638                         page:</para>
 639                     <programlisting>Message: Access denied with code 400 (phase 2). Pattern match "^\w+:/" at
 640   REQUEST_URI_RAW. [file "/etc/apache2/rules-1.6.1/modsecurity_crs_20_
 641   protocol_violations.conf"] [line "74"] [id "960014"] [msg "Proxy access
 642   attempt"] [severity "CRITICAL"] [tag "PROTOCOL_VIOLATION/PROXY_ACCESS"]</programlisting>
 643                 </section>
 644                 <section>
 645                     <title>Producer</title>
 646                     <para>The <literal>Producer</literal> header identifies the product that
 647                         generated the audit log. For example:</para>
 648                     <programlisting>Producer: ModSecurity for Apache/2.5.5 (http://www.modsecurity.org/).</programlisting>
 649                     <para>ModSecurity allows rule sets to add their own signatures to the <literal
 650                             >Producer</literal> information (this is done using the <literal
 651                             >SecComponentSignature</literal> directive). Below is an example of the
 652                             <literal>Producer</literal> header with the signature of one component
 653                         (all one line):</para>
 654                     <programlisting>Producer: ModSecurity for Apache/2.5.5 (http://www.modsecurity.org/);
 655     MyComponent/1.0.0 (Beta).</programlisting>
 656                 </section>
 657                 <section>
 658                     <title>Response-Body-Transformed</title>
 659                     <para>This header will appear in every audit log that contains a response
 660                         body:</para>
 661                     <programlisting>Response-Body-Transformed: Dechunked</programlisting>
 662                     <para>The contents of the header is constant at present, so the header is only
 663                         useful as a reminder that the recorded response body is not identical to the
 664                         one sent to the client. The actual content is the same, except that Apache
 665                         may further compress the body and deliver it in chunks.</para>
 666                 </section>
 667                 <section>
 668                     <title>Sanitised-Args</title>
 669                     <para>The <literal>Sanitised-Args</literal> header contains a list of arguments
 670                         that were sanitised (each byte of their content replaced with an asterisk)
 671                         before logging. For example:</para>
 672                     <programlisting>Sanitised-Args: "old_password", "new_password", "new_password_repeat".</programlisting>
 673                 </section>
 674                 <section>
 675                     <title>Sanitised-Request-Headers</title>
 676                     <para>The <literal>Sanitised-Request-Headers</literal> header contains a list of
 677                         request headers that were sanitised before logging. For example:</para>
 678                     <programlisting>Sanitised-Request-Headers: "Authentication".</programlisting>
 679                 </section>
 680                 <section>
 681                     <title>Sanitised-Response-Headers</title>
 682                     <para>The <literal>Sanitised-Response-Headers</literal> header contains a list
 683                         of response headers that were sanitised before logging. For example:</para>
 684                     <programlisting>Sanitised-Response-Headers: "My-Custom-Header".</programlisting>
 685                 </section>
 686                 <section>
 687                     <title>Server</title>
 688                     <para>The <literal>Server</literal> header identifies the web server. For
 689                         example:</para>
 690                     <programlisting>Server: Apache/2.0.54 (Debian GNU/Linux) mod_ssl/2.0.54 OpenSSL/0.9.7e</programlisting>
 691                     <para>This information may sometimes be present in any of the parts that contain
 692                         response headers, but there are a few cases when it isn't:<orderedlist>
 693                             <listitem>
 694                                 <para>None of the response headers were recoreded.</para>
 695                             </listitem>
 696                             <listitem>
 697                                 <para>The information in the response headers is not accurate
 698                                     because server signature masking was used.</para>
 699                             </listitem>
 700                         </orderedlist></para>
 701                 </section>
 702                 <section>
 703                     <title>Stopwatch</title>
 704                     <para>The <literal>Stopwatch</literal> header provides certain diagnostic
 705                         information that allows you to determine the performance of the web server
 706                         and of ModSecurity itself. It will typically look like this:</para>
 707                     <programlisting>Stopwatch: 1222945098201902 2118976 (770* 4400 -)</programlisting>
 708                     <para>Each line can contain up to 5 different values. Some values can be absent;
 709                         each absent value will be replaced with a dash.</para>
 710                     <para>The meanings of the values are as follows (all values are in
 711                             microseconds):<orderedlist>
 712                             <listitem>
 713                                 <para>Transaction timestamp in microseconds since January 1st,
 714                                     1970.</para>
 715                             </listitem>
 716                             <listitem>
 717                                 <para>Transaction duration.</para>
 718                             </listitem>
 719                             <listitem>
 720                                 <para>The time between the moment Apache started processing the
 721                                     request and until phase 2 of ModSecurity began. If an asterisk
 722                                     is present that means the time includes the time it took
 723                                     ModSecurity to read the request body from the client (typically
 724                                     slow). This value can be used to provide a rough estimate of the
 725                                     client speed, but only with larger request bodies (the smaller
 726                                     request bodies may arrive in a single TCP/IP packet).</para>
 727                             </listitem>
 728                             <listitem>
 729                                 <para>The time between the start of processing and until phase 2 was
 730                                     completed. If you substract the previous value from this value
 731                                     you will get the exact duration of phase 2 (which is the main
 732                                     rule processing phase).</para>
 733                             </listitem>
 734                             <listitem>
 735                                 <para>The time between the start of request processing and util we
 736                                     began sending a fully-buffered response body to the client. If
 737                                     you substract this value from the total transaction duration and
 738                                     divide with the response body size you may get a rough estimate
 739                                     of the client speed, but only for larger response bodies.</para>
 740                             </listitem>
 741                         </orderedlist></para>
 742                 </section>
 743                 <section>
 744                     <title>WebApp-Info</title>
 745                     <para>The <literal>WebApp-Info</literal> header contains information on the
 746                         application to which the recorded transaction belongs. This information will
 747                         appear only if it is known, which will happen if <literal
 748                             >SecWebAppId</literal> was set, or <literal>setsid</literal> or <literal
 749                             >setuid</literal> executed in the transaction.</para>
 750                     <para>The header uses the following format:</para>
 751                     <programlisting>WebApp-Info: "WEBAPPID" "SESSIONID" "USERID"</programlisting>
 752                     <para>Each unknown value is replaced with a dash.</para>
 753                 </section>
 754             </section>
 755             <section>
 756                 <title>Reduced Multipart Request Body (<literal>I</literal>)</title>
 757                 <para>Transactions that deal with file uploads tend to be large, yet the file
 758                     contents is not always relevant from the security point of view. The <literal
 759                         >I</literal> part was designed to avoid recording raw <literal
 760                         >multipart/form-data</literal> request bodies, replacing them with a
 761                     simulated <literal>application/x-www-form-urlencoded</literal> body that
 762                     contains the same key-value parameters.</para>
 763                 <para>The reduced multipart request body will not contain any file information. The
 764                         <literal>J</literal> part (currently not implemented) is intended to carry
 765                     the file metadata.</para>
 766             </section>
 767             <section>
 768                 <title>Multipart Files Information (<literal>J</literal>)</title>
 769                 <para>The purpose of part <literal>J</literal> is to record the information on the
 770                     files contained in a <literal>multipart/form-data</literal> request body. This
 771                     is handy in the cases when the original request body was not recorded, or when
 772                     only a reduced version was recorded (e.g. when part <literal>I</literal> was
 773                     used instead of part <literal>C</literal>).</para>
 774                 <note>
 775                     <para>This part is reserved for future use. It is not implemented in ModSecurity
 776                         2.x.</para>
 777                 </note>
 778             </section>
 779             <section>
 780                 <title>Matched Rules (<literal>K</literal>)</title>
 781                 <para>The matched rules part contains a record of all ModSecurity rules that matched
 782                     during transaction processing. You should note that if a rule that belongs to a
 783                     chain matches then the entire chain will be recorded. This is because, even
 784                     though the disruptive action may not have executed, other per-rule actions have,
 785                     and you will need to see the entire chain in order to understand the
 786                     rules.</para>
 787                 <para>This part is available starting with ModSecurity 2.5.x.</para>
 788             </section>
 789             <section>
 790                 <title>Audit Log Footer (<literal>Z</literal>)</title>
 791                 <para>Part <literal>Z</literal> is a special part that only has a boundary but no
 792                     content. Its only purpose is to signal the end of an audit log.</para>
 793             </section>
 794         </section>
 795         <section>
 796             <title>Storage Formats</title>
 797             <para>ModSecurity supports two audit log storage formats:<orderedlist>
 798                     <listitem>
 799                         <para><emphasis>Serial</emphasis> audit log format - multiple audit log
 800                             files stored in the same file.</para>
 801                     </listitem>
 802                     <listitem>
 803                         <para><emphasis>Concurrent</emphasis> audit log format - one file is used
 804                             for every audit log.</para>
 805                     </listitem>
 806                 </orderedlist></para>
 807             <section>
 808                 <title>Serial Audit Log Format</title>
 809                 <para>The serial audit log format stores multiple audit log entries within the same
 810                     file (one after another). This is often very convinent (audit log entries are
 811                     easy to find) but this format is only suitable for light logging in the current
 812                     ModSecurity implementation because writing to the file is serialised: only one
 813                     audit log entry can be written at any one time.</para>
 814             </section>
 815             <section>
 816                 <title>Concurrent Audit Log Format</title>
 817                 <para>The concurrent audit log format uses one file per audit log entry, and allows
 818                     many transactions to be recorded at once. A hierarchical directory structure is
 819                     used to ensure that the number of files created in any one directory remains
 820                     relatively small. For example:</para>
 821                 <programlisting>$LOGGING-HOME/20081128/20081128-1414/20081128-141417-
 822   egDKy38AAAEAAAyMHXsAAAAA</programlisting>
 823                 <para>The current time is used to work out the directory structure. The file name is
 824                     constructed using the current time and the transaction ID.</para>
 825                 <para>The creation of every audit log in concurrent format is recorded with an entry
 826                     in the concurrent audit log <emphasis>index file</emphasis>. The format of each
 827                     line resembles the common web server access log format. For example:</para>
 828                 <programlisting>192.168.0.111 192.168.0.1 - - [28/Nov/2008:15:06:32 +0000]
 829   "GET /?p=\\ HTTP/1.1" 200 69 "-" "-" NOfRx38AAAEAAAzcCU4AAAAA
 830   "-" /20081128/20081128-1506/20081128-150632-NOfRx38AAAEAAAzcCU4AAAAA
 831   0 1183 md5:ffee2d414cd43c2f8ae151652910ed96</programlisting>
 832                 <para>The tokens on the line are as follows:</para>
 833                 <orderedlist>
 834                     <listitem>
 835                         <para>Hostname (or IP address, if the hostname is not known)</para>
 836                     </listitem>
 837                     <listitem>
 838                         <para>Source IP address</para>
 839                     </listitem>
 840                     <listitem>
 841                         <para>Remote user (from HTTP Authentication)</para>
 842                     </listitem>
 843                     <listitem>
 844                         <para>Local user (from identd)</para>
 845                     </listitem>
 846                     <listitem>
 847                         <para>Timestamp</para>
 848                     </listitem>
 849                     <listitem>
 850                         <para>Request line</para>
 851                     </listitem>
 852                     <listitem>
 853                         <para>Response status</para>
 854                     </listitem>
 855                     <listitem>
 856                         <para>Bytes sent (in the response body)</para>
 857                     </listitem>
 858                     <listitem>
 859                         <para>Referrer information</para>
 860                     </listitem>
 861                     <listitem>
 862                         <para>User-Agent information</para>
 863                     </listitem>
 864                     <listitem>
 865                         <para>Transaction ID</para>
 866                     </listitem>
 867                     <listitem>
 868                         <para>Session ID</para>
 869                     </listitem>
 870                     <listitem>
 871                         <para>Audit log file name (relative to the audit logging home, as configured
 872                             using the <literal>SecAuditLogStorageDir</literal> directive)</para>
 873                     </listitem>
 874                     <listitem>
 875                         <para>Audit log offset</para>
 876                     </listitem>
 877                     <listitem>
 878                         <para>Audit log size</para>
 879                     </listitem>
 880                     <listitem>
 881                         <para>Audit log hash (the has begins with the name of the algorithm used,
 882                             followed by a colon, followed by the hexadecimal representation of the
 883                             hash itself); this hash can be used to verify that the transaction was
 884                             correctly recorded and that it hasn't been modified since.</para>
 885                     </listitem>
 886                 </orderedlist>
 887                 <note>
 888                     <para>Lines in the index file will be up to 3980 bytes long, and the information
 889                         logged will be reduced to fit where necessary. Reduction will occur within
 890                         the individual fields, but the overall format will remain the same. The
 891                         character <literal>L</literal> will appear as the last character on a
 892                         reduced line. A space will be the last character on a line that was not
 893                         reduced to stay within the limit.</para>
 894                 </note>
 895             </section>
 896         </section>
 897         <section>
 898             <title>Transport Protocol</title>
 899             <para>Audit logs generated in multi-sensor deployments are of little use if left on the
 900                 sensors. More commonly, they will be transported to a central logging server using
 901                 the transport protocol described in this section:</para>
 902             <orderedlist>
 903                 <listitem>
 904                     <para>The transport protocol is based on the HTTP protocol.</para>
 905                 </listitem>
 906                 <listitem>
 907                     <para>The server end is an SSL-enabled web server with HTTP Basic Authentication
 908                         configured.</para>
 909                 </listitem>
 910                 <listitem>
 911                     <para>Clients will open a connection to the centralisation web server and
 912                         authenticate (given the end-point URI, the username and the
 913                         password).</para>
 914                 </listitem>
 915                 <listitem>
 916                     <para>Clients will submit every audit log in a single <literal>PUT</literal>
 917                         transaction, placing the file in the body of the request and additional
 918                         information in the request headers (see below for details).</para>
 919                 </listitem>
 920                 <listitem>
 921                     <para>Server will process each submission and respond with an appropriate status
 922                         code: </para>
 923                     <orderedlist>
 924                         <listitem>
 925                             <para>200 (OK) - the submission was processed; the client can delete the
 926                                 corresponding audit log entry if it so desires. The same audit log
 927                                 entry must not be submitted again.</para>
 928                         </listitem>
 929                         <listitem>
 930                             <para>409 (Conflict) - if the submission is in invalid format and cannot
 931                                 be processed. The client should attempt to fix the problem with the
 932                                 submission and attempt delivery again at a later time. This error is
 933                                 generally going to occur due to a programming error in the protocol
 934                                 implementation, and not because of the content of the audit log
 935                                 entry that is being transported. </para>
 936                         </listitem>
 937                         <listitem>
 938                             <para>500 (Internal Server Error) - if the server was unable to
 939                                 correctly process the submission, due to its own fault. The client
 940                                 should re-attempt delivery at a later time. A client that starts
 941                                 receiving 500 reponses to all its submission should suspend its
 942                                 operations for a period of time before continuing.</para>
 943                         </listitem>
 944                     </orderedlist>
 945                 </listitem>
 946             </orderedlist>
 947             <note>
 948                 <para>Server implementations are advised to accept all submissions that correctly
 949                     implement the protocol. Clients are unlikely to be able to overcome problems
 950                     within audit log entries, so such problems are best resolved on the server
 951                     side.</para>
 952             </note>
 953             <note>
 954                 <para>When en error occurs, the server may place an explanation of the problem in
 955                     the text part of the response line.</para>
 956             </note>
 957             <section>
 958                 <title>Request Headers Information</title>
 959                 <para>Each audit log entry submission must contain additional information in the
 960                     request headers:</para>
 961                 <orderedlist>
 962                     <listitem>
 963                         <para>Header <literal>X-Content-Hash</literal> must contain the audit log
 964                             entry hash. Clients should expect the audit log entries to be validated
 965                             against the hash by the server.</para>
 966                     </listitem>
 967                     <listitem>
 968                         <para>Header <literal>X-ForensicLog-Summary</literal> must contain the
 969                             entire concurrent format index line.</para>
 970                     </listitem>
 971                     <listitem>
 972                         <para>The <literal>Content-Lenght</literal> header must be present and
 973                             contain the length of the audit log entry.</para>
 974                     </listitem>
 975                 </orderedlist>
 976             </section>
 977         </section>
 978     </section>
 979 </article>