Exim version 4.31 ----------------- 1. Removed "EXTRALIBS=-lwrap" from OS/Makefile-Unixware7 on the advice of Larry Rosenman. 2. Removed "LIBS = -lresolv" from OS/Makefile-Darwin as it is not needed, and indeed breaks things for older releases. 3. Added additional logging to the case where there is a problem reading data from a filter that is running in a subprocess using a pipe, in order to try to track down a specific problem. 4. Testing facility fudge: when running in the test harness and attempting to connect to 10.x.x.x (expecting a connection timeout) I'm now sometimes getting "No route to host". Convert this to a timeout. 5. Define ICONV_ARG2_TYPE as "char **" for Unixware7 to avoid compiler warning. 6. Some OS don't have socklen_t but use size_t instead. This affects the fifth argument of getsockopt() amongst other things. This is now configurable by a macro called SOCKLEN_T which defaults to socklen_t, but can be set for individual OS. I have set it for SunOS5, OSF1, and Unixware7. Current versions of SunOS5 (aka Solaris) do have socklen_t, but some earlier ones do not. 7. Change 4.30/15 was not doing the test caselessly. 8. The standard form for an IPv6 address literal was being rejected by address parsing in, for example, MAIL and RCPT commands. An example of this kind of address is [IPv6:2002:c1ed:8229:10:202:2dff:fe07:a42a]. Exim now accepts this, as well as the form without the "IPv6" on the front (but only when address literals are enabled, of course). 9. Added some casts to avoid compiler warnings in OS/os.c-Linux. 10. Exim crashed if a message with an empty sender address specified by -f encountered a router with an errors_to setting. This could be provoked only by a command such as exim -f "" ... where an empty string was supplied; "<>" did not hit this bug. 11. Installed PCRE release 4.5. 12. If EHLO/HELO was rejected by an ACL, the value of $sender_helo_name remained set. It is now erased. 13. exiqgrep wasn't working on MacOS X because it didn't correctly compute times from message ids (which are base 36 rather than the normal 62). 14. "Expected" SMTP protocol errors that can arise when PIPELINING is in use were being counted as actual protocol errors, and logged if the log selector +smtp_protocol_error was set. One cannot be perfect in this test, but now, if PIPELINING has been advertised, RCPT following a rejected MAIL, and DATA following a set of rejected RCPTs do not count as protocol errors. In other words, Exim assumes they were pipelined, though this may not actually be the case. Of course, in all cases the client gets an appropriate error code. 15. If a lookup fails in an ACL condition, a message about the failure may be available; it is used if testing the ACL cannot continue, because most such messages specify what the cause of the deferral is. However, some messages (e.g. "MYSQL: no data found") do not cause a defer. There was bug that caused an old message to be retained and used if a later statement caused a defer, replacing the real cause of the deferral. 16. If an IP address had so many PTR records that the DNS lookup buffer was not large enough to hold them, Exim could crash while trying to process the truncated data. It now detects and logs this case. 17. Further to 4.21/58, another change has been made: if (and only if) the first line of a message (the first header line) ends with CRLF, a bare LF in a subsequent header line has a space inserted after it, so as not to terminate the header. 18. Refactoring: tidied an ugly bit of code in appendfile that copied data unnecessarily, used atoi() instead of strtol(), and didn't check the termination when getting file sizes from file names by regex. 19. Completely re-implemented the support for maildirsize files, in the light of a number of problems with the previous contributed implementation (4.30/29). In particular: . If the quota is zero, the maildirsize file is maintained, but no quota is imposed. . If the maildir directory does not exist, it is created before any attempt to write a maildirsize file. . The quota value in the file is just a cache; if the quota is changed in the transport, the new value overrides. . A regular expression is available for excluding directories from the count. 20. The autoreply transport checks the characters in options that define the message's headers; it allows continued headers, but it was checking with isspace() after an embedded newline instead of explicitly looking for a space or a tab. 21. If all the "regular" hosts to which an address was routed had passed their expiry times, and had not reached their retry times, the address was bounced, even if fallback hosts were defined. Now Exim should go on to try the fallback hosts. 22. Increased buffer sizes in the callout code from 1024 to 4096 to match the equivalent code in the SMTP transport. Some hosts send humungous responses to HELO/EHLO, more than 1024 it seems. 23. Refactoring: code in filter.c used (void *) for "any old type" but this gives compiler warnings in some environments. I've now done it "properly", using a union. 24. The replacement for inet_ntoa() that is used with gcc on IRIX systems (because of problems with the built-in one) was declared to return uschar * instead of char *, causing compiler failure. 25. Fixed a file descriptor leak when processing alias/forward files. 26. Fixed a minor format string issue in dbfn.c. 27. Typo in exim.c: ("dmbnz" for "dbmnz"). 28. If a filter file refered to $h_xxx or $message_headers, and the headers contained RFC 2047 "words", Exim's memory could, under certain conditions, become corrupted. 29. When a sender address is verified, it is cached, to save repeating the test when there is more than one recipient in a message. However, when the verification involves a callout, it is possible for different callout options to be set for different recipients. It is too complicated to keep track of this in the cache, so now Exim always runs a verification when a callout is required, relying on the callout cache for the optimization. The overhead is duplication of the address routing, but this should not be too great. 30. Fixed a bug in callout caching. If a RCPT command caused the sender address to be verified with callout=postmaster, and the main callout worked but the postmaster check failed, the verification correctly failed. However, if a subsequent RCPT command asked for sender verification *without* the postmaster check, incorrect caching caused this verification also to fail, incorrectly. 31. Exim caches DNS lookup failures so as to avoid multiple timeouts; however, it was not caching the DNS options (qualify_single, search_parents) that were used when the lookup failed. A subsequent lookup with different options therefore always gave the same answer, though there were cases where it should not have. (Example: a "domains = !$mx_any" option on a dnslookup router: the "domains" option is always processed without any widening, but the router might have qualify_single set.) Now Exim uses the cached value only when the same options are set. 32. Added John Jetmore's "exipick" utility to the distribution. 33. GnuTLS: When an attempt to start a TLS session fails for any reason other than a timeout (e.g. a certificate is required, and is not provided), an Exim server now closes the connection immediately. Previously it waited for the client to close - but if the client is SSL, it seems that they each wait for each other, leading to a delay before one of them times out. 34: GnuTLS: Updated the code to use the new GnuTLS 1.0.0 API. I have not maintained 0.8.x compatibility because I don't think many are using it, and it is clearly obsolete. 35. Added TLS support for CRLs: a tls_crl global option and one for the smtp transport. 36. OpenSSL: $tls_certificate_verified was being set to 1 even if the client certificate was expired. A simple patch fixes this, though I don't understand the full logic of why the verify callback is called multiple times. 37. OpenSSL: a patch from Robert Roselius: "Enable client-bug workaround. Versions of OpenSSL as of 0.9.6d include a 'CBC countermeasure' feature, which causes problems with some clients (such as the Certicom SSL Plus library used by Eudora). This option, SSL_OP_DONT_INSERT_EMPTY_FRAGMENTS, disables the coutermeasure allowing Eudora to connect." 38. Exim was not checking that a write() to a log file succeeded. This could lead to Bad Things if a log got too big, in particular if it hit a file size limit. Exim now panics and dies if it cannot write to a log file, just as it does if it cannot open a log file. 39. Modified OS/Makefile-Linux so that it now contains CFLAGS=-O -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE The two -D definitions ensure that Exim is compiled with large file support, which makes it possible to handle log files that are bigger than 2^31. 40. Fixed a subtle caching bug: if (in an ACL or a set of routers, for instance) a domain was checked against a named list that involved a lookup, causing $domain_data to be set, then another domain was checked against the same list, then the first domain was re-checked, the value of $domain_data after the final check could be wrong. In particular, if the second check failed, it could be set empty. This bug probably also applied to $localpart_data. 41. The strip_trailing_dot option was not being applied to the address given with the -f command-line option. 42. The code for reading a message's header from the spool was incrementing $received_count, but never initializing it. This meant that the value was incorrect (doubled) while delivering a message in the same process in which it was received. In the most common configuration of Exim, this never happens - a fresh exec is done - but it can happen when deliver_drop_privilege is set. 43. When Exim logs an SMTP synchronization error - client data sent too soon - it now includes up to 150 characters of the unexpected data in the log line. 44. The exim_dbmbuild utility uses fixed size buffers for reading input lines and building data strings. The size of both of these buffers was 10 000 bytes - far larger than anybody would *ever* want, thought I. Needless to say, somebody hit the limit. I have increased the maximum line length to 20 000 and the maximum data length of concatenated lines to 100 000. I have also fixed two bugs, because there was no checking on these buffers. Tsk, tsk. Now exim_dbmbuild gives a message and exits with an error code if a buffer is too small. 45. The exim_dbmbuild utility did not support quoted keys, as Exim does in lsearch lookups. Now it does. 46. When parsing a route_list item in a manualroute router, a fixed-length buffer was used for the list of hosts. I made this 1024 bytes long, thinking that nobody would ever have a list of hosts that long. Wrong. Somebody had a whole pile of complicated expansion conditions, and the string was silently truncated, leading to an expansion error. It turns out that it is easier to change to an unlimited length (owing to other changes that have happened since this code was originally written) than to build structure for giving a limitation error. The length of the item that expands into the list of hosts is now unlimited. 47. The lsearch lookup could not handle data where the length of text line was more than 4095 characters. Such lines were truncated, leading to shortened data being returned. It should now handle lines of any length. 48. Minor wording revision: "cannot test xxx in yyy ACL" becomes "cannot test xxx condition in yyy ACL" (e.g. "cannot test domains condition in DATA ACL"). 49. Cosmetic tidy to scripts like exicyclog that are generated by globally replacing strings such as BIN_DIRECTORY in a source file: the replacement no longer happens in comment lines. A list of replacements is now placed at the head of all of the source files, except those whose only change is to replace PERL_COMMAND in the very first #! line. 50. Replaced the slow insertion sort in queue.c, for sorting the list of messages on the queue, with a bottom-up merge sort, using code contributed by Michael Haardt. This should make operations like -bp somewhat faster on large queues. It won't affect queue runners, except when queue_run_in_order is set. 51. Installed eximstats 1.31 in the distribution. 52. Added support for SRV lookups to the dnslookup router. 53. If an ACL referred to $message_body or $message_body_end, the value was not reset for any messages that followed in the same SMTP session. 54. The store-handling optimization for building very long strings was not differentiating between the different store pools. I don't think this actually made any difference in practice, but I've tidied it. 55. While running the routers to verify a sender address, $sender_address was still set to the sender address. This is wrong, because when routing to send a bounce to the sender, it would be empty. Therefore, I have changed it so that, while verifying a sender address, $sender_address is set to <>. (There is no change to what happens when verifying a recipient address.) 56. After finding MX (or SRV) records, Exim was doing a DNS lookup for the target A or AAAA records (if not already returned) without resetting the qualify_single or search_parents options of the DNS resolver. These are inappropriate in this case because the targets of MX and SRV records must be FQDNs. A broken DNS record could cause trouble if it happened to have a target that, when qualified, matched something in the local domain. These two options are now turned off when doing these lookups. 57. It seems that at least some releases of Reiserfs (which does not have the concept of a fixed number of inodes) returns zero and not -1 for the number of available inodes. This interacted badly with check_spool_inodes, which assumed that -1 was the "no such thing" setting. What I have done is to check that the total number of inodes is greater than zero before doing the test of how many are available. 58. When a "warn" ACL statement has a log_message modifier, the message is remembered, and not repeated. This is to avoid a lot of repetition when a message has many recipients that cause the same warning to be written. Howewer, Exim was preserving the list of already written lines for an entire SMTP session, which doesn't seem right. The memory is now reset if a new message is started. 59. The "rewrite" debugging flag was not showing the result of rewriting in the debugging output unless log_rewrite was also set. 60. Avoid a compiler warning on 64-bit systems in dsearch.c by avoiding the use of (int)(handle) when we know that handle contains (void *)(-1). 61. The Exim daemon panic-logs an error return when it closes the incoming connection. However "connection reset by peer" seems to be common, and isn't really an error worthy of noting specially, so that particular error is no long logged. 62. When Exim is trying to find all the local interfaces, it used to panic and die if the ioctl to get the interface flags failed. However, it seems that on at least one OS (Solaris 9) it is possible to have an interface that is included in the list of interfaces, but for which you get a failure error for this call. This happens when the interface is not "plumbed" into a protocol (i.e. neither IPv4 nor IPv6). I've changed the code so that a failure of the "get flags" call assumes that the interface is down. 63. Added a ${eval10: operator, which assumes all numbers are decimal. This makes life easier for people who are doing arithmetic on fields extracted from dates, where you often get leading zeros that should not be interpreted as octal. 64. Added qualify_domain to the redirect router, to override the global setting. 65. If a pathologically long header line contained very many addresses (the report of this problem mentioned 10 000) and each of them was rewritten, Exim could use up a very large amount of memory. (It kept on making new copies of the header line as it rewrote, and never released the old ones.) At the expense of a bit more processing, the header rewriting function has been changed so that it no longer eats memory in this way. 66. The generation of the Received: header has been moved from the time that a message starts to be received, to the time that it finishes. The timestamp in the Received: header should now be very close to that of the <= log line. There are two side-effects of this change: (a) If a message is rejected by a DATA or non-SMTP ACL or local_scan(), the logged header lines no longer include the local Received: line, because it has not yet been created. The same applies to a copy of the message that is returned to a non-SMTP sender when a message is rejected. (b) When a filter file is tested using -bf, no additional Received: header is added to the test message. After some thought, I decided that this is a bug fix. This change does not affect the value of $received_for. It is still set after address rewriting, but before local_scan() is called. 67. Installed the latest Cygwin-specific files from the Cygwin maintainer. 68. GnuTLS: If an empty file is specified for tls_verify_certificates, GnuTLS gave an unhelpful panic error message, and a defer error. I have managed to change this behaviour so that it now rejects any supplied certificate, which seems right, as the list of acceptable certificates is empty. 69. OpenSSL: If an empty file is specified for tls_verify_certificates, OpenSSL gave an unhelpful defer error. I have not managed to make this reject any supplied certificates, but the error message it gives is "no certificate supplied", which is not helpful. 70. exigrep's output now also includes lines that are not associated with any message, but which match the given pattern. Implemented by a patch from Martin Sluka, which also tidied up the Perl a bit. 71. Recipient callout verification, like sender verification, was using <> in the MAIL FROM command. This isn't really the right thing, since the actual sender may affect whether the remote host accepts the recipient or not. I have changed it to use the actual sender in the callout; this means that the cache record is now keyed on a recipient/sender pair, not just the recipient address. There doesn't seem to be a real danger of callout loops, since a callout by the remote host to check the sender would use <>. 72. Exim treats illegal SMTP error codes that do not begin with 4 or 5 as temporary errors. However, in the case of such a code being given after the end of a data transmission (i.e. after ".") Exim was failing to write a retry record for the message. (Yes, there was some broken host that was actually sending 8xx at this point.) 73. An unknown lookup type in a host list could cause Exim to panic-die when the list was checked. (An example that provoked this was putting <; in the middle of a list instead of at the start.) If this happened during a DATA ACL check, a -D file could be left lying around. This kind of configuration error no longer causes Exim to die; instead it causes a defer errror. The incident is still logged to the main and panic logs. 74. Buglet left over from Exim 3 conversion. The message "too many messages in one connection" was written to the rejectlog but not the mainlog, except when address rewriting (yes!) was being logged. 75. Added write_rejectlog option. 76. When a system filter was run not as root (that is, when system_filter_user was set), the values of the $n variables were not being returned to the main process; thus, they were not subsequently available in the $sn variables. 77. Added +return_path_on_delivery log selector. 78. A connection timeout was being treated differently from recipients deferred when testing hosts_max_try with a message that was older than the host's retry timeout. (The host should not be counted, thus allowing all hosts to be tried at least once before bouncing.) This may have been the cause of an occasionally reported bug whereby a message would remain on the queue longer than the retry timeout, but would be bounced if a delivery was forced. I say "may" because I never totally pinned down the problem; setting up timeout/retry tests is difficult. See also the next item. 79. The ultimate address timeout was not being applied to errors that involved a combination of host plus message (for example, a timeout on a MAIL command). When an address resolved to a number of possible hosts, and they were not all tried for each delivery (e.g. because of hosts_max_try), a message could remain on the queue longer than the retry timeout. 80. Sieve bug: "stop" inside "elsif" was broken. Applied a patch from Michael Haardt. 81. Fixed an obscure SMTP outgoing bug which required at least the following conditions: (a) there was another message waiting for the same server; (b) the server returned 5xx to all RCPT commands in the first message so that the message was not completed; (c) the server dropped the connection or gave a negative response to the RSET that Exim sends to abort the transaction. The observed case was a dropped connection after DATA that had been sent in pipelining mode. That is, the server had advertised PIPELINING but was not implementing it correctly. The effect of the bug was incorrect behaviour, such as trying another host, and this could lead to a crash. ****