dkimpy

Folding Whitespace incompatible with the libmilter spec

Asked by Stephen Nightingale

Claus Assman at Sendmail points me to the libmilter specification for folding lines in headers transmitted to Sendmail. It seems to be at variance with RFC6376 Section 2.8 Whitespace.

The Libmilter specification, not an RFC, is the best example of a "samizdat" document I have yet seen. However since it is the operationally ruling protocol, perhaps DKIM folding whitespace should be in alignment with it?

Stephen Nightingale.

From Claus Assman at Sendmail:

Do you mean the milter uses \r\n?
The libmilter docs states:

      * Neither the name nor the value of the header is checked for
        standards compliance. However, each line of the
        header must be under 2048 characters and should be under 998
        characters. If longer headers are needed, make them multi-line.
        To make a multi-line header, insert a line feed (ASCII 0x0a,
        or \n in C) followed by at least one whitespace character
        such as a space (ASCII 0x20) or tab (ASCII 0x09, or \t in
        C). The line feed should NOT be preceded by a carriage return
        (ASCII 0x0d); the MTA will add this automatically. It is the
        filter writer's responsibility to ensure that no standards
        are violated.

My answer:
The folding algorithm in dkim.py is inserting "CRLF<sp>", thus.
This is based on the specification of folding whitespace in RFC6376,
Section 2.8, Page 15.

def fold(header):
     """Fold a header line into multiple crlf-separated lines at column 72.
     """
     i = header.rfind(b"\r\n ")
     if i == -1:
         pre = b""
     else:
         i += 3
         pre = header[:i]
         header = header[i:]
     while len(header) > 72:
         i = header[:72].rfind(b" ")
         if i == -1:
             j = 72
         else:
             j = i + 1
         pre += header[:j] + b"\r\n "
         header = header[j:]
     return pre + header

Question information

Language:
English Edit question
Status:
Answered
For:
dkimpy Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Scott Kitterman (kitterman) said : 		#1

It's also the same requirement in RFC 5322 (and I suspect 2322 and 822, but I didn't check) paragraph 3.2.2, so Sendmail is completely out of line here. I know the implementer of a very popular DKIM milter. I'll ask them how they handled the problem.

Revision history for this message
Scott Kitterman (kitterman) said : 		#2

Other implementations work around this libmilter mis-design as well, so I suppose we should too.

Revision history for this message
Stuart Gathman (stuart-gathman) said : 		#3

The libmilter API is only indirectly related to the SMTP RFCs you guys are quoting. Sendmail does indeed do the RFC compliant thing with header folding. You are complaining about the API to pass multiline headers to libmilter. There are no RFCs for this (the API is what it is), it does not interfere with SMTP RFCs in any way, and frankly, it is a perfectly reasonable API. Your milter is *NOT* an SMTP server. It merely provides extensions to an SMTP server.

Furthermore, you are directly dealing with the *python* API, not the libmilter API. For instance, I could trivially have the header callback in pymilter provide/use a list (array) instead of a string with '\n' separators. You could subclass Milter.Base to do that, or even take a '\r\n' separated string if that makes the rest of your milter code happier. If a lot of people need to reuse SMTP server code in their milters, I could include a Milter.Base subclass standard that uses '\r\n' separators.

Revision history for this message
Stuart Gathman (stuart-gathman) said : 		#4

That said, I forgot to map the separators in dkim-milter.py, so I'm going off to fix it. As to whether dkim.py should cater to the libmilter API, that is a good question. I think the default should be the form most suitable for an SMTP server. Remember, an SMTP proxy is a popular format for python to process mail streams - and that format is happier with dkim.py just the way it is. Maybe we could make the folding separator a global option?

Revision history for this message
Scott Kitterman (kitterman) said : 		#5

On Tuesday, July 03, 2012 07:26:07 PM you wrote:
> If a lot of people need to reuse SMTP server code in
> their milters, I could include a Milter.Base subclass standard that uses
> '\r\n' separators.

I did check and opendkim will deal with \n or \r\n to work around issues with
milter folded headers. I don't think it makes sense for pymilter to deviate
from the libmilter does. I do thank it makes sense to be flexible in dkimpy
since it has to work in multiple environments.

Revision history for this message
Stuart Gathman (stuart-gathman) said : 		#6

I'm tempted to rename Milter.Base.addheader to addheader_raw, and define:

def addheader(self,field,value,idx=-1):
   self.addheader_raw(field,value.replace('\r\n','\n'),idx)

That would Just Work™ in any sane situation, and people putting control chars in SMTP headers deserve far worse than a little search and replace...

Revision history for this message
Scott Kitterman (kitterman) said : 		#7

That doesn't help the original problem though since he's not using pymilter.

Revision history for this message
Stuart Gathman (stuart-gathman) said : 		#8

It's not that hard to just wrap the field_value argument to libmilter with a replace like the above.

Revision history for this message
Stuart Gathman (stuart-gathman) said : 		#9

If he is not using pymilter, why does he care about the dkimpy API? Is he using the pure python API? Then the ppmilter API is the target.

Can you help with this problem?

Provide an answer of your own, or ask Stephen Nightingale for more information if necessary.

To post a message you must log in.