policyd-spf crashes on broken SPF records

Fails in python3 too.

Traceback (most recent call last):
  File "spf.py", line 1673, in <module>
    receiver=socket.gethostname()))
  File "spf.py", line 304, in check
    res,code,exp = query(i=i, s=s, h=h, local=local, receiver=receiver).check()
  File "spf.py", line 503, in check
    spf = self.dns_spf(self.d)
  File "spf.py", line 1051, in dns_spf
    a = [t for t in self.dns_txt(domain) if RE_SPF.match(t)]
  File "spf.py", line 1086, in dns_txt
    return [''.join(a) for a in self.dns(domainname, 'TXT')]
  File "spf.py", line 1193, in dns
    for k, v in DNSLookup(name, qtype, self.strict, self.timeout):
  File "spf.py", line 128, in DNSLookup
    resp = req.req()
  File "/usr/lib/python3/dist-packages/DNS/Base.py", line 213, in req
    self.sendUDPRequest(server)
  File "/usr/lib/python3/dist-packages/DNS/Base.py", line 242, in sendUDPRequest
    r=self.processUDPReply()
  File "/usr/lib/python3/dist-packages/DNS/Base.py", line 115, in processUDPReply
    return self.processReply()
  File "/usr/lib/python3/dist-packages/DNS/Base.py", line 139, in processReply
    r=Lib.DnsResult(u,self.args)
  File "/usr/lib/python3/dist-packages/DNS/Lib.py", line 603, in __init__
    self.storeM(u)
  File "/usr/lib/python3/dist-packages/DNS/Lib.py", line 659, in storeM
    self.answers.append(self.storeRR(u))
  File "/usr/lib/python3/dist-packages/DNS/Lib.py", line 686, in storeRR
    r['data']=getattr(u, mname)()
  File "/usr/lib/python3/dist-packages/DNS/Lib.py", line 483, in getTXTdata
    tlist.append(str(self.getstring(), enc))
  File "/usr/lib/python3.2/encodings/idna.py", line 192, in decode
    result.append(ToUnicode(label))
  File "/usr/lib/python3.2/encodings/idna.py", line 125, in ToUnicode
    return str(label, "ascii")
UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 9: ordinal not in range(128)

This should either be a pyspf or py(3)dns bug.

What distribution are you using and what version of pyspf and pydns do you have? This is actually a problem in the lower level python modules.

On 25/08/2011, at 5:47 AM, Scott Kitterman wrote:

> What distribution are you using

Ubuntu 10.04.3

> and what version of pyspf and pydns do you have?

postfix-policyd-spf-python 0.8.0-1
python-spf 2.0.5-2
python-dns 2.3.4-3

> This is actually a problem in the lower level python modules.

do I need to take any further action on this? (I'm used to the Debian process where the bug would just be reassigned to the correct package in this scenario. I have no idea how the Ubuntu/Launchpad process works from here on in.)

I'll take care of it. Thanks.

I'm seeing this same issue on my mail relay running pypolicyd-spf1.0

> What distribution are you using

I'm using Centos 5.8 with their python26 package installed.

> what version of pyspf and pydns do you have?

I installed pyspf 2.0.7 and pydns 2.3.6 from source. It looks like this is still an issue.

This should be fixed in pyspf (which I'll work on).

--- spf.py.orig 2012-06-14 19:56:11.568543789 +0000
+++ spf.py 2012-06-14 19:59:03.124546459 +0000
@@ -1129,7 +1129,7 @@
             try:
                 return [''.join(s.decode("ascii") for s in a)
                     for a in self.dns(domainname, 'TXT')]
- except UnicodeEncodeError:
+ except UnicodeError:
                 raise PermError('Non-ascii character in SPF TXT record.')
         return []
     def dns_99(self, domainname):
@@ -1138,7 +1138,7 @@
             try:
                 return [''.join(s.decode("ascii") for s in a)
                     for a in self.dns(domainname, 'SPF')]
- except UnicodeEncodeError:
+ except UnicodeError:
                 raise PermError('Non-ascii character in SPF record.')
         return []

is the fix for anyone that wants to patch it locally in the mean time.

I'm marking this bug invalid for pypolicyd-spf because it needs fixing in pyspf. I've committed the above changes upstream and uploaded a patched package for Debian and the Ubuntu development release. I'll also do a post-release update for Ubuntu 12.04.

Fix uploaded to Debian. It will land in Quantal after the next autosync.

Marked high because it can result in mail not getting delivered. SRU uploaded for 12.04 and waiting for SRU team review.

Did you mean to remove the Recommends: for python-authres? It's not mentioned in the changelog.

No. Fixing.

The package is fine as is. I neglected to mention adding the Recommends in the
Unstable/Quantal upload, but it's not needed for the SRU. Thanks for
checking.

Hello russell, or anyone else affected,

Accepted pyspf into precise-proposed. The package will build now and be available in a few hours. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case details of your testing will help us make a better decision. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Actual archive package works as expected.

This bug was fixed in the package pyspf - 2.0.7-1ubuntu0.1

---------------
pyspf (2.0.7-1ubuntu0.1) precise-proposed; urgency=low

  * Add debian/patches/catch_unicode_spf to correctly raise errors when SPF
    records contain unicode (LP: #832480)
 -- Scott Kitterman <email address hidden> Thu, 14 Jun 2012 16:13:23 -0400

Hello,

I have the same problem:
UnicodeDecodeError: 'ascii' codec can't decode byte 0x93 in position 0: ordinal not in range(128)
warning: command /usr/bin/policyd-spf exit status 1
warning: premature end-of-input on private/policyd-spf while reading input attribute name None; identity=helo; ...

I'm using the foloowing system:
fedora 16 3.4.2-1.fc16.x86_64 #1 SMP
pypolicyd-spf-1.0
authres-0.402
dkimpy-0.5.2
pydns-2.3.6
pyspf-2.0.7
PyYAML-3.10

Can You help with this problem?
Best regards, NC

Hello again,

It seams that using the patch from Scott Kitterman (kitterman) wrote on 2012-06-14: #6

This should be fixed in pyspf (which I'll work on).

--- spf.py.orig 2012-06-14 19:56:11.568543789 +0000
+++ spf.py 2012-06-14 19:59:03.124546459 +0000
@@ -1129,7 +1129,7 @@
             try:
                 return [''.join(s.decode("ascii") for s in a)
                     for a in self.dns(domainname, 'TXT')]
- except UnicodeEncodeError:
+ except UnicodeError:
                 raise PermError('Non-ascii character in SPF TXT record.')
         return []
     def dns_99(self, domainname):
@@ -1138,7 +1138,7 @@
             try:
                 return [''.join(s.decode("ascii") for s in a)
                     for a in self.dns(domainname, 'SPF')]
- except UnicodeEncodeError:
+ except UnicodeError:
                 raise PermError('Non-ascii character in SPF record.')
         return []

The crash is gone, but policyd-spf cannot verify the TXT DNS entry correctly. This is the log. The mail is received, but the envelop is not checked, giving the error: Permerror

Jul 16 10:50:39 mailr01 policyd-spf[20450]: None; identity=helo; client-ip=195.1.1.1; helo=bbb.pt; <email address hidden>; <email address hidden>
Jul 16 10:50:39 mailr01 policyd-spf[20450]: Permerror; identity=mailfrom; client-ip=195.1.1.1; helo=bbb.pt; <email address hidden>; <email address hidden>

Hope you can help resolving this issue.
Best regards, NC

Permerror is correct. Non-ASCII Unicode characters aren't allowed in SPF records. The sender needs to correct their record.

Hello Scott,

Thanks for your Help. I have forward the answer to the entity responsible for sending mails with Non-ASCII Unicode.

Best Regards,
NC