While working on Flying Pie Notifier, I ran into a problem with Gmail marking the notification emails as spam. I was able to find the Gmail Bulk Sender Guidelines and make some changes. I think I got Gmail to finally recognize that the notifications are not spam.
My first step was to make sure that SPF was set up correctly and to check the reverse DNS PTR record for the IP address I was sending from. Both of these are simple to set up with just DNS records.
Unfortunately, those changes didn’t seems to fix the problem completely, so I had to move on to the next step, DKIM (DomainKeys Identified Mail). DKIM is a way of cryptographically signing emails that are sent to verify that they are sent from the domain they claim they were sent by.
I found a PHP library called PHP-DKIM that would perform the signing, but it was incompatible with the PEAR Mail_Mime module that I was using to send HTML email. I had to make some modifications to the code to get them to work together.
Here are the steps I used to set up DKIM signing with Pear::Mail_Mime and PHP-DKIM:
Generate a public and private key pair.
First I needed to generate a public and private key pair to use for signing messages and for other people to verify the signature. These commands generate 2 files, key.priv and key.pub, containing my new private and public keys:
openssl genrsa -out key.priv 2048
openssl rsa -in key.priv -out key.pub -pubout -outform PEM
- Publish the public key in a DNS record.
DKIM public keys are published in a special DNS record so the recipient can look up the key and use it to verify the message signature. I followed the PHP-DKIM README and put my keys in a dkim-cfg.php file along with my domain name and a selector name and ran dkim-test.php.
Loading dkim-test.php will tell you if you have any problems with your configuration so far, and if not it gives you a DNS record. For me, it generated this record which I added to my DNS:
p._domainkey.flyingpienotifier.com. TXT "v=DKIM1; k=rsa; g=*; s=email; h=sha1; t=y; p=MHwwDQYJKoZIhvcNAQEBBQADawAwaAJhAODvTnvu4+OO+47m/3iRImsshlhQ+u30 0cHSdCJPndsD2meo63lNl8ELktIZKUEEEGFYNbwuZHWISCmuFHEiQNIHeag+awen 75oPBZqd1ABVdjyTwZa1matMzQhg5rSGpQIDAQAB ;"
I then used http://www.sendmail.org/dkim/checker and http://domainkeys.sourceforge.net/selectorcheck.html to check my record and make sure it was set up correctly.
- Signing Code
I had downloaded the PHP-DKIM library from http://php-dkim.sourceforge.net/, but found that the way it treated headers was not compatible with the PEAR Mail_Mime library that I was already using. PHP-DKIM expects a single header string formatted the way headers are transmitted between servers when sending email. Mail_Mime wants an associative array of headers that it then uses to generate the header string when it sends the final message.
I decided to modify PHP-DKIM to get it to work with the associative array approach, since I thought that was the better of the solutions.
You can find my modified version of the library at http://www.ra726.net/php-dkim.zip. The only file I modified is dkim.php. I changed to use an array for headers and cleaned up the code a bit to make it easier (in my opinion) to read.
To make the library work with Mail_Mime header arrays, I added code to the AddDKIM method that concatenates the headers together. This is simple in PHP, and just requires:
$headers_line = '';
foreach ($headers as $key => $value)
$headers_line .= $key.': '.$value."rn";
After that, I just had to change what the function returned. Instead of returning a new header string, I just return the DKIM signature string. This can then be added to the headers using:
$dkim = AddDKIM($headers, $subject, $message);
$headers['DKIM-Signature'] = $dkim;
This code could be cleaned up to automatically add the header, but it works until I take the time to rewrite the library and make it object oriented.
When I first tried to use my new code to send emails to my own Gmail account, I found that I had some errors in my implementation. In Gmail, you can see the full headers and the results of DKIM and SPF checks by clicking the arrow next to Reply and selecting Show Original. This will show you the full headers of the message, including the verifications added by Gmail.
When I did this on my message, I got an error telling me the body hash was wrong. The body hash is used to verify that the email message was not tampered with during delivery. Since I had sent the message myself, I knew it had not been modified. Therefore, Gmail was calculating the hash based on a different version of the message than my current code.
I was able to find a service that could help me track down the problem. At http://www.port25.com/domainkeys/ you can find information about an automated responder that will let you know the results of the DKIM verification.
By sending my message to the Port25 reflector test address, I got a report letting me know all the details of the DKIM verification process. One of the pieces of information in this email was the canonicalized body of the email. Canonicalization is the process of standardizing the message to specific requirements to make sure both sides are doing the hash calculation on the same data. I could then use the NiceDump function in PHP-DKIM to print out the canonicalized message that I was calculating the hash on. This let me see where I had caused a problem when modifying the library.
Using the same testing address, I was able to find a few other problems with my modifications to the library and get it working. Once I had the bugs worked out, the reflector address reported the DKIM verification as passing. It worked!
There is no definite solution for not having your emails marked as spam. All you can do is try to follow all the guidelines that Gmail and other providers specify. It seems to me that Flying Pie Notifier is no longer being marked as spam, so I think I should be good for now. DKIM is a part of that, but I also had to make sure my mail server and DNS records were all set up correctly.
If you want a really detailed technical explanation of DKIM, you can find the specifications in the RFC at http://www.dkim.org/specs/rfc4871-dkimbase.html
If you have any issues with this code or are trying to implement it yourself, leave a comment and I can try to help you out.
– UPDATE –
Originally this post used a 384 bit private key. This key size is trivially broken now, so I have updated it to specify a 2048 bit key.