Broken DKIM Signatures on Office365
It always came across funny before when other people where having bad experiences with Microsoft support for Office365, leaving us thinking thank god that isn't us!
Now we've found our own issue and wow, was that account accurate.
The support so far has been abysmal!
I almost thought I was taking part in the new series of Channel 4's FoneJacker.
As a bit of background, we have split delivery with our MX pointing at Office365 and email for a select few users being sent to G Suite via a mail flow rule and an Outbound Gateway configured to point to Gmail.
We first noticed that delivery of messages with attachments from one of our 3rd party suppliers wasn't making it to the users on Gmail, but was making it to the users on Office365.
To make matters worse, the 3rd party supplier was getting a very misleading bounce message about SPF records, at first glance instructing them to add "spf.protection.outlook.com" to their SPF.
Your message to firstname.lastname@example.org couldn't be delivered. a1comms.com couldn't confirm that your message was sent from a trusted location.
****.****** Office 365 sam.m**** Action Required Recipient
SPF validation error
How to Fix It Your organization's email admin will have to diagnose and fix your domain's email settings. Please forward this message to your email admin.
More Info for Email Admins
Status code: 550 5.7.23
This error occurs when Sender Policy Framework (SPF) validation for the sender's domain fails. If you're the sender's email admin, make sure the SPF records for your domain at your domain registrar are set up correctly. Office 365 supports only one SPF record (a TXT record that defines SPF) for your domain. Include the following domain name: spf.protection.outlook.
com. If you have a hybrid configuration (some mailboxes in the cloud, and some mailboxes on premises) or if you're an Exchange Online Protection standalone customer, add the outbound IP address of your on-premises servers to the TXT record.
For more information and instructions about configuring SPF records see Customize an SPF record to validate outbound mail sent from your domain and also External Domain Name System records for Office 365.
Original Message Details
Created Date: 12/12/2017 2:10:29 PM Sender Address: ****.****@cloud****.co .uk Recipient Address: email@example.com Subject: Re: FW: **REDACTED**
Reported error: 550 5.7.23 The message was rejected because of Sender Policy Framework violation -> 550 5.7.1 Unauthenticated email from **REDACTED**.co.uk is not accepted due to;domain's DMARC policy. Please contact the administrator of;**REDACTED**.co.ukdomain if this was a legitimate mail. Please;visit; https://support. google.com/mail/answer/2451690 to learn about the;DMARC initiative. d76si12884861pfk.321 - gsmtp DSN generated by: VI1PR0401MB2302.eurprd04.prod. outlook.com Remote server: mx.google.com
Message Hops HOP TIME (UTC) FROM TO WITH RELAY TIME 1 12/12/2017
10.200.17.150 HTTP * 2 12/12/2017
mail-qt0-f169.google.com SMTP 19 sec 3 12/12/2017
mail-qt0-f169.google.com HE1EUR01FT024.mail.protection. outlook.com Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_ 256_CBC_SHA_P384) * 4 12/12/2017
HE1EUR01FT024.eop-EUR01.prod.p rotection.outlook.com VI1PR04CA0068.outlook.office36 5.com Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_ 256_CBC_SHA384) * 5 12/12/2017
VI1PR04CA0068.eurprd04.prod.ou tlook.com VI1PR0401MB2302.eurprd04.prod. outlook.com Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_ 256_CBC_SHA384_P256) *Original Message Headers** REDACTED MESSAGE HEADERS **
Diagnostic-Code: smtp;550 5.7.23 The message was rejected because of Sender Policy Framework violation -> 550 5.7.1 Unauthenticated email from **REDACTED**.co.uk is not accepted due to;domain's DMARC policy. Please contact the administrator of;**REDACTED**.co.uk domain if this was a legitimate mail. Please;visit; https://support.google.com/mail/answer/2451690 to learn about the;DMARC initiative. d76si12884861pfk.321 - gsmtp
X-Display-Name: Samuel Melrose
On further inspection, the actual error is hidden inside the email:
550 5.7.23 The message was rejected because of Sender Policy Framework violation -> 550 5.7.1 Unauthenticated email from **removed**.co.uk is not accepted due to;domain's DMARC policy. Please contact the administrator of;**removed**.co.ukdomain if this was a legitimate mail. Please;visit; https://support.After some help from Google's G Suite support (fabulous by the way, quick responses, helpful and extremely knowledgeable right from the off), we figured out that the issue was that the DKIM signatures are being broken as the messages transit Office365 and our 3rd party has a strict DMARC policy which requires all messages are validated by the receiving party for a valid signature.
google.com/mail/answer/2451690 to learn about the;DMARC initiative. d76si12884861pfk.321 - gsmtp
This only happens on multi-part messages with an attachment, as Microsoft are altering the message somehow and the problem with that, as pointed out by G Suite support, is that most of the body is part of the signature hash, so by altering it the signature hash no longer matches and the integrity of the message is compromised according to DKIM.
While exploring the issue, we looked at the message that was successfully delievered to an Office365 user and the message body was also altered there too, but in a different way. Headers inside the multi-part sections had been added to do with spam handling, as well as given a Content-ID, etc.
The SPAM headers look like this:
X-Microsoft-Exchange-Diagnostics: **REDACTED BASE64 DATA**
X-Microsoft-Antispam-Mailbox-Delivery: **REDACTED STRING**
X-Microsoft-Antispam-Message-Info: **REDACTED BASE64 DATA**
To get to the bottom of the issue, I needed to play with the message until the signature did come back valid, to prove what the problem was, so I dropped the .eml files as .txt on a Linux box and got a copy of "dkimpy-0.6.2/dkimverify.py" to validate the messages on the command line.
It turns out, that hidden in the diff by the fact the multi-part boundary strings have changed, Office365 is adding a newline to the email between the boundaries, which is being included in the part of the body that is verified.
The additional headers, the changes to the multi-part boundary strings and even the headers inside the multi-part sections themselves don't seem to affect the signature; it's just the newline.
As you can see here, example diff from the email with attachment:
24,25c161,163Compared to the email without the attachment:
26,27c88,89And a quick test showed that had fixed the validation:
[email protected]:-$ python dkimpy-0.6.2/dkimverify.py < broken_email_received.txtSo far, Microsoft support haven't understood the problem at all.
signature verification failed
[email protected]:-$ cp broken_email_received.txt br_test.txt; vi br_test.txt
[ edited the file and removed the newline ]
[email protected]:-$ python dkimpy-0.6.2/dkimverify.py < br_test.txt
After emailing them over the details of what I've been investigating, they insisted on calling me to discuss it over the phone, only to get confused and keep telling me over and over again to alter the SPF records.
I reached out to [email protected], who emailed back after a few hours to say if it wasn't a RCE or MITM, they weren't interested.
My last attempt is to teach out to @Office365 on Twitter, who I will be sending a link to this article, so I'll update how that goes.