Broken DKIM Signatures on Office365
It always came across funny before when other people where having bad experiences with Microsoft support for Office365, leaving us thinking thank god that isn’t us!
Now we’ve found our own issue and wow, was that account accurate.
The support so far has been abysmal!
I almost thought I was taking part in the new series of Channel 4’s FoneJacker.
As a bit of background, we have split delivery with our MX pointing at Office365 and email for a select few users being sent to G Suite via a mail flow rule and an Outbound Gateway configured to point to Gmail.
We first noticed that delivery of messages with attachments from one of our 3rd party suppliers wasn’t making it to the users on Gmail, but was making it to the users on Office365.
To make matters worse, the 3rd party supplier was getting a very misleading bounce message about SPF records, at first glance instructing them to add “spf.protection.outlook.com” to their SPF.
Your message to sam.m****@a1comms.com couldn't be delivered. a1comms.com couldn't confirm that your message was sent from a trusted location.
****.****** Office 365 sam.m**** Action Required Recipient
SPF validation error
How to Fix It Your organization's email admin will have to diagnose and fix your domain's email settings. Please forward this message to your email admin.
More Info for Email AdminsStatus code: 550 5.7.23
This error occurs when Sender Policy Framework (SPF) validation for the sender's domain fails. If you're the sender's email admin, make sure the SPF records for your domain at your domain registrar are set up correctly. Office 365 supports only one SPF record (a TXT record that defines SPF) for your domain. Include the following domain name: spf.protection.outlook.com. If you have a hybrid configuration (some mailboxes in the cloud, and some mailboxes on premises) or if you're an Exchange Online Protection standalone customer, add the outbound IP address of your on-premises servers to the TXT record.
For more information and instructions about configuring SPF records see Customize an SPF record to validate outbound mail sent from your domain and also External Domain Name System records for Office 365.Original Message Details
Created Date: 12/12/2017 2:10:29 PM Sender Address: ****.****@cloud****.co .uk Recipient Address: sam.m****@a1comms.com Subject: Re: FW: **REDACTED**
Error Details
Reported error: 550 5.7.23 The message was rejected because of Sender Policy Framework violation -> 550 5.7.1 Unauthenticated email from **REDACTED**.co.uk is not accepted due to;domain's DMARC policy. Please contact the administrator of;**REDACTED**.co.ukdomain if this was a legitimate mail. Please;visit; https://support. google.com/mail/answer/2451690 to learn about the;DMARC initiative. d76si12884861pfk.321 - gsmtp DSN generated by: VI1PR0401MB2302.eurprd04.prod. outlook.com Remote server: mx.google.com
Message Hops HOP TIME (UTC) FROM TO WITH RELAY TIME 1 12/12/2017
2:10:29 PM10.200.17.150 HTTP * 2 12/12/2017
2:10:48 PMmail-qt0-f169.google.com SMTP 19 sec 3 12/12/2017
2:10:48 PMmail-qt0-f169.google.com HE1EUR01FT024.mail.protection. outlook.com Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_ 256_CBC_SHA_P384) * 4 12/12/2017
2:10:48 PMHE1EUR01FT024.eop-EUR01.prod.p rotection.outlook.com VI1PR04CA0068.outlook.office36 5.com Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_ 256_CBC_SHA384) * 5 12/12/2017
2:10:48 PMVI1PR04CA0068.eurprd04.prod.ou tlook.com VI1PR0401MB2302.eurprd04.prod. outlook.com Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_ 256_CBC_SHA384_P256) * Original Message Headers** REDACTED MESSAGE HEADERS **
Final-Recipient: rfc822;sam.m****@a1comms.com
Action: failed
Status: 5.7.23
Diagnostic-Code: smtp;550 5.7.23 The message was rejected because of Sender Policy Framework violation -> 550 5.7.1 Unauthenticated email from **REDACTED**.co.uk is not accepted due to;domain's DMARC policy. Please contact the administrator of;**REDACTED**.co.uk domain if this was a legitimate mail. Please;visit; https://support.google.com/mail/answer/2451690 to learn about the;DMARC initiative. d76si12884861pfk.321 - gsmtp
Remote-MTA: dns;mx.google.com
X-Display-Name: Samuel Melrose
On further inspection, the actual error is hidden inside the email:
550 5.7.23 The message was rejected because of Sender Policy Framework violation -> 550 5.7.1 Unauthenticated email from **removed**.co.uk is not accepted due to;domain’s DMARC policy. Please contact the administrator of;**removed**.co.ukdomain if this was a legitimate mail. Please;visit; https://support.google.com/mail/answer/2451690 to learn about the;DMARC initiative. d76si12884861pfk.321 - gsmtp
After some help from Google’s G Suite support (fabulous by the way, quick responses, helpful and extremely knowledgeable right from the off), we figured out that the issue was that the DKIM signatures are being broken as the messages transit Office365 and our 3rd party has a strict DMARC policy which requires all messages are validated by the receiving party for a valid signature.
This only happens on multi-part messages with an attachment, as Microsoft are altering the message somehow and the problem with that, as pointed out by G Suite support, is that most of the body is part of the signature hash, so by altering it the signature hash no longer matches and the integrity of the message is compromised according to DKIM.
While exploring the issue, we looked at the message that was successfully delievered to an Office365 user and the message body was also altered there too, but in a different way. Headers inside the multi-part sections had been added to do with spam handling, as well as given a Content-ID, etc.
The SPAM headers look like this:
X-Microsoft-Exchange-Diagnostics: **REDACTED BASE64 DATA**
X-Microsoft-Antispam-Mailbox-Delivery: **REDACTED STRING**
X-Microsoft-Antispam-Message-Info: **REDACTED BASE64 DATA**
Content-ID: <**REDACTED**@eurprd04.prod.outlook.com>
On trying to re-create the issue, I sent an email from a gmail.com address to one of the accounts that is using split delivery, so I could see the issue from both sides. This identified something else, that Office365 is altering the multi-part boundry strings as it forwards on the message.
To get to the bottom of the issue, I needed to play with the message until the signature did come back valid, to prove what the problem was, so I dropped the .eml files as .txt on a Linux box and got a copy of “dkimpy-0.6.2/dkimverify.py” to validate the messages on the command line.
It turns out, that hidden in the diff by the fact the multi-part boundary strings have changed, Office365 is adding a newline to the email between the boundaries, which is being included in the part of the body that is verified.
The additional headers, the changes to the multi-part boundary strings and even the headers inside the multi-part sections themselves don’t seem to affect the signature; it’s just the newline.
As you can see here, example diff from the email with attachment:
24,25c161,163
< --001a114733c035484b056025d07f--
< --001a114733c0354850056025d081
---
> --001a114b306e43341b056025d01a--
>
> --001a114b306e43341e056025d01c
Compared to the email without the attachment:
26,27c88,89
< --001a1148e7ba4a6e8b05604cdd8c--
< --001a1148e7ba4a6e9105604cdd8e
---
> --001a11468748570c3605604cdd36--
> --001a11468748570c3a05604cdd38
And a quick test showed that had fixed the validation:
smelrose@vdev2:-$ python dkimpy-0.6.2/dkimverify.py < broken_email_received.txt
signature verification failed
smelrose@vdev2:-$ cp broken_email_received.txt br_test.txt; vi br_test.txt
[ edited the file and removed the newline ]
smelrose@vdev2:-$ python dkimpy-0.6.2/dkimverify.py < br_test.txt
signature ok
So far, Microsoft support haven’t understood the problem at all.
After emailing them over the details of what I’ve been investigating, they insisted on calling me to discuss it over the phone, only to get confused and keep telling me over and over again to alter the SPF records.
I reached out to [email protected], who emailed back after a few hours to say if it wasn’t a RCE or MITM, they weren’t interested.
My last attempt is to teach out to @Office365 on Twitter, who I will be sending a link to this article, so I’ll update how that goes.