Broken DKIM Signatures on Office365

It always came across funny before when other people where having bad experiences with Microsoft support for Office365, leaving us thinking thank god that isn't us!

Now we've found our own issue and wow, was that account accurate.
The support so far has been abysmal!
I almost thought I was taking part in the new series of Channel 4's FoneJacker.

As a bit of background, we have split delivery with our MX pointing at Office365 and email for a select few users being sent to G Suite via a mail flow rule and an Outbound Gateway configured to point to Gmail.

We first noticed that delivery of messages with attachments from one of our 3rd party suppliers wasn't making it to the users on Gmail, but was making it to the users on Office365.

To make matters worse, the 3rd party supplier was getting a very misleading bounce message about SPF records, at first glance instructing them to add "spf.protection.outlook.com" to their SPF.

Your message to sam.m****@a1comms.com couldn't be delivered.

a1comms.com couldn't confirm that your message was sent from a trusted location.

****.****** Office 365 sam.m**** Action Required Recipient SPF validation error

How to Fix It Your organization's email admin will have to diagnose and fix your domain's email settings. Please forward this message to your email admin.

More Info for Email Admins

Status code: 550 5.7.23 This error occurs when Sender Policy Framework (SPF) validation for the sender's domain fails. If you're the sender's email admin, make sure the SPF records for your domain at your domain registrar are set up correctly. Office 365 supports only one SPF record (a TXT record that defines SPF) for your domain. Include the following domain name: spf.protection.outlook.com. If you have a hybrid configuration (some mailboxes in the cloud, and some mailboxes on premises) or if you're an Exchange Online Protection standalone customer, add the outbound IP address of your on-premises servers to the TXT record. For more information and instructions about configuring SPF records see Customize an SPF record to validate outbound mail sent from your domain and also External Domain Name System records for Office 365.

Original Message Details

Created Date: 12/12/2017 2:10:29 PM Sender Address: ****.****@cloud****.co.uk Recipient Address: sam.m****@a1comms.com Subject: Re: FW: **REDACTED**

Error Details

Reported error: 550 5.7.23 The message was rejected because of Sender Policy Framework violation -> 550 5.7.1 Unauthenticated email from **REDACTED**.co.uk is not accepted due to;domain's DMARC policy. Please contact the administrator of;**REDACTED**.co.ukdomain if this was a legitimate mail. Please;visit; https://support.google.com/mail/answer/2451690 to learn about the;DMARC initiative. d76si12884861pfk.321 - gsmtp DSN generated by: VI1PR0401MB2302.eurprd04.prod.outlook.com Remote server: mx.google.com

Message Hops
HOP	TIME (UTC)	FROM	TO	WITH	RELAY TIME
1	12/12/2017 2:10:29 PM		10.200.17.150	HTTP	*
2	12/12/2017 2:10:48 PM		mail-qt0-f169.google.com	SMTP	19 sec
3	12/12/2017 2:10:48 PM	mail-qt0-f169.google.com	HE1EUR01FT024.mail.protection.outlook.com	Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384)	*
4	12/12/2017 2:10:48 PM	HE1EUR01FT024.eop-EUR01.prod.protection.outlook.com	VI1PR04CA0068.outlook.office365.com	Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384)	*
5	12/12/2017 2:10:48 PM	VI1PR04CA0068.eurprd04.prod.outlook.com	VI1PR0401MB2302.eurprd04.prod.outlook.com	Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256)	*

Original Message Headers

** REDACTED MESSAGE HEADERS **

Final-Recipient: rfc822;sam.m****@a1comms.com
Action: failed
Status: 5.7.23
Diagnostic-Code: smtp;550 5.7.23 The message was rejected because of Sender Policy Framework violation -> 550 5.7.1 Unauthenticated email from **REDACTED**.co.uk is not accepted due to;domain's DMARC policy. Please contact the administrator of;**REDACTED**.co.uk domain if this was a legitimate mail. Please;visit; https://support.google.com/mail/answer/2451690 to learn about the;DMARC initiative. d76si12884861pfk.321 - gsmtp
Remote-MTA: dns;mx.google.com
X-Display-Name: Samuel Melrose

On further inspection, the actual error is hidden inside the email:

550 5.7.23 The message was rejected because of Sender Policy Framework violation -> 550 5.7.1 Unauthenticated email from **removed**.co.uk is not accepted due to;domain's DMARC policy. Please contact the administrator of;**removed**.co.ukdomain if this was a legitimate mail. Please;visit; https://support.google.com/mail/answer/2451690 to learn about the;DMARC initiative. d76si12884861pfk.321 - gsmtp

After some help from Google's G Suite support (fabulous by the way, quick responses, helpful and extremely knowledgeable right from the off), we figured out that the issue was that the DKIM signatures are being broken as the messages transit Office365 and our 3rd party has a strict DMARC policy which requires all messages are validated by the receiving party for a valid signature.

This only happens on multi-part messages with an attachment, as Microsoft are altering the message somehow and the problem with that, as pointed out by G Suite support, is that most of the body is part of the signature hash, so by altering it the signature hash no longer matches and the integrity of the message is compromised according to DKIM.

While exploring the issue, we looked at the message that was successfully delievered to an Office365 user and the message body was also altered there too, but in a different way. Headers inside the multi-part sections had been added to do with spam handling, as well as given a Content-ID, etc.

The SPAM headers look like this:

X-Microsoft-Exchange-Diagnostics: **REDACTED BASE64 DATA**
X-Microsoft-Antispam-Mailbox-Delivery: **REDACTED STRING**
X-Microsoft-Antispam-Message-Info: **REDACTED BASE64 DATA**
Content-ID: <**REDACTED**@eurprd04.prod.outlook.com>

On trying to re-create the issue, I sent an email from a gmail.com address to one of the accounts that is using split delivery, so I could see the issue from both sides. This identified something else, that Office365 is altering the multi-part boundry strings as it forwards on the message.

To get to the bottom of the issue, I needed to play with the message until the signature did come back valid, to prove what the problem was, so I dropped the .eml files as .txt on a Linux box and got a copy of "dkimpy-0.6.2/dkimverify.py" to validate the messages on the command line.

It turns out, that hidden in the diff by the fact the multi-part boundary strings have changed, Office365 is adding a newline to the email between the boundaries, which is being included in the part of the body that is verified.

The additional headers, the changes to the multi-part boundary strings and even the headers inside the multi-part sections themselves don't seem to affect the signature; it's just the newline.

As you can see here, example diff from the email with attachment:

24,25c161,163
< --001a114733c035484b056025d07f--
< --001a114733c0354850056025d081
---
> --001a114b306e43341b056025d01a--
>
> --001a114b306e43341e056025d01c

Compared to the email without the attachment:

26,27c88,89
< --001a1148e7ba4a6e8b05604cdd8c--
< --001a1148e7ba4a6e9105604cdd8e
---
> --001a11468748570c3605604cdd36--
> --001a11468748570c3a05604cdd38

And a quick test showed that had fixed the validation:

[email protected]:-$ python dkimpy-0.6.2/dkimverify.py < broken_email_received.txt
signature verification failed

[email protected]:-$ cp broken_email_received.txt br_test.txt; vi br_test.txt
[ edited the file and removed the newline ]

[email protected]:-$ python dkimpy-0.6.2/dkimverify.py < br_test.txt
signature ok

So far, Microsoft support haven't understood the problem at all.

After emailing them over the details of what I've been investigating, they insisted on calling me to discuss it over the phone, only to get confused and keep telling me over and over again to alter the SPF records.

I reached out to [email protected], who emailed back after a few hours to say if it wasn't a RCE or MITM, they weren't interested.

My last attempt is to teach out to @Office365 on Twitter, who I will be sending a link to this article, so I'll update how that goes.