I ran into an issue where mail from my server/domain ended up in Gmail users’ spam folders and so endeavored to resolve it. I didn’t find anything in my online searching where someone was seeing exactly what I was seeing.

This link was helpful for general information around SPF: http://www.openspf.org/FAQ/Common_mistakes

For purposes of this documentation, configuration/values used:

Server IP : 192.168.1.1
Server hostname : host.domain.com (yes, different than example.com domain)
SMTP HELO (exim4) name : mail.example.com

Personal address: user@example.com
Gmail address: user@gmail.com

As for the symptoms, most importantly, email was ending up in Gmail users’ spam folders. In viewing the email header, I could see the reason was a softfail:

Received: ⁨from mail.example.com (host.domain.com. [2001:61f5:41:82c::235]) by mx.google.com with ESMTPS id u26si4316688wrd.422.2017.10.26.12.05.13 for <user@gmail.com> (version=TLS1_2 cipher=AES128-SHA bits=128/128); Thu, 26 Oct 2017 12:05:14 -0700 (PDT)⁩
Authentication-Results: ⁨mx.google.com; spf=softfail (google.com: domain of transitioning user@example.com does not designate 2001:61f5:41:82c::235 as permitted sender) smtp.mailfrom=user@example.com

Now I never did find out (neither from hosting provider nor in my online searching) why Google is printing what appears to be an IPv6 (hexadecimal) address in the response (generally, you should see the IPv4 address of the server instead) log it adds to the header, but I did get the gist that the host was not explicitly authorized to send mail on behalf of example.com.

I’m not going to go into detail about SPF usage (see link at beginning of post), but when sending email from a domain, a SPF DNS entry is required for that domain so that receiving servers can validate email by linking the sending machine to the domain (the SPF entry defines which machines can send mail on behalf of the domain). A single entry may contain multiple rules, separated by spaces, which are read sequentially until one is satisfied. If none are satisfied, the “*all” rule at the end directs the calling server how/if to fail the email message.

At the time I was encountering this issue, the SPF entry for my domain looked like:

example.com IN TXT "v=spf1 ip4:192.168.1.1 a:mail.example.com ~all”

My impression was that specifying the server IP address here (from which email will be sent) would satisfy ISP checks: all emails sent from 192.168.1.1 would be caught and validated by the rule “ip4:192.168.1.1,” and both mail.example.com (HELO name) and host.domain.com (host name) resolve to 192.168.1.1. Evidently, this was not the case. I viewed the addition of “a:mail.example.com,” seen above, as extraneous but worth a shot (read: desperation). Still, no dice. Note: the ‘~’ part of ‘~all’ directs a server to softfail a message if no rules in the entry are satisfied, as opposed to ‘-all’, which directs a server to hardfail.

I initially suspected Google was performing a HELO/EHLO (hereon just “HELO”) check that was failing due to no SPF entry for mail.example.com. Indeed, to satisfy servers which employ 100% HELO checks (or those scenarios where Mail from is empty in the message), a separate SPF entry is required for the HELO name itself (this is a best practice, though I’m not sure how often this check is employed). And so I added an appropriate DNS entry:

mail.example.us IN TXT "v=spf1 a -all"

In other words, explicitly allow email from this HELO FQDN (via the “a” rule), mail.example.com. This resulted in no change; I was still encountering the softfail.

Next I homed on the fact that, in addition to reporting the HELO name, Google was printing my actual hostname in the email header:

Received: ⁨from mail.example.com (host.domain.com. [2001:61f5:41:82c::235]) by mx.google.com with ESMTPS id u26si4316688wrd.422.2017.10.26.12.05.13 for <user@gmail.com> (version=TLS1_2 cipher=AES128-SHA bits=128/128); Thu, 26 Oct 2017 12:05:14 -0700 (PDT)⁩

See that host.domain.com? That’s the hostname of my server. More specifically, it’s the name associated (via rDNS) with IP 192.168.1.1. Google must be performing a reverse DNS lookup to retrieve that hostname. I wondered if it was then performing an SPF check based on that name, so I added a discrete rule for it in my SPF entry:

example.com IN TXT "v=spf1 ip4:192.168.1.1 a:host.domain.com ~all”

This fixed it:

Received: ⁨from mail.example.us (host.domain.com. [2001:61f5:41:82c::235]) by mx.google.com with ESMTPS id p19si797419wrf.42.2017.11.01.08.37.26 for <user@gmail.com> (version=TLS1_2 cipher=AES128-SHA bits=128/128); Wed, 01 Nov 2017 08:37:26 -0700 (PDT)⁩
Authentication-Results: ⁨mx.google.com; spf=pass (google.com: domain of user@example.com designates 2001:61f5:41:82c::235 as permitted sender) smtp.mailfrom=user@example.com⁩

Success! Still not sure what’s up with the IPv6 address, but success, nonetheless.

Upon further investigation (i.e., empirical testing), it appears that Google was not using a HELO check at all in this scenario, which I suppose isn’t to say it never does or never will.

So for Gmail, it looks like the most important thing is for that domain name resolving via rDNS (in my case, host.domain.com) to be present as explicitly ‘allowed’ in the SPF entry instead of relying on the IPv4 rule alone. Whether this would be required if the hostname and HELO name are the same, I don’t know. And whether that IPv6 address getting returned instead of IPv4 has anything to do with it, also not sure, but I’m eyeing that warily and have reached out to my hosting provider.

In summary:
– To satisfy Google (and other ISPs performing check via rDNS) spam check, add rule for server hostname (more specifically, the FQDN returned from rDNS of server IP) as explicitly allowable sender in SPF record for any and all domains from which mail will be sent.
– Additional best practice (not sure how often/ever this is implemented by an email server): and add new SPF entry for HELO name (e.g., “v=spf1 a -all”).