[SOLVED] This message does not pass authentication checks (SPF and DKIM both)

mfoley · 07-25-2022, 06:32 PM

I am sending a message from the mail server 'mail' at domain mercureytech.com using mutt with .muttrc set to:

Code:

set envelope_from_address=them@thatdom.com
set use_envelope_from=yes

The email header is:

Code:

From them@thatdom.com Mon Jul 25 18:14:37 2022
Return-Path: <them@thatdom.com>
Received: from mail.mercureytech.com (localhost [127.0.0.1]) by mail.mercureytech.com (8.17.1/8.15.2) with ESMTP id 26PMEavS008018; Mon, 25 Jul 2022 18:14:3$
Received: (from daemon@localhost) by mail.mercureytech.com (8.17.1/8.17.1/Submit) id 26PMEaaQ008017; Mon, 25 Jul 2022 18:14:36 -0400
Date: Mon, 25 Jul 2022 18:14:36 -0400
From: Brian Smith <them@thatdom.com>
To: maillist@members.booga.org
Subject: Odd emails

The To: recipient is a dummy user that gets routed to /dev/null on receipt. The actual recipient is in the Bcc.

If the actual recipient is anyone at gmail.com, I get the following error:

Code:

   ----- The following addresses had permanent fatal errors -----
<user@gmail.com>
    (reason: 550-5.7.26 This message does not pass authentication checks (SPF and DKIM both)

   ----- Transcript of session follows -----
451 4.4.1 reply: read error from gmail-smtp-in.l.google.com.
... while talking to gmail-smtp-in.l.google.com.:
>>> DATA
<<< 550-5.7.26 This message does not pass authentication checks (SPF and DKIM both
<<< 550-5.7.26 do not pass). SPF check for [] does not pass with ip: [24.142.169.11
<<< 550-5.7.26 ].To best protect our users from spam, the message has been blocked.
<<< 550-5.7.26 Please visit
<<< 550-5.7.26  https://support.google.com/mail/answer/81126#authentication for more
<<< 550 5.7.26 information. j2-20020a05620a410200b006b58e95eaf2si10157318qko.720 - gsmtp
554 5.0.0 Service unavailable

Why? IP 24.142.169.11 is mercureytech.com and it has both DKIM and SPF records:

Code:

# dig +short TXT mercmail._domainkey.mercureytech.com
"v=DKIM1; k=rsa;  p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCwQ1ZH3L21R9ZA4SXV8Tc1E1Zii1 ...

and

Code:

# dig +short mercureytech.com txt
"google-site-verification=4s-Rnsi34vsqGbLivmLVS-lqYgBqK5T0siMfiDtwVOY"
"v=spf1 ip4:24.142.169.11"

(Notice that I even have the google-site-verification TXT record, which is supposed to help somehow).

So, why then am I getting this error? I do have DKIM and SPF records for the email server. Is google simply not allowing the From name to be set to a domain different than the email server's domain? This same messages sent to recipients other than gmail.com seem to work fine.

elgrandeperro · 07-25-2022, 11:55 PM

What is thatdom.com? That is the one that needs the SPF/DKIM record.

mfoley · 07-26-2022, 01:21 PM

Quote:

Originally Posted by elgrandeperro

What is thatdom.com? That is the one that needs the SPF/DKIM record.

thatdom.com is the "sender" of the message. it could be from any email/domain. The email is being forwarded from a sendmail alias script on host mail.mercureytech.com. The sender could be someone@gmail.com. I am using the .muttrc file and the EMAIL env variable to set the sender to someone@gmail.com. So, I have no control over the sender's host's SPF/DKIM records.

Interesting additional experiment ...

I copied several of the rejected email addresses from the mercureytech.com host to my personal server at novatec-inc.com. I ran the exact same mutt email on these addresses with the same .muttrc, same message body, etc.

Code:

EMAIL="Brian Smith <them@thatdom.com>" mutt -F ./muttrc -e 'set content_type="text/html"' -s "Odd emails" -b user@gmail.com maillist@members.booga.org < textfile2

All these emails were sent/delivered with no problem. There are a couple of difference between the mercureytech.com setup and the novatec-inc.com setup:

1) While both hosts' ISP is Spectrum, the mercureytech.com is a Business account with a static IP. The novatec-inc.com is a residential account with a dynamic IP. One would actually think the residential account would be more problematic, but no.

2) The novatec-inc.com SPF/DKIM records are as follows:

Code:

# DKIM:

$ dig +short TXT novatec._domainkey.novatec-inc.com
"v=DKIM1; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCsBGV18rrdOVD9xOirYNlF+JScTb4j4kuPVkMC+KLZTffv ..."

# SPF

$ dig +short novatec-inc.com txt
"v=spf1 ip4:184.57.48.3 ~all"
"google-site-verification=8pUMKzGWw5oxZSgjy1VO3yXb2-z5lDfREHje8YNlCr0"

# Mercureytech.com SPF

# dig +short mercureytech.com txt
"google-site-verification=4s-Rnsi34vsqGbLivmLVS-lqYgBqK5T0siMfiDtwVOY"
"v=spf1 ip4:24.142.169.11"

Notice that the novatec-inc.com SPF has the "~all" suffix whereas the mercureytech.com SPF does not. This is the only difference I can see between the SPF/DKIM records on the two servers. Is it possible that the "all" suffix is required?

Can anyone think of any other reason these emails would be OK to deliver from my personal, residential server and not from the static IP server? (mercureytech.com is not blacklisted)

mfoley · 07-26-2022, 03:57 PM

I added the "~all" to the mercureytech.com SPF record. That didn't help. It appears on the one hand that the sender domain needs the SPF/DKIM records connected to 24.142.169.11. That's the error I get from google. That raises two questions:

1) Is it then not possible to have a mutt "set envelope_from_address=them@thatdom.com" changing the sender address? (at least with gmail.com. The message get delivered w/o problem to all other email recipients)

2) On the other hand, I can send the exact same message with the same envelop_from_address to the same gmail recipient from my personal server and it gets delivered to gmail recipients without problem. Why? The SPF/DKIM records on the two servers are essentially identical. What could be the problem sending from mercureytech.com versus novatec-inc.com. I'm stumped!

elgrandeperro · 07-26-2022, 06:15 PM

Gmail is particularly stringent, other mailers will not even consult spf/dkim. So that is why those work.

Are you using both dkim keys in your mail config? It has to switch depending on the sender's email.

I know dkim-milter will do this, I think there is also a opendkim server that does it.

mfoley · 07-27-2022, 12:57 PM

Quote:

Originally Posted by elgrandeperro

Gmail is particularly stringent, other mailers will not even consult spf/dkim. So that is why those work.

Are you using both dkim keys in your mail config? It has to switch depending on the sender's email.

I know dkim-milter will do this, I think there is also a opendkim server that does it.

I am using opendkim milter.

What do you mean by "both dkim keys"? If I am sending from the mercureytech.com email server and the sender is someone@gmail.com, how would I reference gmail's SPF/DKIM records?

Also oddity is that I can send as someone@gmail.com from my email server at novatec-inc.com and gmail doesn't complain. Puzzling.

mfoley · 07-27-2022, 09:10 PM

I set envelope_from_address=mercureytech.com. Doing that did deliver the email to the gmail recipient, however it went to gmail's SPAM folder. Even after I marked the message as "not spam" in gmail, then sent another message, it also went to SPAM. So, gmail ignores my "not Spam" settings for this recipient for future emails.

Gmail gives no clues as to what is causing this message to be marked as SPAM from this domain (as mentioned several times, I can send the same thing from novatec-inc.com w/o problem).

Futhermore, it is quite an effort to see if there are any spam messages. You have to scroll down the gmail side-bar to "MORE >" (normally hidden 'beneath the fold') and click that, then you can scroll further down to SPAM. The spam box and number of messages in spam are not shown on the usual gmail webpage, so the user has to consciously want to check spam.

I think all this essentially renders this system useless for sending email to google recipients. I think google has quite overdone it for perfectly legitimate emails with all required DNS records. They have no contact for questions about this sort of thing that I can find. I'm at a dead end.

Does anyone have any other suggestions?

mfoley · 07-29-2022, 02:27 AM

Is it possible that Google is just wrong? I ran another simple test, no mutt, no envelope_from_address, no changing sender; just an email from my user on mercureytech.com:

Code:

echo some text | mail -s "Another Test" -b mfoley@novatec-inc.com mercureytech.com

I bcc'd my personal account in order to get the headers which are:

Code:

From mfoley@mail.mercureytech.com  Fri Jul 29 02:52:54 2022
Return-Path: <mfoley@mail.mercureytech.com>
Received: from mail.mercureytech.com (rrcs-24-142-169-11.mail.mercureytech.com [24.142.169.11] (may be forged))
        by server.novatec-inc.com (8.15.2/8.15.2) with ESMTPS id 26T6qrDW016430
        (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO)
        for <mfoley@novatec-inc.com>; Fri, 29 Jul 2022 02:52:53 -0400
Authentication-Results: server.novatec-inc.com;
        dkim=pass (1024-bit key) header.d=mercureytech.com header.i=@mercureytech.com header.b=qtFcXfOL
Received: from mail.mercureytech.com (localhost [127.0.0.1])
        by mail.mercureytech.com (8.17.1/8.15.2) with ESMTP id 26T6qm8o002943;
        Fri, 29 Jul 2022 02:52:49 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=mercureytech.com;
        s=mercmail; t=1659077570;
        bh=cVsk4knlD1G/l8bpiR56bHKhproDfhGUW/GN+irEXA0=;
        h=Date:From:To:Subject;
        b=qtFcXfOLJYAZ+UXQWKqk6eZRsOHwpPLmdsiV6ZRziMdN5pgN4fDpUnt0KMDwGxeVM
         m450UvHJYcrdO3u5ePumL+SSg5MxLe9oVv/lXBLAuULzhbQ7KWnUvchZnsCtcSiv+8
         IEj96ukoXn9HSH403CQldvKvLuN6N93dazVHuU20=
Received: (from mfoley@localhost)
        by mail.mercureytech.com (8.17.1/8.17.1/Submit) id 26T6qm0f002942;
        Fri, 29 Jul 2022 02:52:48 -0400

The message bounced back from Gmail with the following:

Code:

   ----- The following addresses had permanent fatal errors -----
<mercureytech@gmail.com>
    (reason: 550-5.7.25 [24.142.169.11] The IP address sending this message does not have a)

   ----- Transcript of session follows -----
... while talking to gmail-smtp-in.l.google.com.:
>>> DATA
<<< 550-5.7.25 [24.142.169.11] The IP address sending this message does not have a
<<< 550-5.7.25 PTR record setup, or the corresponding forward DNS entry does not
<<< 550-5.7.25 point to the sending IP. As a policy, Gmail does not accept messages
<<< 550-5.7.25 from IPs with missing PTR records. Please visit
<<< 550-5.7.25  https://support.google.com/mail/answer/81126#ip-practices for more
<<< 550 5.7.25 information. gu6-20020a056214260600b004736cd065e4si1910813qvb.459 - gsmtp
554 5.0.0 Service unavailable

Unless I'm mistaken (which I may be), the PTR record refers to the reverse DNS entry. For IP 24.142.169.11:

Code:

$ nslookup 24.142.169.11
11.169.142.24.in-addr.arpa      name = rrcs-24-142-169-11.mail.mercureytech.com.

Is this not the PTR record Google is looking for? What am I missing? The "may be forged" in my Received: header may be a clue, but I can't figure out why I get that and what's not matching.

TenTenths · 07-29-2022, 03:40 AM

This is a very handy site: https://www.appmaildev.com/en/dkim

When you click "Next" it gives you an e-mail address to send a mail to and it does a bunch of tests (spf/dkim) and can give advice on stuff.

mfoley · 07-30-2022, 11:38 AM

TenTenths - thanks for that site. That will come in very handy. I used it to check my email. It came back and said I had no SPF record. I did have an SPF record as shown in my post #3, above. However, the SPF was for mercureytech.com and the envelop header From was mfoley@mail.mercureytech.com. So, I added the following two records to sendmail.mc:

Code:

MASQUERADE_AS(`mercureytech.com')
FEATURE(`masquerade_envelope')

That changed the From envelop header to mfoley@mercureytech.com. Then the appmaildev.com site passed all: SFP, DKIM, DMARC and PTR (rDNS). Also, the message did get delivered to gmail, but went right to SPAM. In addition, the bcc of that message to my personal server still shows "(may be forged)".

So, progress, but still not quite there yet. Any idea what could still be causing the "may be forged" warning? According to a website,

Quote:

[The] Mail Transport Agent does a reverse hostname lookup of the IP address of the connecting client, and a lookup of the IP addresses associated with that hostname. If the client IP address does not appear in that list then the "may be forged" tag is added.

The reverse DNS lookup is:

Code:

$ host 24.142.169.11
11.169.142.24.in-addr.arpa domain name pointer rrcs-24-142-169-11.mail.mercureytech.com.

Perhaps this should be just rrcs-24-142-169-11.mercureytech.com (no mail.)? I have it as mail.mercureytech.com because Bathroy in post https://www.linuxquestions.org/quest...4/#post6356530 said:

Quote:

Originally Posted by bathory

Since you're asking for the rDNS of a mail server, it should point to the FQDN of the mail server.
I.e. if "dig mx mydom.com" points to blah.mydom.com, then the IP should resolve back to blah.mydom.com.

Also, I have "define(`confDOMAIN_NAME', `mail.mercureytech.com')" in sendmail.mc.

Should I change my rDNS and confDOMAIN_NAME to just mercureytech.com or leave one or both as is assuming I understand Bathroy's comment correctly?

Could the "may be forged" warning be because of something else entirely? What?

I can't really experiment at will with this as I have to have the ISP (Spectrum) set up the rDNS and I can't ask them to change it too often.

RSVP THX

mfoley · 08-01-2022, 06:44 PM

I think I've exhausted this thread and some of the original issues have been resolved thanks to the link provided by TenTenths. All the issues related to the mutt email client and SPF/DKIM errors from Google serve only to clutter and confuse. I'm going to post another thread simply asking about the "may be forged" issue. If I get that resolved, I'm hopeful that will take care of my remaining issue with email from this server going to gmail spam.

I've posted a new, simpler thread: https://www.linuxquestions.org/quest...47#post6371247

Thanks all for your help on this!