SMTP flow different for Send Test Message and Contact Page

michaelboatright · September 14, 2024, 3:08pm

@ben - I am starting a new Topic related, but it looks like different from my email issues, although it is also an email issue.

My understanding is that Backlight → Admin → Special Links → Sent Test Email should make the same SMTP call, RIGHT? It doesn’t look it is doing so.

If I have my site set up as shown, I can successfully send a test message using Backlight → Admin → Special Links → Sent Test Email. The message gets flagged as spam and quarantined by Microsoft Outlook/Exchange server but that is a separate issue I need to work out with Microsoft.

However, with the EXACT same configuration, if I use a contact form to send the message, while I get a confirmation message, I see no evidence that the message is actually sent to the SMTP server.

Send Test Message
Message 0005 20240914 09:13 Send Test Email/Backlight

Email Configuration

Site Email
f64@michaelboatright.com

Mail Send Type
smtp
SMTP Host mail.mikeboatright.com
SMTP Username sendmail@mikeboatright.com
SMTP Password
[ hidden ]
SMTP Port
465
SMTP Security Protocol
ssl

In this case, the message is sent from the smtp server to Microsoft, routed by my MX records:

Message sent to Microsoft Exchange
Message quarantined by Microsoft:

(parsed message header)

Received: from BN9PR03CA0581.namprd03.prod.outlook.com (2603:10b6:408:10d::16)

by MW4PR20MB5177.namprd20.prod.outlook.com (2603:10b6:303:1e9::10) with

Microsoft SMTP Server (version=TLS1_2,

cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7962.21; Sat, 14 Sep

2024 13:25:43 +0000

Received: from BN1PEPF00005FFE.namprd05.prod.outlook.com

(2603:10b6:408:10d:cafe::a2) by BN9PR03CA0581.outlook.office365.com

(2603:10b6:408:10d::16) with Microsoft SMTP Server (version=TLS1_2,

cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7939.24 via Frontend

Transport; Sat, 14 Sep 2024 13:25:43 +0000

Authentication-Results: spf=fail (sender IP is 50.63.11.72)

smtp.mailfrom=michaelboatright.com; dkim=none (message not signed)

header.d=none;dmarc=fail action=quarantine

header.from=michaelboatright.com;compauth=fail reason=000

Received-SPF: Fail (protection.outlook.com: domain of michaelboatright.com

does not designate 50.63.11.72 as permitted sender)

receiver=protection.outlook.com; client-ip=50.63.11.72;

helo=p3nlsmtp01-05.prod.phx3.secureserver.net;

Received: from p3nlsmtp01-05.prod.phx3.secureserver.net (50.63.11.72) by

BN1PEPF00005FFE.mail.protection.outlook.com (10.167.243.230) with Microsoft

SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.7918.13

via Frontend Transport; Sat, 14 Sep 2024 13:25:43 +0000

Received: from p3plzcpnl507520.prod.phx3.secureserver.net ([132.148.178.109])

by : HOSTING RELAY : with ESMTPS

id pSlUsfWv9SfrapSlVsHkX9; Sat, 14 Sep 2024 06:24:41 -0700

X-CMAE-Analysis: v=2.4 cv=Ws7gMsfv c=1 sm=1 tr=0 ts=66e58e99

a=LaqYqv29eB/rMBfUcyhO9g==:117 a=RUM0BeRAcIrqMunYA3KyVg==:17

a=EaEq8P2WXUwA:10 a=RihNTfNFAAAA:8 a=m1f1a3DxAAAA:8 a=KVhygYe4wm9h7nsHNEYA:9

a=R0IRZARYTUsA:10 a=ZXulRonScM0A:10 a=dm1I0ddMwPOaf7S6yjKE:22

a=VhXK4N_Ophck9J-WlG__:22 a=B0QbiScC9Q_YWS3Ngr6m:22

X-SECURESERVER-ACCT: sendmail@mikeboatright.com

Received: from [107.180.51.12] (port=44314 helo=www.michaelboatright.com)

by p3plzcpnl507520.prod.phx3.secureserver.net with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384

(Exim 4.96.2)

(envelope-from f64@michaelboatright.com)

id 1spSlO-00FPjv-27

for f64@michaelboatright.com;

Sat, 14 Sep 2024 06:24:40 -0700

Date: Sat, 14 Sep 2024 13:24:33 +0000

To: Michael Boatright Photography f64@michaelboatright.com

From: Michael Boatright Photography f64@michaelboatright.com

Reply-To: Michael Boatright Photography f64@michaelboatright.com

Subject: Message 0005 20240914 09:24

Message-ID: 6c2ca9a5903510b00a8a7b005c322d57@www.michaelboatright.com

X-Priority: 3

X-Mailer: PHPMailer (phpmailer.sourceforge.net) [version 2.0.4]

MIME-Version: 1.0

Content-Type: text/plain

X-AntiAbuse: This header was added to track abuse, please include it with any abuse report

X-AntiAbuse: Primary Hostname - p3plzcpnl507520.prod.phx3.secureserver.net

X-AntiAbuse: Original Domain - michaelboatright.com

X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]

X-AntiAbuse: Sender Address Domain - michaelboatright.com

X-Get-Message-Sender-Via: p3plzcpnl507520.prod.phx3.secureserver.net: authenticated_id: sendmail@mikeboatright.com

X-Authenticated-Sender: p3plzcpnl507520.prod.phx3.secureserver.net: sendmail@mikeboatright.com

X-Source:

X-Source-Args:

X-Source-Dir:

X-CMAE-Envelope: MS4xfE6k4+D8+XEqore8NWnKHsPWWfHcW/T+V9lug2QTlmvBhcgRj4zH5AFBLgtIuv58whFtDlfuNrhLEmMg5X6uLO0swlqBkrLhJaQs4/siJMWGd0Dsg+ThZiS

OjZ+DBBb4f6KsHcQakZta1q2m/OmfX7tZXdCEQnii9bkUS5ImcEh7vwPrvOofqOqyxTJk1SGSyVEbzta+zm9WcdCBwPnzgL/Z02INKGFhcDtlFgEaAuR8kw/

05jToda4UF3r+lbT/AoPQA==

Return-Path: f64@michaelboatright.com

X-EOPAttributedMessage: 0

X-EOPTenantAttributedMessage: 508b66d3-0bf4-492a+D4-b0d8-550eb5409373:0

X-MS-PublicTrafficType: Email

X-MS-TrafficTypeDiagnostic: BN1PEPF00005FFE:EE_

MW4PR20MB5177:EE_

X-MS-Office365-Filtering-Correlation-Id: be619783-efff-4c39-1d55-08dcd4c0bbfd

X-Forefront-Antispam-Report:

CIP:50.63.11.72;CTRY:US;LANG:en;SCL:5;SRV:;IPV:NLI;SFV:SPM;H:p3nlsmtp01-05.prod.phx3.secureserver.net;PTR:p3nlsmtp01-05.prod.phx3.secureserver.net;CAT:SPOOF;SFS:(13230040);DIR:INB;

X-Microsoft-Antispam: BCL:0;ARA:13230040;
"

However, if I Use Contact Form (Contact_Page_NoIndex - Michael Boatright Photography)

I get a Response from Backlight, but NOTHING ELSE

image2402×886 176 KB

I see no evidence at all that anything was sent through the sendmail server. Nothing actually gets sent (at least from the server). This seems like a bug since we are actually getting a different behavior from one flow (send test email) and another (contact form) that essentially have the same config and message contents.

Ben · September 14, 2024, 11:08pm

The test email function uses the same mail settings as the contact form.

When sending you emails from the contact form:

	$emailHelper = new EmailHelper();
	$emailHelper->transport(__c('MAIL_SEND_TYPE'));

	$emailHelper->subject(__t('CONTACT_EMAIL_SUBJECT'));
	$emailHelper->plainTextMessage($plainMessage);
	$emailHelper->htmlMessage($htmlMessage);
	$emailHelper->from(__c('COMPANY_NAME'), __c('VENDOR_EMAIL_EMAIL'));
	$emailHelper->to(__c('COMPANY_NAME'), __c('SITE_EMAIL'));
	$emailHelper->replyTo($name, $email);
	$emailHelper->send();

When sending emails with the test email function:

	$emailHelper = new EmailHelper();
	$emailHelper->transport(__c('MAIL_SEND_TYPE'));

	$emailHelper->subject($email['subject']);
	$emailHelper->plainTextMessage($email['body']);
	$emailHelper->htmlMessage($email['body']);
	$emailHelper->from(__c('COMPANY_NAME'), $email['fromEmail']);
	$emailHelper->to(__c('COMPANY_NAME'), $email['toEmail']);

	if ($email['debugEmail'] == 'yes') {
		$emailHelper->setDebug(true);
	}

	$result = $emailHelper->send();

The email content can differ between the two, being the subject, from and to addresses. The behaviour can differ from that. Do you know whether your test from and to addresses match the Vendor and Site email settings?

michaelboatright · September 15, 2024, 1:08am

Ben,
Yes, the information is the same. In the test email, I did not change anything except for the Subject, Message Body and Show Debug Messages.

Unfortunately, there doesn’t seem to be a way to turn Show Debug Messages on for the contact form. I cannot tell if it is even calling sendmail at all, or with what or why no email gets sent to Microsoft Exchange ( only that with essentially the same inputs, the test email connects to the smtp server and a messaged is delivered (and subsequently quarantined) by the Microsoft Exchange Server.

Can you tell me what the actual differences are between the two sends, other than $emailHelper->setDebug(true); ?

Ben · September 15, 2024, 9:31am

There shouldn’t be any differences then. You could try adding the debug line for the contact form to see if there’s anything different in the debug output.

That’s in the file backlight/modules/module-admin/application/delegates/ContactDelegate.php

michaelboatright · September 15, 2024, 1:47pm

@ben FOUND IT!

There is one minor difference, and it makes all the difference in the world. In ContactDelegate.php, there is this line of code (line 177) that doesn’t have an equivalent in the Test Message path:

$emailHelper->replyTo($name, $email);

This sets the replyTo to be the name and email of the person submitting the Contact. Since the replyTo address is different from the from address, it is flagged as SPAM. It never even made it out of my SMTP server before getting rejected.

How I figured this out has to do with SMTP authentication (and since GMAIL is starting to require authentication as are most). When Microsoft receives an email from outside its domain with the sendTo and from and I presume replyTo, it has a hard and fast rule, that cannot even be overridden to quarantine the email as most probably spam.

In addition, GoDaddy requires that there be an email account on the SMTP server in order to authenticate to it (GMAIL does or is about to have the same rule). But I couldn’t use my email address because it is already hosted by REAL Microsoft (as opposed to the “pretend” Microsoft that Godaddy has implented…when they first did that, something didn’t smell right and I opted to pay the same amount of $$$ to Microsoft…not may favorite company in the world, but sometimes you have to choose between pour dance partners), there is an MX record on my server sending everything to my email address (SITE_EMAIL in Backlight). That and lord knows what else, made it impossible (we did try) to define an email, but their system creates a system mailbox based on the domain name…it’s complicated.

All this played out in the previous Topic that I opened up. And the work around was that because I had another domain (mikeboatright dotcom) in GD’s DNS, pointing to my server (a LONG time, like a couple of decades ago that was my host url), we were able to create a small (250mb) mailbox on the GD MTA on my server…sendmail at mikeboatright dot com.

This morning, I changed the VENDOR_EMAIL_EMAIL to sendmail at mikeboatright dot com.

And then sent a SPECIAL LINKS test message:

Not only did this message sail through the GD SMTP server, once it got to Microsoft, it was not quarantined! It was delivered to my Outlook mailbox! Huzzah! I was on to something.

So then, I created a contact with $email = sendmail at mikeboatright dot com

Again, it sailed through both GD SMTP and Microsoft Defender (spambot) and was delivered to my Outlook mailbox.

To verify this was the problem, I then sent another email with a $email as something besides VENDOR_EMAIL. This one failed the spam test at the GoDaddy SMTP server and was not forwarded to Microsoft.

So THEN I went into ContactDelegate.php, commented out line 177 where returnTo was being set with $name and $email and force returnTo to COMPANY_NAME and VENDOR_EMAIL_EMAIL:

I then sent a new contact with $email as something besides VENDOR_EMAIL. This sailed through the GD SMTP server, with a returnTo matching the from (both sendmail at mikeboatright dot com) and and Microsoft Defender (spambot) and was delivered to my Outlook mailbox.

On last question…somehow I managed to get the Send acknowledgement bunged up in ContactDelegate.php. I’m going to try to figure out how, but if off the top of your head, you remember where that’s set, I’d appreciate a heads up as to where to look. I probably accidentally hit a space or something while in the cPanel edit window looking at ContactDelegate.php.

michaelboatright · September 15, 2024, 1:57pm

A parenthetical comment here. I am certain that the ahah moment came AFTER figuring out how to set up RSS (much easier than I thought it would be) to publish contact messages…even though there is only a tangental connection between RSS and SMTP.

Ben · September 16, 2024, 10:46am

Hi @michaelboatright, some excellent detective work there!

If you need to restore a file to the original that we provide, then the easiest way is to visit the Backlight Modules page and reinstall the module that included that file.

That is interesting that the Reply To is checked. I had originally coded it this way (rather than attempting to set the From to be the user-supplied email address) to avoid emails being flagged as spam, or not even going out in the first place.

I had hopes of running tests to mail-tester.com with and without the Reply To set, but I can’t get any emails going out from my Google Apps account, or my Gmail account. I’m assuming that they no longer support ‘insecure apps’ as they call them. That’s a pain. As site owners, Backlight users can be deemed as trusted users for their own email service. Calling our implementation and configuration as “insecure” is like saying that we don’t trust ourselves (as Backlight users).

I will start looking into adding OAuth support for gmail/Google Apps. I don’t know whether that will benefit other users of other mail providers.

michaelboatright · September 16, 2024, 1:36pm

Thanks, Ben!

Yeah the GoDaddy hosting support rep yesterday (something else they did troubleshooting my problem broke FTP) kinda sat up in his chair when I told him I got my CompSci degree in 1982 (GA Tech BSICS COOP '82). You can take the geek out of programming, but you can’t take programming out of the geek.

Any chance of a silent update? Otherwise next update I’ll have to go and put my patch back in…

I don’t know about how many others are using this code path, but personally, I would never REPLY to the contact email to reach out to a potential customer. It’s rather tacky, IMHO. Not to mention that some might rather not have their TO address exposed like that. And it has become pretty common practice that if replyTo does not equal from, it’s spam. That’s in the first of 3 rules that Microsoft applies to all incoming email, and not even possible to turn off or change.

…And, as you assumed…that’s pretty much how you expected it to work, i.e., how Send Test Email works.

Thanks again!