smtplib send_message should add Date header if it is missing, per RFC5322

[HenningvonBargen]

BPO	28879
Nosy	@bitdancer, @soltysh, @jenstroeger, @elafontaine
PRs	bpo-28879 : add date if missing in smtplib.send_message #2655 gh-73065: Add Date header if missing in smtplib send_message #5176
Files	issue_28879.patch issue_28879_V2.patch issue_28879_V3.patch issue_28879_V4.patch: Python3 complete changes issue_28879_python2.patch issue_28879_python2_overkill.patch issue28879_v5.patch Resent_heuristic.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2016-12-05.15:48:15.178>
labels = ['3.7', 'easy', 'type-bug', 'library']
title = 'smtplib send_message should add Date header if it is missing, per RFC5322'
updated_at = <Date 2018-09-08.23:03:27.435>
user = 'https://bugs.python.org/HenningvonBargen'

bugs.python.org fields:

activity = <Date 2018-09-08.23:03:27.435>
actor = '_savage'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2016-12-05.15:48:15.178>
creator = 'Henning.von.Bargen'
dependencies = []
files = ['45929', '45958', '45959', '45969', '46001', '46002', '46459', '46460']
hgrepos = []
issue_num = 28879
keywords = ['patch', 'easy']
message_count = 32.0
messages = ['282425', '282429', '282549', '282553', '282558', '283275', '283313', '283347', '283361', '283362', '283412', '283431', '283436', '283572', '283574', '283593', '283604', '283645', '283647', '283847', '283848', '283849', '285686', '285694', '285697', '286037', '286504', '286505', '288046', '297496', '297528', '324865']
nosy_count = 5.0
nosy_names = ['r.david.murray', 'maciej.szulik', 'Henning.von.Bargen', '_savage', 'Eric Lafontaine']
pr_nums = ['2655', '5176']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue28879'
versions = ['Python 3.6', 'Python 3.7']

Linked PRs

• gh-73065: Add Date header if missing in smtplib send_message #136850

[HenningvonBargen]

I'm using CPython 2.7 with the smtplib and email modules to send emails with SMTP.
Today, one of our clients complained that the email sent is not RFC 5322 compliant because the required Date header is missing. The RFC states in section 3.6.:

"The only required header fields are the origination date field and
the originator address field(s). All other header fields are
syntactically optional."

Our program has been sending millions of email message this way and this is the first complaint.

I guess that the library doesn't add the header field automatically.

[bitdancer]

That is correct. Most SMTP gateways add the header on submission if it is missing. At least a few other MUA programs do not automatically add the Date header, they let the SMTP server do it. I have one person who sends me email that saw this same problem (I had my server set to reject messages without the date header), and they weren't using smtplib to send. (I think the sender used Exchange, though I don't remember for sure.)

[HenningvonBargen]

I can give a little more information.
First, I created a very simple stand-alone test script (for Python >= 2.6):

#!/bin/env python
# -- coding: utf-8 --

import smtplib

# Adjust these!

HOST = "smtp.nowhere.local"
PORT = 25

from_name = u"Test Python smtplib"
from_addr = u"valid.address@nowhere.local"

to_name = u"Python Test Recipient"
to_addr = u"some.address@yourcompany.com"

subject = "Test Nr. 1"
smtp = smtplib.SMTP(HOST, PORT, timeout=10)
smtp.set_debuglevel(1)
smtp.sendmail(from_addr, [to_addr], subject)
smtp.quit()

This script shows the same behavior, which shows that the problem is not related to my program, but it is indeed a problem with smtplib.py.

Unfortunately, I could only test this with Python 2.6, because I'm not allowed to install Python 2.7 on that machine; OTOH I have to run it on that machine to have access to the mail server.

The observations are:

• The message is sent (I sent it to one of my email addresses).

• Spam Assassin flags the message with additional header lines: X-Amavis-Alert: BAD HEADER SECTION, Missing required header field: "Date"
  X-Spam-Status: No, score=-2.36 ...

• In some cases the message is silently dropped, because the missing date header caues a negative score; which in turn may be bad enough to classify it as spam, depending on the content.

• The SMTP server used here is Postfix.

• When I receive the message, a Date header is present (obviously it has been inserted during the message's journey through the internet).

Personal opinion:
I think classifying a message as spam just because a practically useless header field is missing is bad behavior of Spam Assaassin.
Nevertheless, the standard library should try to conform to RFC 5322.
It is good practise to be forgiving while reading but pedantic while writing.

[bitdancer]

As I implied but did not say explicitly, this is the expected behavior of smtplib. You are responsible for adding any headers to the message that you want smtplib to send. In particular, the 'sendmail' method takes a string to send, and smtplib does not modify it except for cr/lf transformation. smtplib does not itself understand RFC5322 message syntax.

We can, however, add a Date header in the new send_message method of the python3 email library, because it accepts a Message object, and smtplib can use the knowledge the Message object encapsulates to check for the Date header and add one if it is missing. That's why I've left this issue open. I have now adjusted versions accordingly (ie: this is not a bug in python2.7, it is an enhancement request for Python3). Sorry I wasn't clear about this earlier.

Hmm. Actually, we can argue that it is an RFC compliance issue, as you have suggested, and change it in 3.6 as well, since it isn't likely to break anyone's working code. So I'll put 3.6 in the versions unless someone objects to that logic. But even after this is changed in python3, the smtplib sendmail method will not add a Date header, only the send_message method.

[elafontaine]

Hi, Not sure this is where the comment goes...

I work with the smtplib and email libraries. I understand Henning von Bargen when he say that we should have a way to support RFC 5322 without asking the user to understand how to support it. The issue is that the SMTP protocol is NOT the protocol that format the e-mail. SMTP is the protocol that identify the "from", the "to", start encryption and finally transfert the message. The actual e-mail content is all passed inside the SMTP "DATA" Command. I strongly believe that an email should not be modified by a SMTP library.

the discussion should be focused on trying to make it available to the user WITHOUT changing the current behavior of email.message class.

In other words, I disagree to change the SMTPlib module and suggest that it's how you construct your email in the first place that should consider it;
class MessageRfc5322(email.message.Message):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
if self.get('Date', None) is None:
self.add_header('Date', email.utils.formatdate())

msg = email.message_from_string(string_message, MessageRfc5322)

But, that's my opinion as someone who uses the smtplib and email library but also need to support rfc822 clients...

[soltysh]

I tend to agree with Eric Lafontaine, looking at the quote Henning von Bargen posted the originator address field is also required, but yet we don't explicitly check its presence in the code, but rely on the SMTP server to error out.

[bitdancer]

The sendmail function will never modify the RFC822+ content. send_message, however, already does several manipulations of the message headers to make sending email simpler. Practicality (make it easy to send messages without knowing the details of the SMTP/RFC5322 rules) beats purity (an SMTP library should not modify the content of the DATA) in this case, especially since smtplib *does* provide the purity version if you prefer to work that way.

That is, 'sendmail' is pure SMTP, while send_message is a practical enhancement that provides additional services to the caller, and can (and does and probably should) do checks for RFC required headers.

If someone wants to do a purity refactor (to which I would not object, and would in fact encourage), the RFC5322-aware code could be factored out into one or more functions or Message object methods in the email library that smtplib would call from the send_message method.

My visualization is that the email library should allow you to construct a valid email in whatever order you want (adding the Date header late in the process, for example), but should support validating the email before it is sent. One way to do this would be to have the SMTP policy do the unambiguous fixups such as date headers, and raising errors for the rest (probably only if the strict flag is on, at this point) when the message is serialized.

Note that we previously fixed send_message to add a Resent-Date header if there are Resent- headers and no Resent-Date, so the precedent is already set (that is, smptlib send_message is already "not pure" :)

[soltysh]

I've chatted a bit with David about this feature. Here are some thoughts:

• check what SMTP standard says about some validation rules
• add validate method, probably into email package

[elafontaine]

Hi all,

Thanks for the enlightment. I never figured that there was a send_message function XD. Never needed it and it's true that the example in the email library use sendmail and not send_message.
https://docs.python.org/2/library/smtplib.html#smtplib.SMTP.sendmail
https://docs.python.org/3.5/library/smtplib.html#smtplib.SMTP.send_message

This function is fairly recent (python 3.2) from what I see.

Reading the documentation of the python 3.5 send_message function :
"[...] If from_addr is None or to_addrs is None, send_message fills those arguments with addresses extracted from the headers of msg as specified in RFC 5322: from_addr is set to the Sender field if it is present, and otherwise to the From field. to_addrs combines the values (if any) of the To, Cc, and Bcc fields from msg. "

As we're already using this function for convenience of the RFC 5322, then I agree to add it. We should also modify the doc & comment inside the code to make it clear that date is added if absent and following RFC 5322. (I've looked at the source and the send_message only mention RFC2822 in the comments, no RFC 5322).

Finally, why would we want to add a validate fonction to the email library? What would it do ? validate that we respect a certain RFC? Who other than SMTPlib would use it? I would like to understand the reasonning behind it.

Again, all this are opinions to let the discussion continue :).

For now, what I see we need to do (this bullet point list is intended to be expanded with what you think we need to do):

• implement a patch for the code to add a missing "Date" field if it doesn't exist
• Modify the documentation at the SMTPLib for the send_message to mention that it add missing date using the email.utils.formatdate
• Modify the comment of the send_message code to mention RFC 5322 in there (ideally with the section of the RFC).
• Fix it on all Python3 versions? It should have been supported since 3.2 right?

As it's my first time trying to contribute... I still don't know how to do so...

Regards,
Eric Lafontaine
eric.lafontaine1@gmail.com <= if you can help me outside of this discussion to contribute, it would be my pleasure.

[bitdancer]

Sure the comments can be updated. Some of them elsewhere have been already.

The reason for the email library to have a validation function is that it has an 'SMTP' policy that is designed to produce valid SMTP messages when the message is serialized. (It also has an HTTP policy that is designed to produce valid HTTP header blocks...though I'm sure there are bugs there as well). These policies are relatively new, even newer than the send_message method of smtplib.

If this gets done it time it could be fixed in 3.5...Larry is planning the final non-security-fix 3.5 release some time after 3.6.0 final goes out the door, as is our tradition. After that it could only go into what will very shortly be the maintenance release (3.6) and the next feature release (3.7).

To contribute, create a diff against the tip of the default branch and post it here. (We will be switching to github "soon", but posting a patch here will always work). See docs.python.org/devguide for more details on contributing.