I have setup sendmail to relay mail from the local server destined for external email addresses to use a SMART HOST. I have set this up many times on other systems with no problems. All use the same version of sendmail 8.14.4. I have set up configurations with and without authentication using access when needed. This server was working until we switch the SMART HOST over to smtp.office365.com which requires TLS

How do I setup sendmail to use TLS when talking to the relay server?
How can I debug/trace the communication between sendmail and the replay server.

I have setup the certificates for TLS as per the documentation in sendmail.mc

dnl # Rudimentary information on creating certificates for sendmail TLS:
dnl # cd /etc/pki/tls/certs; make sendmail.pem
dnl # Complete usage:
dnl # make -C /etc/pki/tls/certs usage
dnl #
define(`confCACERT_PATH', `/etc/pki/tls/certs')dnl
define(`confCACERT', `/etc/pki/tls/certs/ca-bundle.crt')dnl
define(`confSERVER_CERT', `/etc/pki/tls/certs/sendmail.pem')dnl
define(`confSERVER_KEY', `/etc/pki/tls/certs/sendmail.pem')dnl
dnl #

I have feeling this only configures TLS for incoming connections to the server, and not out going connections to the relay server from sendmail. Perhaps I need to set something up in submit.mc

This is the error I get in the bounce back email:
(reason: 530 5.7.57 Client not authenticated to send mail. [MW2PR16CA0045.namprd16.prod.outlook.com])

I believe the client is not authenticated because it is not using TLS.

Linux is CentOS 6.10.

You don't say if your domain on O365 has SMTP AUTH disabled (I'm guessing that's it's enabled as default)

If SMTP AUTH is enabled then you need both TLS and SMTP AUTH configured on your relaying host.

It's also likely that you'll only be able to configure auth settings for a single mailbox in sendmail, so all the mail will appear to be coming from that O365 account.

I don't have access to the O365 setup. I am guessing it is enabled, because TLS is required by the other client I am using, and it has no problems. That is why I am wanting to configure sendmail to use TLS when it is acting as a client in relay mode.

This is what I get in maillog when I try to send an email that requires relaying.

Dec 5 10:26:16 wmat sendmail[187480]: STARTTLS=client, relay=smtp.office365.com, version=TLSv1/SSLv3, verify=OK, cipher=ECDHE-RSA-AES256-GCM-SHA384, bits=256/256
Dec 5 10:26:21 wmat sendmail[187480]: 2B5HMGDG187478: to=<xxxxxxx@gmail.com>, ctladdr=<root@domain> (0/0), delay=00:04:05, xdelay=00:04:05, mailer=relay, pri=120468, relay=smtp.office365.com [52.96.121.2], dsn=5.0.0, stat=Service unavailable
Dec 5 10:26:21 wmat sendmail[187480]: 2B5HMGDG187478: 2B5HQLDF187480: DSN: Service unavailable

Make use of this site: https://www.appmaildev.com/en/dkim (yes, it's their DKIM checker but it will also tell you if the mail was sent with SSL/TLS or unencrypted.

I solved the problem. I had two issues. First my access file had an extra period at the end of the server name in the AuthInfo line, so it wasn't using the username/password at all.
After fixing that it complained about Mechanisms not being sufficient, so I followed the instructions in sendmail.mc to enable saslauthd, and uncomment the 2 lines for mechanisms.

TRUST_AUTH_MECH(`EXTERNAL DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnl
define(`confAUTH_MECHANISMS', `EXTERNAL GSSAPI DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')

Excellent glad you got sorted. I kinda felt the main issue would be AUTH related.