Recent issue with UniFi Controller (version 5.12.35) where test emails were failing on a remote SMTP server (Postfix) configured for STARTTLS listening on port 25.
The workaround was to open port 587 on the firewall and redirect it to port 25 of the server. Then configure the UniFi controller to use port 587 and uncheck the SSL option.
Be sure that your SSL certificate on the mail server is valid. Otherwise the controller will fail to send the email and Postfix logs may look similar to the following:
connect from xxx
lost connection after STARTTLS from xxx