The Job and Constraints

A project I am working on needed a self hosted mailing list. Because it is 2018 and email is hard and not getting flagged as spam was important we aren’t self hosting email, but instead using a 3rd party, Rackspace in this case, to provide email. To be fair, it seems that in this day and age it’s less and less likely one would find themselves installing a mailing list manager on the same server as their mail setup.

The rest of the setup was: Ubuntu 16.04, and an Nginx webserver.

The Implementation

Mailman seemed the obvious solution, it is selfhosted, opensource, and provides a web interface to management. Unfortunately all the defaults for everything used pretty much assumed local a local mailserver, so I had to find tutorials for hooking mailman up to a remote SMTP server to send, and to fetch its mail from a remote server, in this case via POP3. The solutions were pretty obvious and standard, use postfix locally as the MTA and configure it to send via the remote mail system, and rig up fetchmail to get the account’s remote mail.

Remote mail setup

The first thing to do was setup the mailman email account and the required aliases on the actual mail server. Which in this case meant going into Rackspace’s email management panel and setting up one mailbox for mailman and then the required aliases:

mailman-bounces
mailman-confirm
mailman-join
mailman-leave
mailman-owner
mailman-request
mailman-subscribe
mailman-unsubscribe

And then in our case since we wanted to establish a second list we also set up 9 new aliases for that also all pointing to mailman

new-list
new-list-bounces
new-list-confirm
new-list-join
new-list-leave
new-list-owner
new-list-request
new-list-subscribe
new-list-unsubscribe

Local Setup

Basics

apt install mailman fetchmail fcgiwrap mailutils libsasl2-modules

Mailman setup

In /etc/mailman/mm_cfg.py change

DEFAULT_EMAIL_HOST = 'mydomain.com'
DEFAULT_URL_HOST = 'lists.mydomain.com'
# Forcing HTTPS for security
DEFAULT_URL_PATTERN = 'https://%s/cgi-bin/mailman/'

(Hint for later: If at a later point you change some URLs and need to update the mailman site, it caches everything which can be a pain. However withlist -l -a -r fix_url -- -v should do the trick)

The system needs to be configured to deliver mail to the mailman program and not to mail boxes. To do this we use local aliases by editing /etc/aliases

## mailman mailing list
mailman:              "|/var/lib/mailman/mail/mailman post mailman"
mailman-admin:        "|/var/lib/mailman/mail/mailman admin mailman"
mailman-bounces:      "|/var/lib/mailman/mail/mailman bounces mailman"
mailman-confirm:      "|/var/lib/mailman/mail/mailman confirm mailman"
mailman-join:         "|/var/lib/mailman/mail/mailman join mailman"
mailman-leave:        "|/var/lib/mailman/mail/mailman leave mailman"
mailman-owner:        "|/var/lib/mailman/mail/mailman owner mailman"
mailman-request:      "|/var/lib/mailman/mail/mailman request mailman"
mailman-subscribe:    "|/var/lib/mailman/mail/mailman subscribe mailman"
mailman-unsubscribe:  "|/var/lib/mailman/mail/mailman unsubscribe mailman"

## new-list mailing list
new-list:              "|/var/lib/mailman/mail/mailman post new-list"
new-list-admin:        "|/var/lib/mailman/mail/mailman admin new-list"
new-list-bounces:      "|/var/lib/mailman/mail/mailman bounces new-list"
new-list-confirm:      "|/var/lib/mailman/mail/mailman confirm new-list"
new-list-join:         "|/var/lib/mailman/mail/mailman join new-list"
new-list-leave:        "|/var/lib/mailman/mail/mailman leave new-list"
new-list-owner:        "|/var/lib/mailman/mail/mailman owner new-list"
new-list-request:      "|/var/lib/mailman/mail/mailman request new-list"
new-list-subscribe:    "|/var/lib/mailman/mail/mailman subscribe new-list"
new-list-unsubscribe:  "|/var/lib/mailman/mail/mailman unsubscribe new-list"

and then run

newaliases

We need to actually have mailman create our mailing lists

newlist mailman
newlist new-list

Nginx setup for website

Mailman is only comes with an Apache drop in config file, so I had to do some google-fu to dig up recommended settings for Nginx.

So here is the full /etc/nginx/sites-available/lists.mydomain.com

server {
       server_name lists.mydomain.com;
       root /var/www/lists;
       if ($http_host != "lists.mydomain.com") {
                 rewrite ^ http://lists.mydomain.com$request_uri permanent;
       }
       index index.php index.html;
       location = /favicon.ico {
                log_not_found off;
                access_log off;
       }
       location = /robots.txt {
                allow all;
                log_not_found off;
                access_log off;
       }
       # Deny all attempts to access hidden files such as .htaccess, .htpasswd, .DS_Store (Mac).
       location ~ /\. {
                deny all;
                access_log off;
                log_not_found off;
       }

        location /cgi-bin/mailman {
               root /usr/lib/;
               fastcgi_split_path_info (^/cgi-bin/mailman/[^/]*)(.*)$;
               include /etc/nginx/fastcgi_params;
               fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
               fastcgi_param PATH_INFO $fastcgi_path_info;
               fastcgi_param PATH_TRANSLATED $document_root$fastcgi_path_info;
               fastcgi_intercept_errors on;
               fastcgi_pass unix:/var/run/fcgiwrap.socket;
        }
        location /images/mailman {
               alias /usr/share/images/mailman;
        }
        location /pipermail {
               alias /var/lib/mailman/archives/public;
               autoindex on;
        }

        # More ease of use

        location = / {
           rewrite ^ /cgi-bin/mailman/listinfo permanent;
        }

        location / {
            rewrite ^ /cgi-bin/mailman$uri?$args;
        }
        location = /mailman/ {
            rewrite ^ /cgi-bin/mailman/listinfo permanent;
        }
}

I made the empty /var/www/lists as the landing point. This is where the previously installed fcgiwrap comes into play. It’s a simple fast-cgi wrapper for binary cgi-bin style programs, which is what mailman uses.

Then of course I ran certbot and wrapped the whole site in SSL.

Configuring Postfix for remote sending

Mailman needs a local MTA to send all its mail. We aren’t installing it on the server where we actually want our domain’s email to be sent from so we need to install a local MTA and configure it to use the actual mail server. Postfix was picked by default.

First we needed to tell Postfix about the mailman account on the remote server and how to use it. Postfix uses SASL password files, so edit /etc/postfix/sasl/sasl_passwd

secure.emailsrvr.com:465 mailman@mydomain.com:PASSWORD

Where the first part is your remote email relay, and the second part is the mailman account we setup there and the password. Then run postmap sasl_passwd which generates sasl_passwd.db. Run chmod 600 sasl_passwd* to secure them.

Then we had to make relatively minimal changes to Postfix to get sending to work. Adding to and changing /etc/postfix/main.cf

smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination
myhostname = www.mydomain.com # don't want it to think it's mydomain.com and try to deliver that mail locally
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
mydestination = localhost # not mydomain.com because we want to use the relay for all sending
relayhost = secure.emailsrvr.com:465
inet_interfaces = loopback-only
inet_protocols = all

# enable SASL authentication
smtp_sasl_auth_enable = yes
# disallow methods that allow anonymous authentication.
smtp_sasl_security_options =
# where to find sasl_passwd
smtp_sasl_password_maps = hash:/etc/postfix/sasl/sasl_passwd
# Enable STARTTLS encryption
smtp_use_tls = yes

Then restart postfix and test with

echo "body of your test email" | mail -s "This is a Test Subject" -a "From: mailman@mydomain.com" you@somewhere.com

Using journalctl -u postfix.service you should see something like

Mar 30 20:32:11 hostname postfix/smtp[12918]: 7370513D7E2: to=<you@somewhere.com>, relay=secure.emailsrvr.com[166.78.79.129]:465, delay=0.4, delays=0.02/0.02/0.21/0.14, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as BBF3860168)

If it is not working, you will instead see something like

Mar 30 20:21:54 hostname postfix/smtp[12710]: 1974B13D7E2: to=<you@somewhere.com>, relay=secure.emailsrvr.com[184.106.54.10]:465, delay=0.18, delays=0.01/0.02/0.11/0.03, dsn=5.7.1, status=bounced (host secure.emailsrvr.com[184.106.54.10] said: 554 5.7.1 <mailman@mydomain.com>: Sender address rejected: Access denied (in reply to RCPT TO command))

Fetching the mailman’s remote email via POP3

We now have a mailman setup that can be manage via the web, and send emails, but it cannot get back the responses, to which we’ve already configured it to handle. So this is the last piece of the puzzle.

While most of the reference material talked about a .fetchmailrc the correct place to do the configuration was /etc/fetchmailrc

set daemon 60
set syslog

poll secure.emailsrvr.com with proto POP3
user 'mailman@mydomain.com' there with password 'PASSWORD'
to

'new-list@mydomain.com' = 'new-list'
'new-list-bounces@mydomain.com' = 'new-list-bounces'
'new-list-confirm@mydomain.com' = 'new-list-confirm'
'new-list-join@mydomain.com' = 'new-list-join'
'new-list-leave@mydomain.com' = 'new-list-leave'
'new-list-owner@mydomain.com' = 'new-list-owner'
'new-list-request@mydomain.com' = 'new-list-request'
'new-list-subscribe@mydomain.com' = 'new-list-subscribe'
'new-list-unsubscribe@mydomain.com' = 'new-list-unsubscribe'

'mailman@mydomain.com' = 'mailman'
'mailman-bounces@mydomain.com' = 'mailman-bounces'
'mailman-confirm@mydomain.com' = 'mailman-confirm'
'mailman-join@mydomain.com' = 'mailman-join'
'mailman-leave@mydomain.com' = 'mailman-leave'
'mailman-owner@mydomain.com' = 'mailman-owner'
'mailman-request@mydomain.com' = 'mailman-request'
'mailman-subscribe@mydomain.com' = 'mailman-subscribe'
'mailman-unsubscribe@mydomain.com' = 'mailman-unsubscribe'

The set daemon 60 command sets the poll rate, in this case to once every 60 seconds.

To enable fetchmail, you need to edit /etc/defaults/fetchmail and set START_DAEMON=yes.

Start the fetchmail service and check the logs for errors. If there aren’t any then with this final piece you should be done.

Testing

Go to your webportal and try and subscribe to a list. Confirm you got the email. Click the link to join, then try emailing in a post to new-list@mydomain.com to see if it gets posted (or caught for moderation). If all this is working, then congrats! Your setup is done. If not, well, start reading the logs (Is not a good time to mention that journalctl not doing line-wrap really throws me).

Conclusion

This was the full set up (as I remembered it a day later) that I got working. The reference material I used to cobble together this setup and tutorial is below. Unfortunately while to me this doesn’t seem like an outlandish use case, it’s wildly not the default and needed a lot of work to get going. If the above tutorial seems simple then I worry I’ve forgotten something. This definitely took me some time (and frustration) to piece together as each part (nginx, sending via relay, fetching remote mail) were all non default use cases. Also some of this software, especially mailman, is very old, I mean cgi-bin style web apps was last in style in the 90s!

References