How to set up an email server on a VPS

Quick answer: Install a mail stack — classic Postfix + Dovecot, or an all-in-one bundle like docker-mailserver or Mailcow — then configure MX, SPF, DKIM, DMARC and a PTR record. Plan a few hours for the first setup and ongoing time for maintenance: a mail server is a service you operate, not a one-off install.

Overview

A complete mail server is several cooperating services: SMTP for sending/receiving (Postfix), IMAP/POP3 for mailbox access (Dovecot), spam filtering (rspamd/SpamAssassin), TLS certificates and authentication DNS records. All-in-one bundles package these together and are the practical choice for most self-hosters. Be honest with yourself about the effort — read why self-hosted email is difficult first.

Before you start

• A VPS with at least 2 GB RAM (Mailcow recommends 6 GB+ with ClamAV enabled).
• A domain, DNS access, and a hostname like mail.yourdomain.com pointing to the server.
• A PTR record matching that hostname.
• Docker installed, if you choose a containerized bundle.

Step-by-step guide

1. Set the hostname:

   hostnamectl set-hostname mail.yourdomain.com

2. Create DNS records: an A record for mail, MX pointing to it, then SPF, DKIM and DMARC.
3. Install the stack. Classic route:

   apt install postfix dovecot-imapd dovecot-lmtpd rspamd
   postconf -e 'myhostname = mail.yourdomain.com'

   Bundle route: follow the official docker-mailserver or Mailcow installation guide.
4. Issue TLS certificates (Let's Encrypt) for the mail hostname.
5. Create mailboxes and test IMAP/SMTP login from a mail client.
6. Send a test message to a mailbox you control at a large provider and inspect the Authentication-Results header — SPF, DKIM and DMARC should all pass.

Common issues

• Outbound mail rejected: missing PTR or SPF — verify with an IP reputation check.
• Certificates fail to issue: the A record for the mail hostname must resolve to the VPS before running certbot.
• Low memory: spam/antivirus filters are RAM-hungry; upgrade the VPS or disable ClamAV on small instances.

When to contact support

Ask via ticket if you need a PTR record set, have questions about mail-port connectivity from your VPS, or are unsure whether your planned sending volume fits the abuse policy.

Frequently asked questions

Which mail stack should a beginner choose?

An all-in-one bundle such as Mailcow or docker-mailserver is the practical choice: it packages Postfix, Dovecot, spam filtering and TLS together with sane defaults and one config.

How much RAM does a self-hosted mail server need?

Plan at least 2 GB for a light Postfix and Dovecot setup; full bundles with antivirus scanning like Mailcow with ClamAV are comfortable from about 6 GB of RAM.

How long does the initial setup realistically take?

Expect a few hours for the first working setup including DNS records and TLS, then ongoing monthly time for updates, monitoring and deliverability upkeep afterwards.

Related articles

• How to set reverse DNS / PTR record
• How to configure SPF
• How to configure DKIM
• How to configure DMARC
• Why self-hosted email is difficult

Ready to get started? Order a VPS at Cloud2Y →