diff --git a/content/2.getting-started/2.installation.md b/content/2.getting-started/2.installation.md index 1adc920..909cf26 100644 --- a/content/2.getting-started/2.installation.md +++ b/content/2.getting-started/2.installation.md @@ -65,3 +65,14 @@ docker run -d \ Once this has started, Caddy will issue an SSL certificate for your domain and you'll be able to immediately access the Postal web interface and login with the user you created in one of the previous steps. ![Image](/screenshots/Screen-Shot-2021-07-29-23-26-18.23-Qwv2DD40v4jMEoaHtE.png) + +## Next steps + +Once you can log in: + +1. Create an organization and a mail server. +2. Add the domain you want to send from and follow the [Sending domains](/features/sending-domains) page to verify it and publish its SPF, DKIM and return path records. +3. Create an **SMTP** or **API** [credential](/features/smtp-authentication) and send a test message. +4. Point the `smtp` section of `postal.yml` at your new mail server so that Postal can send its own notification e-mails, then `postal restart`. + +The full set of commands provided by the `postal` helper is documented on [The postal command](/getting-started/postal-command) page. diff --git a/content/2.getting-started/4.dns-configuration.md b/content/2.getting-started/4.dns-configuration.md index a5ca5b1..dd8c999 100644 --- a/content/2.getting-started/4.dns-configuration.md +++ b/content/2.getting-started/4.dns-configuration.md @@ -99,7 +99,7 @@ You may wish to replace ~all with -all to make the SPF ## Return Path -The return path domain is the default domain that is used as the `MAIL FROM` for all messages sent through a mail server. You should add DNS records as below. +The return path domain is the default domain that is used as the `MAIL FROM` for all messages sent through a mail server, so bounces and auto-responses are delivered here. It is also the domain Postal signs messages with (using your `signing.key`) whenever a sending domain does not yet have a working DKIM record of its own. You should add DNS records as below. @@ -133,7 +133,7 @@ The return path domain is the default domain that is used as the `MAIL FROM` for - +
postal._domainkey.rp.postal.example.com TXTValue from postal default-dkim-recordValue from postal default-dkim-record (the postal selector is the value of dns.dkim_identifier)
@@ -141,7 +141,7 @@ The return path domain is the default domain that is used as the `MAIL FROM` for ## Route domain -If you wish to receive incoming e-mail by forwarding messages directly to routes in Postal, you'll need to configure a domain for this just to point to your server using an MX record. +If you wish to receive incoming e-mail by forwarding messages directly to routes in Postal (each route has a unique `{token}@routes.postal.example.com` address - see [Routing incoming e-mail](/features/routing-incoming-email)), you'll need to configure a domain for this just to point to your server using an MX record. @@ -186,6 +186,10 @@ If you would like to make use of Click and Open Tracking then you should set up
+## Records for each sending domain + +The records above are for the Postal installation itself. Each domain you send mail *from* additionally needs its own SPF, DKIM and (optionally) return path and MX records, which Postal generates for you and checks automatically. These are described on the [Sending domains](/features/sending-domains) page. + ## Example Postal Configuration In your `postal.yml` you should have something that looks like the below to cover the key DNS records. diff --git a/content/2.getting-started/7.configuration-reference.md b/content/2.getting-started/7.configuration-reference.md new file mode 100644 index 0000000..646e0c3 --- /dev/null +++ b/content/2.getting-started/7.configuration-reference.md @@ -0,0 +1,240 @@ +--- +title: Configuration reference +description: 'Every Postal configuration option, its default value and environment variable name.' +category: Installation +--- +This page lists every configuration option available in Postal 3.3.7. Each option can be set in the `postal.yml` configuration file under the section heading shown, or with the environment variable shown. See [Configuration](/getting-started/configuration) for how the file and environment variables are loaded. + +This list is derived from the [configuration schema](https://github.com/postalserver/postal/blob/main/lib/postal/config_schema.rb) in the Postal repository. The upstream repository also publishes it as an [example YAML file](https://github.com/postalserver/postal/blob/main/doc/config/yaml.yml) and a [list of environment variables](https://github.com/postalserver/postal/blob/main/doc/config/environment-variables.md), which may include options added after this page was written. + +An empty default means the option is unset unless you provide a value. Array options are written as YAML lists in the file and as comma-separated values in environment variables. + +## postal + +Installation-wide settings. + +| Option | Environment variable | Default | Description | +|---|---|---|---| +| `web_hostname` | `POSTAL_WEB_HOSTNAME` | `postal.example.com` | The hostname that the Postal web interface runs on. | +| `web_protocol` | `POSTAL_WEB_PROTOCOL` | `https` | The HTTP protocol to use for the Postal web interface. | +| `smtp_hostname` | `POSTAL_SMTP_HOSTNAME` | `postal.example.com` | The hostname that the Postal SMTP server runs on. | +| `use_ip_pools` | `POSTAL_USE_IP_POOLS` | `false` | Should IP pools be enabled for this installation? See [IP Pools](/features/ip-pools). | +| `default_maximum_delivery_attempts` | `POSTAL_DEFAULT_MAXIMUM_DELIVERY_ATTEMPTS` | `18` | The maximum number of delivery attempts. After this many attempts a message is hard failed (and, for outgoing mail, the recipient is added to the suppression list). | +| `default_maximum_hold_expiry_days` | `POSTAL_DEFAULT_MAXIMUM_HOLD_EXPIRY_DAYS` | `7` | The number of days to hold a message before they will be expired. Held messages are released without action after this long. | +| `default_suppression_list_automatic_removal_days` | `POSTAL_DEFAULT_SUPPRESSION_LIST_AUTOMATIC_REMOVAL_DAYS` | `30` | The number of days an address will remain in a suppression list before being removed. | +| `default_spam_threshold` | `POSTAL_DEFAULT_SPAM_THRESHOLD` | `5` | The default threshold at which a message should be treated as spam. Used as the initial value for new mail servers; changeable per server. | +| `default_spam_failure_threshold` | `POSTAL_DEFAULT_SPAM_FAILURE_THRESHOLD` | `20` | The default threshold at which a message should be treated as spam failure. Used as the initial value for new mail servers; changeable per server. | +| `use_local_ns_for_domain_verification` | `POSTAL_USE_LOCAL_NS_FOR_DOMAIN_VERIFICATION` | `false` | Domain verification and checking usually checks with a domain's nameserver. Enable this to check with the server's local nameservers. | +| `use_resent_sender_header` | `POSTAL_USE_RESENT_SENDER_HEADER` | `true` | Append a `Resent-Sender` header (containing the envelope sender) to all outgoing e-mails. | +| `signing_key_path` | `POSTAL_SIGNING_KEY_PATH` | `$config-file-root/signing.key` | Path to the private key used for signing. RSA private key used to sign webhook/HTTP endpoint requests and for the return path DKIM record. | +| `smtp_relays` | `POSTAL_SMTP_RELAYS` | `[]` | An array of SMTP relays in the format of smtp://host:port. Format `smtp://host:port?ssl_mode=MODE` where MODE is `Auto`, `STARTLS`, `TLS` or `None`. When set, all outgoing mail goes via the relays. See [SMTP relays](/getting-started/configuration#smtp-relays). | +| `trusted_proxies` | `POSTAL_TRUSTED_PROXIES` | `[]` | An array of IP addresses to trust for proxying requests to Postal (in addition to localhost addresses). IP addresses or CIDR ranges. | +| `allowed_request_destinations` | `POSTAL_ALLOWED_REQUEST_DESTINATIONS` | `[]` | Hostnames or IP/CIDR ranges that outbound webhook and HTTP endpoint requests are permitted to reach even when they resolve to a private, loopback, link-local or otherwise reserved address. All other such destinations are blocked to prevent SSRF. See [Blocked destinations](/developer/http-payloads#blocked-destinations). | +| `queued_message_lock_stale_days` | `POSTAL_QUEUED_MESSAGE_LOCK_STALE_DAYS` | `1` | The number of days after which to consider a lock as stale. Messages with stale locks will be removed and not retried. See [Workers & background tasks](/other/workers-and-background-tasks#stale-locks). | +| `batch_queued_messages` | `POSTAL_BATCH_QUEUED_MESSAGES` | `true` | When enabled queued messages will be de-queued in batches based on their destination. | + +## web_server + +Settings for the `postal web-server` process. See also the `PORT` and `BIND_ADDRESS` environment variables. + +| Option | Environment variable | Default | Description | +|---|---|---|---| +| `default_port` | `WEB_SERVER_DEFAULT_PORT` | `5000` | The default port the web server should listen on unless overriden by the PORT environment variable. | +| `default_bind_address` | `WEB_SERVER_DEFAULT_BIND_ADDRESS` | `127.0.0.1` | The default bind address the web server should listen on unless overriden by the BIND_ADDRESS environment variable. | +| `max_threads` | `WEB_SERVER_MAX_THREADS` | `5` | The maximum number of threads which can be used by the web server. | + +## worker + +Settings for `postal worker` processes. See [Workers & background tasks](/other/workers-and-background-tasks). + +| Option | Environment variable | Default | Description | +|---|---|---|---| +| `default_health_server_port` | `WORKER_DEFAULT_HEALTH_SERVER_PORT` | `9090` | The default port for the worker health server to listen on. | +| `default_health_server_bind_address` | `WORKER_DEFAULT_HEALTH_SERVER_BIND_ADDRESS` | `127.0.0.1` | The default bind address for the worker health server to listen on. | +| `threads` | `WORKER_THREADS` | `2` | The number of threads to execute within each worker. The database connection pool is grown automatically to at least `threads + 3`. | + +## main_db + +The main MariaDB database which stores organizations, servers, users, domains, routes and the message queue. + +| Option | Environment variable | Default | Description | +|---|---|---|---| +| `host` | `MAIN_DB_HOST` | `localhost` | Hostname for the main MariaDB server. | +| `port` | `MAIN_DB_PORT` | `3306` | The MariaDB port to connect to. | +| `username` | `MAIN_DB_USERNAME` | `postal` | The MariaDB username. | +| `password` | `MAIN_DB_PASSWORD` | | The MariaDB password. | +| `database` | `MAIN_DB_DATABASE` | `postal` | The MariaDB database name. | +| `pool_size` | `MAIN_DB_POOL_SIZE` | `5` | The maximum size of the MariaDB connection pool. Workers automatically grow this to at least `worker.threads + 3`. | +| `encoding` | `MAIN_DB_ENCODING` | `utf8mb4` | The encoding to use when connecting to the MariaDB database. | + +## message_db + +Connection details for the MariaDB server on which a separate database is created for each mail server. This may be the same server as `main_db`. Databases are named `{database_name_prefix}-server-{id}`. + +| Option | Environment variable | Default | Description | +|---|---|---|---| +| `host` | `MESSAGE_DB_HOST` | `localhost` | Hostname for the MariaDB server which stores the mail server databases. | +| `port` | `MESSAGE_DB_PORT` | `3306` | The MariaDB port to connect to. | +| `username` | `MESSAGE_DB_USERNAME` | `postal` | The MariaDB username. | +| `password` | `MESSAGE_DB_PASSWORD` | | The MariaDB password. | +| `encoding` | `MESSAGE_DB_ENCODING` | `utf8mb4` | The encoding to use when connecting to the MariaDB database. | +| `database_name_prefix` | `MESSAGE_DB_DATABASE_NAME_PREFIX` | `postal` | The MariaDB prefix to add to database names. | + +## logging + +See [Logging](/features/logging). + +| Option | Environment variable | Default | Description | +|---|---|---|---| +| `rails_log_enabled` | `LOGGING_RAILS_LOG_ENABLED` | `false` | Enable the default Rails logger. | +| `sentry_dsn` | `LOGGING_SENTRY_DSN` | | A DSN which should be used to report exceptions to Sentry. | +| `enabled` | `LOGGING_ENABLED` | `true` | Enable the Postal logger to log to STDOUT. | +| `highlighting_enabled` | `LOGGING_HIGHLIGHTING_ENABLED` | `false` | Enable highlighting of log lines. | + +## gelf + +Send log output to a Graylog/GELF server over UDP in addition to STDOUT. Enabled when `host` is set. + +| Option | Environment variable | Default | Description | +|---|---|---|---| +| `host` | `GELF_HOST` | | GELF-capable host to send logs to. | +| `port` | `GELF_PORT` | `12201` | GELF port to send logs to. | +| `facility` | `GELF_FACILITY` | `postal` | The facility name to add to all log entries sent to GELF. | + +## smtp_server + +Settings for the `postal smtp-server` process. See [SMTP TLS](/features/smtp-tls) and [SMTP Authentication](/features/smtp-authentication). + +| Option | Environment variable | Default | Description | +|---|---|---|---| +| `default_port` | `SMTP_SERVER_DEFAULT_PORT` | `25` | The default port the SMTP server should listen on unless overriden by the PORT environment variable. | +| `default_bind_address` | `SMTP_SERVER_DEFAULT_BIND_ADDRESS` | `::` | The default bind address the SMTP server should listen on unless overriden by the BIND_ADDRESS environment variable. `::` listens on all IPv4 and IPv6 addresses. | +| `default_health_server_port` | `SMTP_SERVER_DEFAULT_HEALTH_SERVER_PORT` | `9091` | The default port for the SMTP server health server to listen on. | +| `default_health_server_bind_address` | `SMTP_SERVER_DEFAULT_HEALTH_SERVER_BIND_ADDRESS` | `127.0.0.1` | The default bind address for the SMTP server health server to listen on. | +| `tls_enabled` | `SMTP_SERVER_TLS_ENABLED` | `false` | Enable TLS for the SMTP server (requires certificate). | +| `tls_certificate_path` | `SMTP_SERVER_TLS_CERTIFICATE_PATH` | `$config-file-root/smtp.cert` | The path to the SMTP server's TLS certificate. | +| `tls_private_key_path` | `SMTP_SERVER_TLS_PRIVATE_KEY_PATH` | `$config-file-root/smtp.key` | The path to the SMTP server's TLS private key. | +| `tls_ciphers` | `SMTP_SERVER_TLS_CIPHERS` | | Override ciphers to use for SSL. | +| `ssl_version` | `SMTP_SERVER_SSL_VERSION` | `SSLv23` | The SSL versions which are supported. An OpenSSL version constant such as `SSLv23`, `TLSv1_2` or `TLSv1_3`. | +| `proxy_protocol` | `SMTP_SERVER_PROXY_PROTOCOL` | `false` | Enable proxy protocol for use behind some load balancers (supports proxy protocol v1 only). When enabled the `220` greeting is delayed until the `PROXY` line is received. | +| `log_connections` | `SMTP_SERVER_LOG_CONNECTIONS` | `false` | Enable connection logging. | +| `max_message_size` | `SMTP_SERVER_MAX_MESSAGE_SIZE` | `14` | The maximum message size to accept from the SMTP server (in MB). Checked once the whole message has been received; clients receive `552 Message too large`. | +| `log_ip_address_exclusion_matcher` | `SMTP_SERVER_LOG_IP_ADDRESS_EXCLUSION_MATCHER` | | A regular expression to use to exclude connections from logging. Ruby regular expression matched against the client IP address. | + +## dns + +The DNS names your installation uses. See [DNS configuration](/getting-started/dns-configuration) and [Sending domains](/features/sending-domains). + +| Option | Environment variable | Default | Description | +|---|---|---|---| +| `mx_records` | `DNS_MX_RECORDS` | `mx1.postal.example.com, mx2.postal.example.com` | The names of the default MX records. | +| `spf_include` | `DNS_SPF_INCLUDE` | `spf.postal.example.com` | The location of the SPF record. | +| `return_path_domain` | `DNS_RETURN_PATH_DOMAIN` | `rp.postal.example.com` | The return path hostname. | +| `route_domain` | `DNS_ROUTE_DOMAIN` | `routes.postal.example.com` | The domain to use for hosting route-specific addresses. | +| `track_domain` | `DNS_TRACK_DOMAIN` | `track.postal.example.com` | The CNAME which tracking domains should be pointed to. | +| `helo_hostname` | `DNS_HELO_HOSTNAME` | | The hostname to use in HELO/EHLO when connecting to external SMTP servers. Falls back to `postal.smtp_hostname`. When sending from an IP pool address, the address's own hostname is used instead. | +| `dkim_identifier` | `DNS_DKIM_IDENTIFIER` | `postal` | The identifier to use for DKIM keys in DNS records. Per-domain DKIM selectors are `{dkim_identifier}-{random}`, e.g. `postal-AB1CDE._domainkey`. | +| `domain_verify_prefix` | `DNS_DOMAIN_VERIFY_PREFIX` | `postal-verification` | The prefix to add before TXT record verification string. Verification TXT records are `{domain_verify_prefix} {token}`. | +| `custom_return_path_prefix` | `DNS_CUSTOM_RETURN_PATH_PREFIX` | `psrp` | The domain to use on external domains which points to the Postal return path domain. Domains may CNAME `{prefix}.yourdomain.com` to `return_path_domain`. | +| `timeout` | `DNS_TIMEOUT` | `5` | The timeout to wait for DNS resolution. | +| `resolv_conf_path` | `DNS_RESOLV_CONF_PATH` | `/etc/resolv.conf` | The path to the resolv.conf file containing addresses for local nameservers. Used for MX lookups when sending and, when `postal.use_local_ns_for_domain_verification` is enabled, for domain checks. | + +## smtp + +The SMTP server Postal uses to send its **own** e-mails (password resets, send limit warnings, suspension notices, test messages). This is not used for mail sent through your mail servers. Once Postal is running you can point this at one of your own mail servers. + +| Option | Environment variable | Default | Description | +|---|---|---|---| +| `host` | `SMTP_HOST` | `127.0.0.1` | The hostname to send application-level e-mails to. | +| `port` | `SMTP_PORT` | `25` | The port number to send application-level e-mails to. | +| `username` | `SMTP_USERNAME` | | The username to use when authentication to the SMTP server. | +| `password` | `SMTP_PASSWORD` | | The password to use when authentication to the SMTP server. | +| `authentication_type` | `SMTP_AUTHENTICATION_TYPE` | `login` | The type of authentication to use. `plain`, `login` or `cram_md5`. | +| `enable_starttls` | `SMTP_ENABLE_STARTTLS` | `false` | Use STARTTLS when connecting to the SMTP server and fail if unsupported. | +| `enable_starttls_auto` | `SMTP_ENABLE_STARTTLS_AUTO` | `true` | Detects if STARTTLS is enabled in the SMTP server and starts to use it. | +| `openssl_verify_mode` | `SMTP_OPENSSL_VERIFY_MODE` | `peer` | When using TLS, you can set how OpenSSL checks the certificate. Use 'none' for no certificate checking. `peer` or `none`. | +| `from_name` | `SMTP_FROM_NAME` | `Postal` | The name to use as the from name outgoing emails from Postal. | +| `from_address` | `SMTP_FROM_ADDRESS` | `postal@example.com` | The e-mail to use as the from address outgoing emails from Postal. | + +## rails + +Settings for the underlying Rails application. + +| Option | Environment variable | Default | Description | +|---|---|---|---| +| `environment` | `RAILS_ENVIRONMENT` | `production` | The Rails environment to run the application in. Leave as `production`. | +| `secret_key` | `RAILS_SECRET_KEY` | | The secret key used to sign and encrypt cookies and session data in the application. Generated for you by `postal bootstrap`. Changing it invalidates all sessions. | + +## rspamd + +See [Spam & Virus Checking](/features/spam-and-virus-checking). If both rspamd and spamd are enabled, only rspamd is used. + +| Option | Environment variable | Default | Description | +|---|---|---|---| +| `enabled` | `RSPAMD_ENABLED` | `false` | Enable rspamd for message inspection. | +| `host` | `RSPAMD_HOST` | `127.0.0.1` | The hostname of the rspamd server. | +| `port` | `RSPAMD_PORT` | `11334` | The port of the rspamd server. | +| `ssl` | `RSPAMD_SSL` | `false` | Enable SSL for the rspamd connection. | +| `password` | `RSPAMD_PASSWORD` | | The password for the rspamd server. | +| `flags` | `RSPAMD_FLAGS` | | Any flags for the rspamd server. | + +## spamd + +See [Spam & Virus Checking](/features/spam-and-virus-checking). + +| Option | Environment variable | Default | Description | +|---|---|---|---| +| `enabled` | `SPAMD_ENABLED` | `false` | Enable SpamAssassin for message inspection. | +| `host` | `SPAMD_HOST` | `127.0.0.1` | The hostname for the SpamAssassin server. | +| `port` | `SPAMD_PORT` | `783` | The port of the SpamAssassin server. | + +## clamav + +See [Spam & Virus Checking](/features/spam-and-virus-checking). + +| Option | Environment variable | Default | Description | +|---|---|---|---| +| `enabled` | `CLAMAV_ENABLED` | `false` | Enable ClamAV for message inspection. | +| `host` | `CLAMAV_HOST` | `127.0.0.1` | The host of the ClamAV server. | +| `port` | `CLAMAV_PORT` | `2000` | The port of the ClamAV server. | + +## smtp_client + +Timeouts (in seconds) for outgoing SMTP connections made by workers when delivering mail. + +| Option | Environment variable | Default | Description | +|---|---|---|---| +| `open_timeout` | `SMTP_CLIENT_OPEN_TIMEOUT` | `30` | The open timeout for outgoing SMTP connections. | +| `read_timeout` | `SMTP_CLIENT_READ_TIMEOUT` | `30` | The read timeout for outgoing SMTP connections. | + +## migration_waiter + +See [Our container image](/other/containers#waiting-for-database-migrations). + +| Option | Environment variable | Default | Description | +|---|---|---|---| +| `enabled` | `MIGRATION_WAITER_ENABLED` | `false` | Wait for all migrations to run before starting a process. | +| `attempts` | `MIGRATION_WAITER_ATTEMPTS` | `120` | The number of attempts to try waiting for migrations to complete before start. | +| `sleep_time` | `MIGRATION_WAITER_SLEEP_TIME` | `2` | The number of seconds to wait between each migration check. | + +## oidc + +See [OpenID Connect](/features/oidc). + +| Option | Environment variable | Default | Description | +|---|---|---|---| +| `enabled` | `OIDC_ENABLED` | `false` | Enable OIDC authentication. | +| `local_authentication_enabled` | `OIDC_LOCAL_AUTHENTICATION_ENABLED` | `true` | When enabled, users with passwords will still be able to login locally. If disable, only OpenID Connect will be available. | +| `name` | `OIDC_NAME` | `OIDC Provider` | The name of the OIDC provider as shown in the UI. | +| `issuer` | `OIDC_ISSUER` | | The OIDC issuer URL. | +| `identifier` | `OIDC_IDENTIFIER` | | The client ID for OIDC. | +| `secret` | `OIDC_SECRET` | | The client secret for OIDC. | +| `scopes` | `OIDC_SCOPES` | `openid, email` | Scopes to request from the OIDC server. Must include enough scopes for the provider to return the user's e-mail address. | +| `uid_field` | `OIDC_UID_FIELD` | `sub` | The field to use to determine the user's UID. | +| `email_address_field` | `OIDC_EMAIL_ADDRESS_FIELD` | `email` | The field to use to determine the user's email address. | +| `name_field` | `OIDC_NAME_FIELD` | `name` | The field to use to determine the user's name. | +| `discovery` | `OIDC_DISCOVERY` | `true` | Enable discovery to determine endpoints from .well-known/openid-configuration from the Issuer. | +| `authorization_endpoint` | `OIDC_AUTHORIZATION_ENDPOINT` | | The authorize endpoint on the authorization server (only used when discovery is false). | +| `token_endpoint` | `OIDC_TOKEN_ENDPOINT` | | The token endpoint on the authorization server (only used when discovery is false). | +| `userinfo_endpoint` | `OIDC_USERINFO_ENDPOINT` | | The user info endpoint on the authorization server (only used when discovery is false). | +| `jwks_uri` | `OIDC_JWKS_URI` | | The JWKS endpoint on the authorization server (only used when discovery is false). | diff --git a/content/2.getting-started/8.postal-command.md b/content/2.getting-started/8.postal-command.md new file mode 100644 index 0000000..d6d483e --- /dev/null +++ b/content/2.getting-started/8.postal-command.md @@ -0,0 +1,83 @@ +--- +title: The postal command +description: 'Reference for the postal helper command installed from the postalserver/install repository.' +category: Installation +--- + +The `postal` command used throughout these docs is a small Bash script from the [installation helper repository](https://github.com/postalserver/install), which the [pre-requisites](/getting-started/prerequisites) page has you clone to `/opt/postal/install` and symlink to `/usr/bin/postal`. It wraps Docker Compose so that you don't need to remember the container commands. + +All commands operate on a `docker-compose.yml` in `/opt/postal/install`, which is generated for you from a template (`templates/docker-compose.v3.yml`) the first time you run `bootstrap`, `upgrade` or any command that needs it. Docker Compose is run with the project name `postal`, so the containers are named `postal-web-1`, `postal-smtp-1`, `postal-worker-1` and so on. + +## Running Postal + +| Command | Description | +|---|---| +| `postal start` | Start all services in the background (`docker compose up -d`). Extra arguments are passed through, e.g. `postal start web`. | +| `postal stop` | Stop and remove the containers (`docker compose down`). | +| `postal restart` | Restart all containers. Required after changing `postal.yml`. | +| `postal status` | Show the running containers (`docker compose ps`). | +| `postal logs [service]` | Show logs for all services or one of `web`, `smtp`, `worker`. Extra arguments are passed through, e.g. `postal logs -f worker`. | +| `postal bash [service]` | Open a shell inside a running service container. | +| `postal dc [args]` | Run any other `docker compose` command against the Postal project, e.g. `postal dc pull`. | + +## Setup and upgrade + +| Command | Description | +|---|---| +| `postal bootstrap hostname [path]` | Create initial configuration in `path` (default `/opt/postal/config`): `postal.yml` from the example file with your hostname and a random `rails.secret_key` filled in, a `Caddyfile`, and a 1024-bit RSA `signing.key`. Existing files are never overwritten. Also generates `docker-compose.yml` for the latest release. | +| `postal initialize` | Pull the image and run `postal initialize` inside a temporary container to create the main database and load the schema. | +| `postal upgrade [version]` | Upgrade to the given version (or the latest release). See [Upgrading](/getting-started/upgrading). | +| `postal upgrade-db` | Run database migrations only, without pulling a new image or restarting. Useful after restoring a database from another installation. | +| `postal set-version x.x.x` | Regenerate `docker-compose.yml` for a specific version without pulling or restarting anything. | + +## Other tools + +These run the corresponding command from the [container image](/other/containers#other-commands) in a temporary `runner` container. + +| Command | Description | +|---|---| +| `postal make-user` | Interactively create a global administrator user (prompts for e-mail address, first name, last name and password). | +| `postal default-dkim-record` | Print the DKIM TXT record to publish at `postal._domainkey.{return path domain}`. | +| `postal test-app-smtp address` | Send a test e-mail using the `smtp` section of your configuration. | +| `postal console` | Open a Rails console against your installation. | +| `postal version` | Print the version of Postal in the configured image. | + +## Options + +| Option | Description | +|---|---| +| `--version x.x.x` | With `bootstrap` or `upgrade`, use this version instead of looking up the latest release on GitHub. | +| `--no-git-pull` | With `upgrade`, skip updating the helper repository first. | +| `--dev` | Print the commands that would be run instead of running them. | + +Looking up the latest release requires `curl` and `jq` and makes an unauthenticated request to the GitHub API, which is rate limited. If you hit the limit, pass an explicit version. + +## Customising the installation + +### Number of workers + +To run more than one worker container, add a `docker-compose.override.yml` (see below) that sets the number of replicas for the `worker` service. This survives upgrades, whereas `postal start --scale worker=3` is reset to one worker the next time `postal upgrade` runs. + +```yaml +services: + worker: + deploy: + replicas: 3 +``` + +### Overriding the compose file + +Docker Compose automatically merges `docker-compose.override.yml` from the same directory. Use this for changes such as [log drivers](/features/logging#redirecting-logs-to-the-host-syslog), extra environment variables or resource limits, so that they survive upgrades (the generated `docker-compose.yml` is replaced on every upgrade). + +### Hooks + +If a `hooks` directory exists in `/opt/postal/install`, the Bash script with the matching name in it is run at various points. Available hook names are `pre-start`, `post-start`, `pre-stop`, `post-stop`, `pre-restart`, `post-restart`, `pre-initialize-pull`, `pre-initialize`, `post-initialize`, `pre-upgrade-pull`, `post-upgrade-pull`, `pre-upgrade-db`, `post-upgrade-db`, `post-upgrade`, `pre-bootstrap`, `post-bootstrap` and `set-postal-version`. + +## The generated compose file + +For reference, the v3 template starts the following services, all using host networking and mounting `/opt/postal/config` at `/config`: + +* `web` - `postal web-server` +* `smtp` - `postal smtp-server` with the `NET_BIND_SERVICE` capability +* `worker` - `postal worker` +* `runner` - a `tools` profile service used by the helper for one-off commands; it is not started by `postal start` diff --git a/content/3.features/click-and-open-tracking.md b/content/3.features/click-and-open-tracking.md index 7d8acb9..cfd1708 100644 --- a/content/3.features/click-and-open-tracking.md +++ b/content/3.features/click-and-open-tracking.md @@ -1,6 +1,6 @@ --- title: Click & Open Tracking -description: '' +description: 'Track when recipients open your e-mails and click links within them.' category: Features --- @@ -8,22 +8,35 @@ Postal supports tracking opens and clicks from e-mails. This allows you to see w - ## How it works -Once enabled, Postal will automatically scan your outgoing messages and replace any links and images with new URLs that go via your Postal web server. When the link is clicked, Postal will log the click and redirect to the user to the original URL automatically. The links that are included in the e-mail should be on the same domain as the sender and therefore you need to configure a subdomain like `click.yourdomain.com` and point it to your Postal server. +Once enabled, Postal will automatically scan your outgoing messages and replace any links with new URLs that go via your Postal web server, and insert a tracking image into HTML messages. When the link is clicked, Postal will log the click and redirect the user to the original URL automatically. The links that are included in the e-mail should be on the same domain as the sender and therefore you need to configure a subdomain like `click.yourdomain.com` and point it to your Postal server. + +For a message to be rewritten, all of the following must be true: + +* The message is an outgoing message with an authenticated sending domain. +* The server has a **tracking domain** attached to that sending domain (e.g. `click.yourdomain.com` for messages from `yourdomain.com`). +* The tracking domain's DNS check has passed - its status must be **OK**. Postal checks for a CNAME record at the tracking domain pointing at the value of `dns.track_domain` in your configuration (`track.postal.example.com` in the [DNS configuration](/getting-started/dns-configuration) examples). Tracking domains are re-checked automatically every hour. +* The message does not have an `X-AMP: skip` header. + +Tracking domains have three options which are all enabled by default: + +* **SSL** - rewritten URLs use `https://`. You must have a valid certificate for the tracking domain on your web proxy. It is **highly** recommended to leave this enabled as anything else is likely to cause problems with reputation and user experience. +* **Track loads** - a 1×1 pixel image is inserted before the closing `` tag of HTML parts. When loaded, a `MessageLoaded` webhook is sent. +* **Track clicks** - `http://` and `https://` URLs in plain text parts and `href` attributes in HTML parts are replaced with tracking URLs. When clicked, a `MessageLinkClicked` webhook is sent. ## Configuring your web server -To avoid messages being marked as spam, it's important that the subdomain that Postal uses in the re-written URLs is on the same domain as that sending the message. This means if you are sending mail from `yourdomain.com`, you'll need to setup `click.yourdomain.com` (or whatever you choose) to point to your Postal server. +To avoid messages being marked as spam, it's important that the subdomain that Postal uses in the re-written URLs is on the same domain as that sending the message. This means if you are sending mail from `yourdomain.com`, you'll need to set up `click.yourdomain.com` (or whatever you choose) to point to your Postal server. -There are two ways how to achive that traffic to `click.yourdomain.com` will reach Postal: -1. Adding CNAME record for `click.yourdomain.com` to previously configured `track.postal.example.com` followed by Caddy additional configuration. -2. Configuring custom proxy for `click.yourdomain.com` on your webserver. +There are two parts to this: + +1. Add a CNAME record for `click.yourdomain.com` pointing to your configured `dns.track_domain` (e.g. `track.postal.example.com`). This is what Postal's DNS check looks for. +2. Configure your web proxy to send requests for `click.yourdomain.com` to the Postal web server with the `X-Postal-Track-Host: 1` header added. Postal uses this header (and nothing else - the `Host` header is not checked) to decide that a request is a tracking request. ### Additional Caddy configuration -If you used Caddy as proxy for overall Postal traffic it easy to add additional proxy its config `/opt/postal/config/Caddyfile`: +If you used Caddy as the proxy for overall Postal traffic it is easy to add an additional proxy to its config `/opt/postal/config/Caddyfile`: ``` # ... previous content @@ -48,15 +61,25 @@ Once you have configured this, you should be able to visit your chosen domain in ### Setting up tracking domain If you're happy things are working, you can enable tracking as follows: -1. Find the web server you wish to enable tracking on in the Postal web interface +1. Find the mail server you wish to enable tracking on in the Postal web interface 2. Go to the **Domains** item 3. Select **Tracking Domains** 4. Click **Add a tracking domain** -5. Enter the domain that you have configured and choose the configuration you want to use. It is **highly** recommended that you use SSL for these connections. Anything else is likely to cause problems with reputation and user experience. +5. Enter the subdomain (a single label such as `click`) and select the sending domain it belongs to, then choose the options you want to use. + +Postal will check the CNAME record immediately. You can re-run the check with the **Check DNS** button; tracking is only applied to messages once the status is **OK**. + +## Requests served by the tracking host + +| Path | Behaviour | +|---|---| +| `/img/{server-token}/{message-token}` | Records an open (once per request) and returns the 1×1 PNG. | +| `/{server-token}/{link-token}` | Records the click and responds with a `307` redirect to the original URL. Returns `404 Link not found` for an unknown token. | +| Anything else (including `/`) | Returns `200 Hello.` - useful for checking your proxy configuration. | ## Disabling tracking on a per e-mail basis -If you don't wish to track anything in an email you can add a header to your e-mails before sending it. +If you don't wish to track anything in an email you can add a header to your e-mails before sending it. No links or images will be rewritten. ```text X-AMP: skip @@ -64,7 +87,7 @@ X-AMP: skip ## Disabling tracking for certain link domains -If there are certain domains you don't wish to track links from, you can define these on the tracking domain settings page. For example, if you list `yourdomain.com` no links to this domain will be tracked. +If there are certain domains you don't wish to track links from, you can define these on the tracking domain settings page (one per line). The host of each link is compared exactly against this list, so `yourdomain.com` excludes links to `https://yourdomain.com/...` but not `https://www.yourdomain.com/...` - list each host separately. ## Disabling tracking on a per link basis diff --git a/content/3.features/health-metrics.md b/content/3.features/health-metrics.md index 100da03..157be92 100644 --- a/content/3.features/health-metrics.md +++ b/content/3.features/health-metrics.md @@ -1,26 +1,53 @@ --- title: Health & Metrics -description: '' +description: 'Monitor the health of Postal processes and scrape Prometheus metrics.' category: Features --- -The Postal worker and SMTP server processes come with additional functionality that allows you to monitor the health of the process as well as look at live metrics about their performance. +The Postal worker and SMTP server processes come with additional functionality that allows you to monitor the health of the process as well as look at live metrics about their performance. + +The web server does not run a health server; use your web proxy or an HTTP check against the login page for that process. ## Port numbers -By default, the health server listens on a different port for each type of process. +By default, the health server listens on `127.0.0.1` on a different port for each type of process. -* Worker - listens on port `9090` -* SMTP server - listens on port `9091` +* Worker - listens on port `9090` (`worker.default_health_server_port` / `worker.default_health_server_bind_address`) +* SMTP server - listens on port `9091` (`smtp_server.default_health_server_port` / `smtp_server.default_health_server_bind_address`) Unlike other services, if these ports are in use when the process starts, the health server will simply not start but the rest of the process will run as normal. This will be shown in the logs. -To configure these ports you can set the `HEALTH_SERVER_PORT` and `HEALTH_SERVER_BIND_ADDRESS` environment variables. +To override these for an individual process (for example when running several workers on one host) you can set the `HEALTH_SERVER_PORT` and `HEALTH_SERVER_BIND_ADDRESS` environment variables. + +## Endpoints + +| Path | Response | +|---|---| +| `/health` | `OK` when the process is running. This can be used for health check monitoring. | +| `/metrics` | Metrics in the standard Prometheus text exposition format. | +| `/` | The process name, PID and hostname, e.g. `worker (pid: 12, host: postal1)`. | ## Metrics -The metrics are exposed at `/metrics` and are in a standard Prometheus exporter format. This means they can be scraped by any tool that can ingest Prometheus metrics. This will then allow them to be turned in to graphs as appropriate. +The metrics are exposed at `/metrics` and are in a standard Prometheus exporter format. This means they can be scraped by any tool that can ingest Prometheus metrics. + +### SMTP server + +| Metric | Type | Labels | Description | +|---|---|---|---| +| `postal_smtp_server_connections_total` | counter | | The number of connections made to the SMTP server. | +| `postal_smtp_server_tls_connections_total` | counter | | The number of successful TLS (STARTTLS) connections established. | +| `postal_smtp_server_exceptions_total` | counter | `type`, `error` | The number of server exceptions encountered. `type` is `client-accept` or `data`; `error` is the exception class. | +| `postal_smtp_server_commands_total` | counter | `command` | The number of key commands received (`EHLO`, `HELO`, `RSET`, `AUTH PLAIN`, `AUTH LOGIN`, `AUTH CRAM-MD5`, `STARTLS`, `PROXY`). | +| `postal_smtp_server_client_errors` | counter | `error` | The number of error responses sent to clients, for example `invalid-credentials`, `authentication-required`, `message-too-large`, `from-name-invalid`, `route-rejected`, `server-suspended`, `loop-detected`. | +| `postal_smtp_server_messages_total` | counter | `type`, `tls` | The number of messages accepted. `type` is `outgoing`, `incoming` or `bounce`; `tls` is `yes` or `no`. | -## Health checks +### Worker -The `/health` endpoint will return "OK" when the process is running. This can be used for health check monitoring. +| Metric | Type | Labels | Description | +|---|---|---|---| +| `postal_worker_job_executions` | counter | `thread`, `job` | The number of jobs worked where work was completed. `job` is `ProcessQueuedMessagesJob` or `ProcessWebhookRequestsJob`. | +| `postal_worker_job_runtime` | histogram | `thread`, `job` | The time taken to process jobs (in seconds). | +| `postal_worker_errors` | counter | `error` | The number of errors encountered while processing jobs, labelled by exception class. | +| `postal_worker_task_runtime` | histogram | `task` | The time taken to run each [scheduled task](/other/workers-and-background-tasks#scheduled-tasks) (in seconds). | +| `postal_message_queue_latency` | histogram | | The length of time between a message being queued and being dequeued (in seconds). A rising value indicates you need more worker capacity. | diff --git a/content/3.features/ip-pools.md b/content/3.features/ip-pools.md index d98bdd6..55a177b 100644 --- a/content/3.features/ip-pools.md +++ b/content/3.features/ip-pools.md @@ -1,27 +1,68 @@ --- title: IP Pools -description: '' +description: 'Send messages from different IP addresses based on server, sender or recipient.' category: Features --- Postal supports sending messages from different IP addresses. This allows you to configure certain sets of IPs for different mail servers or send from different IPs based on the sender or recipient addresses. ## Enabling IP pools -By default, IP pools are disabled and all email is sent from any IP address on the host running the workers. To use IP pools, you'll need to enable them in the configuration file. You can do this by setting the following in your `postal.yml` configuration file. You'll then need to restart Postal using `postal stop` and `postal start`. +By default, IP pools are disabled and all email is sent from whichever address the host's routing table selects. To use IP pools, you'll need to enable them in the configuration file. You can do this by setting the following in your `postal.yml` configuration file. You'll then need to restart Postal using `postal stop` and `postal start`. ```yaml postal: use_ip_pools: true ``` +::callout{icon="i-heroicons-information-circle"} +The worker binds outgoing SMTP connections to the exact IP addresses you configure, and it discovers which addresses it can use by inspecting the network interfaces of the host (or container) it runs in. This means the addresses must be configured on the worker's own network interfaces - in Docker terms, workers must run with host networking (which the standard installation does). +:: + ## Configuring IP pools -Once you have enabled IP pools, you'll need to set them up within the web interface. You'll see an **IP Pools** link in the top right of the interface. From here you can add pools and then add IP addresses within them. +Once you have enabled IP pools, you'll need to set them up within the web interface as a global administrator. You'll see an **IP Pools** link in the top right of the interface. From here you can add pools and then add IP addresses within them. + +### IP addresses + +Each IP address in a pool has the following attributes: + +* **IPv4 address** - required and unique across all pools. +* **IPv6 address** - optional. If a recipient's mail server is only reachable over IPv6 and the selected address has no IPv6 address, that mail server will be skipped. +* **Hostname** - required. This is used as the `HELO`/`EHLO` hostname when Postal connects to remote mail servers from this address. It should match the reverse DNS (PTR) record for the address - Postal does not manage PTR records for you. +* **Priority** - an integer from 0 to 100 (default 100) which weights how often this address is chosen relative to others in the same pool. For example, with three addresses at priorities 1, 50 and 100, the priority 1 address receives a tiny percentage of mail, priority 50 roughly a third and priority 100 roughly two thirds. An address with priority 0 is never selected. + +### Assigning pools to organizations and servers + +Once an IP pool has been added, you'll need to assign it to any organization that should be permitted to use it. Open up the organization and choose **IPs** and then tick the pools you want to allocate. A pool marked as the **default** pool is automatically allocated to every newly created organization. + +Once allocated to an organization, you can assign the IP pool to servers from the server's **Settings** page. All outgoing mail from that server will use the pool unless an IP pool rule says otherwise. + +### IP pool rules -Once an IP pool has been added, you'll need to assign it any organization that should be permitted to use it. Open up the organization and choose **IPs** and then tick the pools you want to allocate. +Rules allow individual messages to be sent from a different pool based on their sender or recipient. Rules can be created at the organization level (**IP Rules** in the organization menu) or on a specific server (**IP Rules** in the server settings). Each rule has: -Once allocated to an organization, you can assign the IP pool to servers from the server's **Settings** page. You can also use the IP pool to configure IP rules for the organization or server. +* **To addresses** - a list (one per line) of addresses or domains matched against the recipient (`RCPT TO`) of the message. +* **From addresses** - a list of addresses or domains matched against the `From` header of the message. +* The **IP pool** to use when the rule matches. + +A rule matches if *any* entry in either list matches. An entry containing `@` must match the full address exactly (any `+tag` in the recipient's address is ignored). An entry without `@` must match the domain of the address exactly. Wildcards and subdomain matching are not supported. + +Rules are evaluated in this order and the first match wins: + +1. The server's own rules, newest first. +2. The organization's rules, newest first. +3. The server's assigned IP pool. + +If none of these produce a pool the message is sent without a fixed source address. + +## How addresses are allocated + +When a message is queued, Postal selects an IP address from the resulting pool using the priorities above and stores it against the queued message. All retries for that message use the same address. When several queued messages to the same destination domain are batched together they must also share the same address. + +Each worker process only picks up queued messages whose allocated address is configured on the host it is running on (or messages with no allocated address). ::callout{icon="i-heroicons-exclamation-triangle" color="amber"} -It's very important to make sure that the IP addresses you add in the web interface are actually configured on your Postal servers. If the IPs don't exist on the server, message delivery may fail or messages will not be dequeued correctly. +It's very important to make sure that the IP addresses you add in the web interface are actually configured on a host running a Postal worker. If an address is not present on any worker host, messages allocated to it will sit in the queue indefinitely and never be processed. Likewise, removing an address from a host (or from Postal) while messages are queued for it will leave those messages stranded. :: + +Removing an IP pool is only possible once it has no addresses and no servers assigned to it. diff --git a/content/3.features/logging.md b/content/3.features/logging.md index 02979cd..ae9f481 100644 --- a/content/3.features/logging.md +++ b/content/3.features/logging.md @@ -1,19 +1,54 @@ --- title: Logging -description: '' +description: 'How Postal logs and how to configure where logs are sent.' category: Features --- -All Postal processes log to STDOUT and STDERR which means their logs are managed by whatever engine is used to run the container. In the default case, this is Docker. +All Postal processes log to STDOUT and STDERR which means their logs are managed by whatever engine is used to run the container. In the default case, this is Docker, so you can view logs with `postal logs` (or `postal logs smtp` for a single service). + +## Log configuration options + +The following options in the `logging` section of `postal.yml` control what is logged. + +```yaml +logging: + # Enable the Postal logger to log to STDOUT (default true). When false, Postal's + # own log lines are discarded. + enabled: true + # Enable the standard Rails request logger for the web server (default false). + rails_log_enabled: false + # Enable ANSI colour highlighting of log lines (default false). + highlighting_enabled: false + # A DSN which should be used to report exceptions to Sentry. When set, exceptions + # raised in any process are reported to Sentry. + sentry_dsn: +``` + +Each log line includes the component that produced it (for example `smtp-server`, `worker`, `health-server`) and, where relevant, a `trace_id` which allows you to follow a single SMTP connection or message through the logs. + +### SMTP server logging + +The SMTP server logs every command it receives and every response it sends for each connection, identified by the connection's trace ID. Two further options in the `smtp_server` section can be used to tune this. + +```yaml +smtp_server: + # Log a line whenever a connection is opened or closed (default false). + log_connections: false + # A regular expression matched against the client IP address. Connections from + # matching addresses are not logged at all. Useful for excluding load balancer + # health checks, e.g. "^10\\.0\\.0\\." + log_ip_address_exclusion_matcher: +``` + +By default, the SMTP server stops logging a connection's traffic once the `DATA` command is received so that message content is not written to the logs. A global administrator can enable **Log SMTP data** in a server's **Advanced Settings** to log the full message content for connections authenticated by that server's credentials. This is intended for debugging only. ## Redirecting logs to the host syslog If you want to send your log data to the host system's syslog then you can configure this. This is useful if you wish to use external tools like `fail2ban` to block users from accessing your system. -The quickest way to achieve this is to use a docker compose overide file in `/opt/postal/install/docker-compose.override.yml`. The contents of this file, would contain the following: +The quickest way to achieve this is to use a docker compose override file in `/opt/postal/install/docker-compose.override.yml`. The contents of this file would contain the following: ```yaml -version: "3.9" services: smtp: logging: @@ -26,7 +61,7 @@ If you wanted to put worker and web server logs there too, you can define those. ## Limiting the size of logs -Docker cam be configured to limit the size of the log files it stores. To avoid storing large numbers of log files, you should configure this appropriately. This can be achieved by setting a maximum size in your `/etc/docker/daemon.json` file. +Docker can be configured to limit the size of the log files it stores. To avoid storing large numbers of log files, you should configure this appropriately. This can be achieved by setting a maximum size in your `/etc/docker/daemon.json` file. ```json { @@ -39,12 +74,12 @@ Docker cam be configured to limit the size of the log files it stores. To avoid ## Sending logs to Graylog -Postal includes support for sending log output to a central Graylog server over UDP. This can be configured using the following options: +Postal includes support for sending log output to a central Graylog (or any GELF-capable) server over UDP. This is enabled by setting a `gelf.host`; logs continue to be written to STDOUT as well. ```yaml gelf: # GELF-capable host to send logs to - host: + host: # GELF port to send logs to port: 12201 # The facility name to add to all log entries sent to GELF diff --git a/content/3.features/mail-server-settings.md b/content/3.features/mail-server-settings.md new file mode 100644 index 0000000..d93c72b --- /dev/null +++ b/content/3.features/mail-server-settings.md @@ -0,0 +1,101 @@ +--- +title: Mail Server Settings +description: 'Modes, limits, retention, held messages, the suppression list and other per-server settings.' +category: Features +--- + +Each mail server within an organization has its own settings, found under **Settings** in the server menu. Some settings are only visible to [global administrators](/features/users-and-permissions) under **Advanced Settings**. + +## Server settings + +| Setting | Description | +|---|---| +| **Name** | A display name, unique within the organization. | +| **Permalink** | A short identifier (letters, digits and hyphens) used in SMTP usernames (`org-permalink/server-permalink`) and URLs. It cannot be changed after the server is created. | +| **Mode** | `Live` or `Development`. See below. | +| **IP pool** | Only shown when [IP pools](/features/ip-pools) are enabled. The pool that outgoing mail from this server is sent from unless an IP pool rule matches. | +| **Postmaster** | The contact address included in bounce messages Postal generates when an incoming message cannot be delivered. Defaults to `postmaster@` the message's domain. | + +### Live and Development mode + +In **Live** mode all mail is routed normally. In **Development** mode every outgoing and incoming message is placed in the held queue with the note "Server is in development mode." instead of being delivered to recipients or endpoints. Messages are still parsed, inspected and visible in the web interface, and count towards the server's send limit. Individual held messages can be released manually from the web interface, which delivers them despite the mode. + +If you only want to hold messages from a particular application or environment rather than the whole server, set the **hold** option on that application's [credential](/features/smtp-authentication#holding-messages-from-a-credential) instead. + +## Spam + +The **Spam threshold** and **Spam failure threshold** for incoming mail are set here. See [Spam & Virus Checking](/features/spam-and-virus-checking#classifying-spam). + +## Retention + +Each server has three retention settings which are shown on the **Retention** page and can be changed by a global administrator under **Advanced Settings**. They are enforced by a background task which runs once a day at 03:00 (server time, normally UTC). + +| Setting | Default | Description | +|---|---|---| +| **Raw message retention days** | 30 | How many days the raw content of messages (headers, bodies and attachments) is kept. Raw data is stored in one table per day, and whole days are removed once they are older than this. After removal the message still appears in lists and searches but its content and attachments can no longer be viewed, and a queued message whose raw data has been removed will fail with "Raw message has been removed". | +| **Raw message retention size** | 2048 MB | The total disk space raw message data may use. When exceeded, whole days are removed starting with the oldest until usage is under the limit. | +| **Message retention days** | 60 | How many days message metadata (the message record itself, its deliveries, clicks, loads and spam checks) is kept. Older messages are deleted entirely. | + +Leaving any of these blank disables that limit ("Indefinitely" / "No limit"). The **Retention** page also shows the current total size of the server's message database. + +## Send limit + +A global administrator can set a **Send limit** for a server under **Advanced Settings**. This is the maximum number of outgoing messages accepted in a rolling 60 minute window; the current usage is shown on the **Send Limit** page. Incoming messages are counted but not limited. + +* When the volume reaches **90%** of the limit, the server is marked as *approaching* its limit. +* When the volume reaches the limit, every further outgoing message is held with the note "Message held because send limit (N) has been reached." until the volume drops. Releasing a held message while the server is still over the limit holds it again. + +Once a minute Postal checks for servers that have recently approached or exceeded their limit and, at most once per hour for each state, e-mails every user in the organization and triggers the `SendLimitApproaching` / `SendLimitExceeded` [webhook events](/developer/webhooks#send-limit-events). The e-mails are sent using the `smtp` section of your Postal configuration. + +## Held messages + +Messages that are held are not delivered but remain visible under **Messages → Held** where they can be released (re-queued for delivery) or the hold cancelled. Each held message triggers a `MessageHeld` webhook. A message may be held for any of the following reasons: + +| Reason | Details recorded | Can be released manually? | +|---|---|---| +| Server is in Development mode | "Server is in development mode." | Yes | +| Credential is set to hold | "Credential is configured to hold all messages authenticated by it." | Yes | +| Recipient is on the suppression list | "Recipient (…) is on the suppression list (reason: …)" | Yes | +| Send limit reached | "Message held because send limit (…) has been reached." | Only once the volume has dropped below the limit | +| Server or organization is suspended | "Mail server has been suspended…" | No - it will be held again until unsuspended | +| Incoming spam on a route set to Quarantine | "Message placed into quarantine." | Yes | +| Incoming mail on a route set to Hold | "Message has been accepted but not sent to any endpoints." | Yes (marked as Processed) | + +Held messages expire after the number of days set by `postal.default_maximum_hold_expiry_days` (default **7**). An hourly task cancels the hold on expired messages, recording a `HoldCancelled` delivery with the note "The hold on this message has been removed without action." The message is not delivered. + +## Suppression list + +Each server maintains a suppression list of recipient addresses that Postal will not deliver to, viewable under **Messages → Suppressions**. Addresses are added automatically when Postal is sending **outgoing** mail: + +* **Too many hard fails** - a permanent (`5xx`) rejection from the recipient's mail server when there has already been at least one other hard fail to the same address in the previous 24 hours. +* **Too many soft fails** - the message has been retried the maximum number of times (`postal.default_maximum_delivery_attempts`, default 18) without success. + +Incoming bounce messages do **not** add addresses to the list. + +While an address is on the list, new messages to it are held (see above) rather than attempted. Releasing such a message manually bypasses the list, and if the delivery then succeeds the address is removed from the list. Entries are otherwise removed automatically after `postal.default_suppression_list_automatic_removal_days` (default **30**) days. + +## Advanced settings (administrators only) + +| Setting | Description | +|---|---| +| **Send limit** | See above. | +| **Allow sender header** | Permits any `From` address as long as a `Sender` header contains an address on one of the server's verified domains. See [From/Sender validation](/features/smtp-authentication#fromsender-validation). Applies to SMTP and the API. | +| **Privacy mode** | When enabled, the `Received` header Postal adds to submitted messages omits the submitting client's IP address, reverse DNS hostname and HELO name, leaving only `by {hostname} with SMTP/HTTP; {date}`. | +| **Log SMTP data** | Log the full content of messages submitted using this server's credentials in the SMTP server log. Debugging only. See [Logging](/features/logging#smtp-server-logging). | +| **Outbound spam threshold** | Enables spam scanning of outgoing messages; messages scoring at or above this are failed. Blank disables outbound scanning. See [Spam & Virus Checking](/features/spam-and-virus-checking). | +| **Message retention days**, **Raw message retention days**, **Raw message retention size** | See [Retention](#retention). | + +### Suspending a server + +An administrator can **suspend** a server from **Advanced Settings** by entering a reason. While suspended: + +* Every message processed for the server (incoming and outgoing) is held with the reason "Mail server has been suspended". +* The SMTP server rejects `RCPT TO` for the server's routes and credentials with `535 Mail server has been suspended`. +* API requests using the server's credentials return the `ServerSuspended` error. +* All users in the organization are e-mailed about the suspension. + +Organizations also carry a suspension flag which suspends all of their servers at once; there is no interface for this, but it can be set from `postal console` (`Organization.find_by(permalink: "my-org").update(suspended_at: Time.now)`). Use **Unsuspend server** to restore normal operation; held messages must then be released manually. + +## Deleting a server + +Deleting a server (**Settings → Delete**, which requires typing the server's name to confirm) marks it as deleted immediately and hides it from the interface. The server's message database is dropped by an hourly background task shortly afterwards. This cannot be undone. diff --git a/content/3.features/oidc.md b/content/3.features/oidc.md index a4552cd..97ce665 100644 --- a/content/3.features/oidc.md +++ b/content/3.features/oidc.md @@ -1,48 +1,49 @@ --- title: OpenID Connect -description: '' +description: 'Delegate authentication to an external OpenID Connect identity provider.' category: Features --- Postal supports OpenID Connect (OIDC) allowing you to delegate authentication to an external service. When enabled, there are various changes: * You are not required to enter a password when you add new users. -* When a user first logs in with OIDC, they will be matched to a local user based on their e-mail address. -* On subsequent logins, the user will be matched based on their unique identifier provided by the OIDC issuer. +* When a user logs in with OIDC, Postal first looks for a local user that has previously been linked to the identity provided (matched on the unique identifier from the OIDC issuer). If none is found, it looks for a local user with a matching e-mail address that has **not** yet been linked to any OIDC identity. +* When a user is matched, their local account is linked to the OIDC identity. Their e-mail address and name are updated from the identity provider and **any local password is removed**. +* If no matching user is found, login is refused with the message "No user was found matching your identity. Please contact your administrator." Postal does not create users automatically. * Users without local passwords cannot reset their password through Postal. -* Users cannot change their local password when associated with an OIDC identity. -* Existing users that currently have a password will continue to be able to use that password until it is linked with an OIDC identity. +* Users cannot change their local password once associated with an OIDC identity. +* Existing users that currently have a password will continue to be able to use that password until they log in with OIDC and are linked. ![Screenshot](/screenshots/oidc.png) ## Configuration -To get started, you'll need to find an OpenID Connect enabled provider. You should create your application within the provider in order to obtain a identifier (client ID) and a secret (client secret). +To get started, you'll need to find an OpenID Connect enabled provider. You should create your application within the provider in order to obtain an identifier (client ID) and a secret (client secret). -You may be prompted to provide a "redirect URI" during this process. You should enter `https://postal.yourdomain.com/auth/oidc/callback`. +You may be prompted to provide a "redirect URI" during this process. You should enter `https://postal.yourdomain.com/auth/oidc/callback` (using your configured `postal.web_protocol` and `postal.web_hostname`). -Finally, you'll need to place your configuration in the Postal config file as normal. +Finally, you'll need to place your configuration in the Postal config file as normal and restart Postal. ```yaml oidc: # Start by enabling OIDC for your installation. enabled: true - - # The name of the OIDC provider as shown in the UI. For example: - # "Login with My Proivder". + + # The name of the OIDC provider as shown in the UI. For example: + # "Login with My Provider". name: My Provider - - # The OIDC issuer URL provided to you by your Identity provider. + + # The OIDC issuer URL provided to you by your Identity provider. # The provider must support OIDC Discovery by hosting their configuration # at https://identity.example.com/.well-known/openid-configuration. issuer: https://identity.example.com - + # The client ID for OIDC identifier: abc1234567890 # The client secret for OIDC secret: zyx0987654321 - + # Scopes to request from the OIDC server. You'll need to find these from your # provider. You should ensure you request enough scopes to ensure the user's # email address is returned from the provider. @@ -51,31 +52,55 @@ oidc: - email ``` -If your Identity Provider does not support OpenID Connect discovery (which is enabled by default, you can manually configure it.) For full details of the options available see the [example config file](https://github.com/postalserver/postal/blob/main/doc/config/yaml.yml). +### Field mapping + +By default, Postal will look for the user's unique identifier in the `sub` field, their e-mail address in the `email` field and their name in the `name` field of the user info returned by the provider. These can be overridden if these values can be found elsewhere. + +```yaml +oidc: + # ... + uid_field: sub + email_address_field: email + name_field: name +``` + +### Providers without discovery -By default, Postal will look for an email address in the `email` field and a name in the `name` field. These can be overriden using configuration if these values can be found elsewhere. +If your identity provider does not support OpenID Connect discovery (which is enabled by default), you can disable discovery and configure each endpoint manually. + +```yaml +oidc: + # ... + discovery: false + authorization_endpoint: https://identity.example.com/oauth2/authorize + token_endpoint: https://identity.example.com/oauth2/token + userinfo_endpoint: https://identity.example.com/oauth2/userinfo + jwks_uri: https://identity.example.com/oauth2/jwks +``` + +For the full list of options see the [configuration reference](/getting-started/configuration-reference#oidc). ## Logging in -Once enabled, you can log in by pressing the **Login with xxx** button on the login page. This will direct you to your chosen identity provider. Once authorised, you will be directed back to the application. If a user exists matching the e-mail address returned by the OpenID provider, it will be linked and you will be logged in. If not, an error will be displayed. +Once enabled, you can log in by pressing the **Login with xxx** button on the login page. This will direct you to your chosen identity provider. Once authorised, you will be directed back to the application and matched to a local user as described above. If the identity provider reports an error, you will be returned to the login page with the message "An issue occurred while logging you in with OpenID". ## Debugging -Details about the user matching process will be displayed in the web server logs when the callback from the Identity provider happens. +Details about the user matching process are written to the web server logs when the callback from the identity provider happens. This includes the full set of claims received from the provider and which lookup (by UID or by e-mail address) succeeded or failed. ## Disabling local authentication -Once you have established your OpenID Connect set up, you can fully disable local authentication. This will change the login page as well as user management options. +Once you have established your OpenID Connect set up, you can fully disable local authentication. This removes the e-mail/password form and the password reset link from the login page, and any attempt to log in or reset a password locally is refused with "Local authentication is not enabled". ```yaml -oidc: +oidc: # ... local_authentication_enabled: false ``` ## Using Google as an identity provider -Setting up Postal to authenticate with Google is fairly straight forward. You'll need to use the Google Cloud console to generate a client ID and secret ([see docs](https://developers.google.com/identity/openid-connect/openid-connect)). When prompted for a redirect URI, you should be `https://postal.yourdomain.com/auth/oidc/callback`. The following configuration can be used to enable this: +Setting up Postal to authenticate with Google is fairly straight forward. You'll need to use the Google Cloud console to generate a client ID and secret ([see docs](https://developers.google.com/identity/openid-connect/openid-connect)). When prompted for a redirect URI, you should use `https://postal.yourdomain.com/auth/oidc/callback`. The following configuration can be used to enable this: ```yaml oidc: diff --git a/content/3.features/routing-incoming-email.md b/content/3.features/routing-incoming-email.md new file mode 100644 index 0000000..5dc242b --- /dev/null +++ b/content/3.features/routing-incoming-email.md @@ -0,0 +1,95 @@ +--- +title: Routing Incoming E-Mail +description: 'Routes, endpoints and what happens to mail that arrives at your Postal server.' +category: Features +--- + +As well as sending mail, each Postal mail server can receive mail for the domains you have added to it. Incoming mail is matched to a **route**, and each route decides what should happen to the message - most commonly delivering it to an **endpoint** such as your application's HTTP URL, another SMTP server or an ordinary e-mail address. + +## Getting mail to Postal + +There are two ways to get incoming mail into Postal. + +**Point your MX records at Postal.** Add the MX records from your `dns.mx_records` configuration (for example `mx1.postal.example.com` and `mx2.postal.example.com`, both at priority 10) to the domain. Postal's [domain DNS checks](/features/sending-domains#dns-checks) will show a green tick once they are visible. All mail for addresses on that domain will then arrive at your Postal SMTP server. + +**Forward mail from an existing mail server.** If the domain already has a mail server, you don't need to change any DNS. Every route has a unique forwarding address of the form `{token}@{route domain}` (e.g. `a1b2c3d4@routes.postal.example.com`), shown in the **Address** field when you edit the route. Forward mail from your existing server to this address and it will be treated exactly as if it had been sent to the route's real address. The route domain must have an MX record pointing at Postal - see [DNS configuration](/getting-started/dns-configuration#route-domain). + +## Routes + +Routes are managed under **Routing → Routes** in the server menu. A route consists of: + +* **Name and domain** - the address to route, e.g. `support` @ `yourdomain.com`. The domain must be a verified domain belonging to the server or its organization. The name may be: + * an ordinary local part (lower case letters, digits, `-` and `.`); + * `*` to receive mail for every address on the domain ([wildcards](/other/wildcards-and-address-tags)); + * `__returnpath__` with no domain, to receive mail sent to the server's return path address ([return path routes](/other/auto-responders-and-bounces#return-path-routes)). Only one of these may exist per server and it must deliver to an HTTP endpoint. + + Mail to `name+anything@domain` also matches the route for `name@domain` ([address tags](/other/wildcards-and-address-tags)). A given name/domain combination can only be routed once across your whole installation. + +* **Endpoint** - either one of the server's endpoints, or one of the special modes below. +* **Additional endpoints** - see below. +* **Spam mode** - `Mark`, `Quarantine` or `Fail`. See [Spam & Virus Checking](/features/spam-and-virus-checking#classifying-spam). + +### Route modes + +The **Endpoint** dropdown also offers four special modes in place of a real endpoint: + +| Mode | At `RCPT TO` time | When processed | +|---|---|---| +| **Endpoint** (an HTTP, SMTP or address endpoint) | Accepted | Delivered to the endpoint. | +| **Accept** | Accepted | Recorded as **Processed** with no delivery. The message is stored and visible in the web interface. | +| **Hold** | Accepted | Placed in the held queue (status **Held**). You can release it from the web interface, after which it is marked **Processed**. Held messages expire after `postal.default_maximum_hold_expiry_days` (default 7). | +| **Bounce** | Accepted | Marked as **HardFail** and a bounce is sent back to the sender explaining the message was not delivered. | +| **Reject** | Rejected with `550 Route does not accept incoming messages` | Never stored. | + +A message is only rejected at SMTP time by the **Reject** mode (and by a suspended server). Everything else is accepted, queued, and processed by a worker. + +### Additional endpoints + +A route with a real endpoint can also deliver the same message to any number of additional endpoints. A separate copy of the message (with its own ID and delivery history) is created for each endpoint. Additional endpoints on a wildcard (`*`) route must be HTTP endpoints. Additional endpoints cannot be used with the Accept, Hold, Bounce or Reject modes. + +### Processing order + +When a worker processes an incoming message it performs the following steps in order: + +1. If the message is a bounce for something Postal sent, link it to the original message and stop ([bounces](/other/auto-responders-and-bounces)). +2. Run spam and virus inspection if enabled. If the score is at or above the server's **spam failure threshold**, hard fail. +3. If the server is in **Development** mode, hold the message. +4. Apply the route's spam mode if the message was marked as spam (Quarantine holds, Fail hard fails). +5. Apply the route mode (Accept, Hold, Bounce or deliver to the endpoint). +6. On a hard failure from an endpoint, send a bounce to the sender (unless the endpoint returned `429`). + +## Endpoints + +Endpoints are created under **Routing** in the server menu and can be shared by any number of routes on that server. Deleting an endpoint changes any routes using it to **Reject**. + +### HTTP endpoints + +Deliver the message to your application as an HTTP `POST`. All options and the payload formats are described on the [Receiving e-mail by HTTP](/developer/http-payloads) page. + +### SMTP endpoints + +Forward the message to another SMTP server. + +| Field | Description | +|---|---| +| **Hostname** | The server to connect to. Postal connects directly to this host (no MX lookup is performed). | +| **Port** | Defaults to `25`. | +| **SSL mode** | `None` - never use TLS. `Auto` - use STARTTLS if the server offers it, without verifying the certificate, falling back to plain text if the TLS handshake fails. `TLS` - connect with implicit TLS (e.g. port 465) and verify the certificate. `STARTTLS` - see the note below. | + +::callout{icon="i-heroicons-exclamation-triangle" color="amber"} +In Postal 3.3.7 the STARTTLS option on SMTP endpoints does not work as intended due to a mismatch in the source code (the endpoint stores STARTTLS but the SMTP client checks for STARTLS), so it currently behaves the same as None. Use Auto or TLS until this is fixed. +:: + +The message is forwarded with an envelope sender of `{server token}@{return path domain}` so that bounces come back to Postal, and a `Resent-Sender` header is added if `postal.use_resent_sender_header` is enabled. Connection timeouts are controlled by `smtp_client.open_timeout` and `smtp_client.read_timeout` (default 30 seconds each). Temporary (`4xx`) responses are retried; permanent (`5xx`) responses hard fail and cause a bounce. + +### Address endpoints + +Forward the message to an ordinary e-mail address. Postal looks up the MX records for the address's domain (or uses your configured `postal.smtp_relays` if any) and delivers it as it would an outgoing message, with the `RCPT TO` replaced by the target address. Each address may only be added once per server. + +## Delivery, retries and bounces + +Deliveries to endpoints follow the same rules as outgoing mail: temporary failures are retried with an exponential back-off (`5 minutes × 1.3ⁿ`) up to `postal.default_maximum_delivery_attempts` times (default 18, roughly 31 hours), after which the message is hard failed. A hard failure of an incoming message causes a bounce to be sent to the original sender. Each attempt is recorded on the message's **Activity** tab and triggers the corresponding [webhook](/developer/webhooks#message-status-events). + +## Limits + +The SMTP server accepts messages up to `smtp_server.max_message_size` (default 14 MB). Incoming volume is counted in the server's statistics but is not subject to the server's send limit. diff --git a/content/3.features/sending-domains.md b/content/3.features/sending-domains.md new file mode 100644 index 0000000..b8659ed --- /dev/null +++ b/content/3.features/sending-domains.md @@ -0,0 +1,113 @@ +--- +title: Sending Domains +description: 'Adding and verifying the domains you send mail from, and the DNS records Postal checks.' +category: Features +--- + +Before a mail server can send mail from an address, the address's domain must be added to Postal and **verified**. Postal then checks the domain's SPF, DKIM, MX and return path DNS records and shows the results in the web interface so that you can ensure your mail is delivered reliably. + +::callout{icon="i-heroicons-information-circle"} +This page covers the DNS records for domains you send from. The records that your Postal installation itself needs (the return path domain, SPF include, MX hostnames and so on) are described under DNS configuration. +:: + +## Organization and server domains + +Domains can be added in two places: + +* **Organization domains** (organization menu → **Domains**) are available to every mail server in the organization. +* **Server domains** (server menu → **Domains**) are only available to that server. + +The same domain name may be added at both levels or to several servers, each with its own DKIM key and verification. When an outgoing message is authenticated, Postal looks for a verified domain matching the `From` address's domain, preferring a server-level domain over an organization-level one. + +Only the exact domain is matched - to send from `news.yourdomain.com` you need to add `news.yourdomain.com` as well as `yourdomain.com`. + +## Verifying a domain + +When you add a domain you must prove that you control it. Global administrators skip this step and their domains are verified immediately. Other users choose one of two methods: + +**DNS** - add a TXT record at the domain itself (the apex, `@`) with the value shown, which is `postal-verification {token}` (the prefix is set by `dns.domain_verify_prefix`). Then click **Verify TXT record**. The token is a 32 character random string. + +**E-Mail** - Postal sends a 6 digit code to one of `webmaster@`, `postmaster@`, `admin@`, `administrator@` or `hostmaster@` at the domain (or any of its parent domains). Enter the code to complete verification. This requires the `smtp` section of Postal's configuration to be working. + +Unverified domains cannot be used for sending, in routes, or for tracking domains. + +## DNS checks + +Once verified, open the domain (the **DNS Setup** page) to see the records you need to add. Postal checks these records immediately when you press **Check my records are correct**, and re-checks every domain automatically once an hour. The results are shown as ticks and crosses in the domain list. + +By default Postal queries the domain's own authoritative nameservers so that changes are seen without waiting for caches to expire. Set `postal.use_local_ns_for_domain_verification: true` to use the resolvers from `dns.resolv_conf_path` instead. + +### SPF + +A TXT record at the apex of the domain beginning `v=spf1` which includes your installation's SPF include, e.g. + +```text +v=spf1 a mx include:spf.postal.example.com ~all +``` + +| Status | Meaning | +|---|---| +| `OK` | A `v=spf1` record containing `include:{dns.spf_include}` was found. | +| `Missing` | No `v=spf1` record exists. | +| `Invalid` | An SPF record exists but does not include your Postal SPF include. If you already have an SPF record for another service, add `include:spf.postal.example.com` to it rather than creating a second record. | + +### DKIM + +Postal generates a 1024-bit RSA key pair for each domain when it is added. Publish the public key in a TXT record named `{selector}._domainkey.yourdomain.com`, where the selector is `{dns.dkim_identifier}-{6 random letters}` (for example `postal-KJHDSA._domainkey`). The exact name and value are shown on the DNS Setup page and look like: + +```text +v=DKIM1; t=s; h=sha256; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQ...; +``` + +| Status | Meaning | +|---|---| +| `OK` | Exactly one TXT record exists and its value matches. | +| `Missing` | No TXT record was returned for the name. | +| `Invalid` | Either more than one TXT record exists at the name, or the value does not match the one provided. Check it has been copied exactly. | + +Outgoing messages from a domain whose DKIM status is `OK` are signed with the domain's key (`d=yourdomain.com`). If the DKIM record is not `OK`, messages are still sent but are signed with the installation's key using `d={dns.return_path_domain}` instead - which is why you must also publish the record from `postal default-dkim-record` at `postal._domainkey.{return path domain}` (see [DNS configuration](/getting-started/dns-configuration#return-path)). + +Signatures use `rsa-sha256` with relaxed canonicalisation and cover the `From`, `Sender`, `Reply-To`, `Subject`, `Date`, `Message-ID`, `To`, `Cc`, `MIME-Version`, `Content-Type`, `Content-Transfer-Encoding`, `Resent-*`, `In-Reply-To`, `References` and `List-*` (including `List-Unsubscribe-Post`) headers where present. + +### Return path + +The **return path** is the SMTP envelope sender (`MAIL FROM`) used for outgoing mail, and it is where bounces are sent. By default it is `{server token}@{dns.return_path_domain}`, a hostname belonging to your Postal installation. This is fine, but because the envelope domain differs from your `From` domain it will not give SPF **alignment** for DMARC. + +To fix this, add a CNAME record at `psrp.yourdomain.com` (the prefix is set by `dns.custom_return_path_prefix`) pointing to your installation's return path domain, e.g. + +```text +psrp.yourdomain.com. CNAME rp.postal.example.com. +``` + +Once the check passes, Postal uses `{server token}@psrp.yourdomain.com` as the envelope sender for mail from this domain. Because it is a CNAME, the SPF and DKIM records you published for the return path domain during installation apply automatically, and Postal's SMTP server accepts bounces for any domain beginning with the custom return path prefix. + +| Status | Meaning | +|---|---| +| `OK` | A single CNAME pointing at `{dns.return_path_domain}` was found. Postal will use the custom return path. | +| `Missing` | No record exists. Postal uses the default return path. This is acceptable but not recommended. | +| `Invalid` | A record exists but does not point at the right hostname. | + +### MX + +MX records are only needed if you want to **receive** mail for the domain through Postal (see [Routing incoming e-mail](/features/routing-incoming-email)). Postal checks that every hostname listed in `dns.mx_records` appears among the domain's MX records (case-insensitively). + +| Status | Meaning | +|---|---| +| `OK` | All of your Postal MX hostnames are present. | +| `Missing` | None are present. Incoming mail will not reach Postal, which is fine if you only send. | +| `Invalid` | Some but not all are present. | + +### Overall status and notifications + +A domain is considered fully configured when SPF and DKIM are `OK` and both MX and Return Path are either `OK` or `Missing`. If an automatic hourly check finds a **server-level** domain in any other state, a [`DomainDNSError` webhook](/developer/webhooks#dns-error-event) is triggered. Domains with problems are also highlighted at the top of the server's pages. + +## Sending from any domain + +A server-level domain can be flagged so that the server may send from **any** `From` address once the normal checks have failed. This is intended for trusted internal systems and is shown with an **Any** label in the domain list. There is no interface for setting this flag; an administrator can set it from `postal console`: + +```ruby +org = Organization.find_by(permalink: "my-org") +org.servers.find_by(permalink: "my-server").domains.find_by(name: "yourdomain.com").update(use_for_any: true) +``` + +Messages sent this way are signed with that domain's DKIM key. diff --git a/content/3.features/smtp-authentication.md b/content/3.features/smtp-authentication.md index 3cf62bf..e68d0ab 100644 --- a/content/3.features/smtp-authentication.md +++ b/content/3.features/smtp-authentication.md @@ -1,25 +1,73 @@ --- title: SMTP Authentication -description: '' +description: 'How clients authenticate to the Postal SMTP server to send outgoing mail.' category: Features --- -For sending outgoing emails through the Postal SMTP server you will need to generate a credential through the Postal web interface. This credential is associated with a server and allows you to send mail from any domain associated with that domain (or the organization that owns the domain.) +For sending outgoing emails through the Postal SMTP server you will need to generate a **credential** through the Postal web interface (**Credentials** in the server menu, choose the **SMTP** type). This credential is associated with a server and allows you to send mail from any verified domain associated with that server (or the organization that owns the server). + +The connection details you need are shown on the server's **Help → Sending e-mail** page in the web interface: + +* **Server address** - the value of `postal.smtp_hostname` in your configuration. +* **Port** - the value of `smtp_server.default_port` (default `25`). The SMTP server supports STARTTLS when [SMTP TLS](/features/smtp-tls) is enabled. +* **Username** - `organization-permalink/server-permalink` (this is only checked for `CRAM-MD5`, see below). +* **Password** - the key of your SMTP credential. ## Authentication types -When authenticating to the SMTP server, there are three supported authentication types. +The SMTP server advertises `AUTH CRAM-MD5 PLAIN LOGIN` in response to `EHLO`. There are three supported authentication types. -* `PLAIN` - the credentials are passed in plain text to the server. When using this, you can provide any string as the username (e.g. `x`) and the password should contain your credential string. -* `LOGIN` - the credentials are passed Base64-encoded to the server. As above, you can use anything as the username and the password should contain the credential string (Base64-encoded). -* `CRAM-MD5` - this is a challenge-response mechanism based on the HMAC-MD5 algorithm. Unlike the above two mechanism, the username does matter and should contain the organization and server permalinks separated by a `/` or `_` character. The password used should be the value from your credential. +* `PLAIN` - the credentials are passed in plain text (Base64-encoded) to the server. When using this, you can provide any string as the username (e.g. `x`) and the password should contain your credential key. +* `LOGIN` - the username and password are prompted for in turn, each Base64-encoded. As above, the username is ignored and the password should contain the credential key. +* `CRAM-MD5` - this is a challenge-response mechanism based on the HMAC-MD5 algorithm. Unlike the above two mechanisms, the username does matter and should contain the organization and server permalinks separated by a `/` or `_` character (for example `my-org/my-server`). The shared secret is the credential key. Postal will try every SMTP credential on the named server until one produces the correct response. + +On success the server responds with `235 Granted for {organization}/{server}`. Every successful authentication updates the credential's **last used** time shown in the web interface. ## From/Sender validation -When sending outgoing email through the SMTP server, it is important that the `From` header contains a domain that is owned by the server or its organization. If this it not valid, you will receive a `530 From/Sender name is not valid` error. +When sending outgoing email through the SMTP server, it is important that the `From` header contains a domain that has been added and verified on the server or its organization. If it does not, the message will be rejected at the end of `DATA` with `530 From/Sender name is not valid`. + +The check works as follows: -If you have enabled "Allow Sender Header" for the server, you can include this domain in the `Sender` header instead and any value you wish in the `From` header. +1. Every address in the `From` header is checked. If all of them belong to verified domains, the message is accepted. +2. If that fails and the server has **Allow sender header** enabled (in the server's **Advanced Settings**, administrators only), the `Sender` header is checked in the same way. This allows you to send with any `From` address provided you include a `Sender` header containing an address on one of your domains. +3. An administrator can also mark one server-owned domain as usable for any address; if none of the above match, that domain is used. + +Only the domain part of the address is compared, and it must match exactly (subdomains of a verified domain are not accepted). ## IP-based authentication -Postal has the option to authenticate clients based on their IP address. To use this, you need to create an **SMTP-IP** credential for the IP or network you wish to allow to send mail. Use this carefully to avoid creating an open relay. +Postal has the option to authenticate clients based on their IP address. To use this, you need to create a credential with the type **SMTP-IP** and enter the IP address or CIDR network (IPv4 or IPv6) you wish to allow in the **Network** field. Use this carefully to avoid creating an open relay. + +No `AUTH` command is needed. When an unauthenticated client sends a `RCPT TO` that does not match any incoming route, Postal looks for an SMTP-IP credential whose network contains the client's IP address. If several match, the most specific network (longest prefix) wins. Once matched, the connection is treated exactly as if it had authenticated with that credential, including From/Sender validation. + +Unlike other credential types, the network of an SMTP-IP credential can be edited after it has been created. + +## Holding messages from a credential + +Any credential can be set to **Hold messages from this credential**. All messages submitted using that credential will be placed in the server's held queue rather than being delivered, which is useful for development environments. Held messages can be released from the web interface. See [Mail server settings](/features/mail-server-settings#held-messages). + +## SMTP responses + +The following are the most common responses you may receive from the Postal SMTP server. + +| Response | Meaning | +|---|---| +| `235 Granted for org/server` | Authentication succeeded. | +| `535 Invalid credential` | `PLAIN`/`LOGIN`: the password did not match any SMTP credential. | +| `535 Denied` | `CRAM-MD5`: the username did not match a server, or no credential produced the expected response. | +| `535 Authenticated failed - protocol error` | `PLAIN`: the Base64 payload did not contain both a username and a password. | +| `535 Mail server has been suspended` | The server (or its organization) that the recipient or credential belongs to has been suspended. | +| `530 Authentication required` | The recipient does not match any route on this installation and the client has not authenticated. | +| `530 From/Sender name is not valid` | See [From/Sender validation](#fromsender-validation). | +| `503 EHLO/HELO first please` | `MAIL FROM` was sent before `EHLO`/`HELO`. | +| `501 Invalid RCPT TO` | The recipient address has no domain part. | +| `550 Invalid server token` / `550 Invalid route token` | Mail to the return path or route domain used an unknown token. | +| `550 Route does not accept incoming messages` | The matching route is set to **Reject**. | +| `550 Loop detected` | The message has already passed through this Postal server more than four times. | +| `552 Message too large (maximum size NMB)` | The message exceeds `smtp_server.max_message_size` (default 14 MB). The size is checked when the whole message has been received. | +| `502 TLS not available` | `STARTTLS` was requested but `smtp_server.tls_enabled` is false. | + +## Line endings + +Postal only accepts the RFC-compliant `.` sequence as the end of message data (to prevent SMTP smuggling). Clients that send bare `` line endings will find that their messages never complete. A warning is logged when a line without `` is received. diff --git a/content/3.features/smtp-tls.md b/content/3.features/smtp-tls.md index 1ae3b77..82e5df1 100644 --- a/content/3.features/smtp-tls.md +++ b/content/3.features/smtp-tls.md @@ -1,18 +1,24 @@ --- title: SMTP TLS -description: '' +description: 'Enable STARTTLS on the Postal SMTP server.' category: Features --- By default, Postal's SMTP server is not TLS enabled however you can enable it by generating and providing a suitable certificate. We recommend that you use a certificate issued by a recognised certificate authority, but this isn't essential to use this feature. +::callout{icon="i-heroicons-information-circle"} +Postal supports opportunistic TLS using the STARTTLS command on its normal port. It does not provide an implicit TLS ("SMTPS", port 465) listener. If you need implicit TLS you will need to terminate it with a separate proxy in front of Postal. +:: + ## Key & certificate locations -Certificates should be placed in your `/opt/postal/config` directory. +Certificates should be placed in your `/opt/postal/config` directory (which is mounted at `/config` inside the containers). By default Postal looks for: * `/opt/postal/config/smtp.key` - the private key in PEM format * `/opt/postal/config/smtp.cert` - the certificate in PEM format +The certificate file may contain a full chain: the first certificate in the file is used as the server certificate and any further certificates are sent as the intermediate chain. + ### Generating a self signed certificate You can use the command below to generate a self-signed certificate. @@ -29,10 +35,24 @@ Once you have a key and certificate you will need to enable TLS in the configura smtp_server: # ... tls_enabled: true - # tls_certificate_path: other/path/to/cert/within/container - # tls_private_key_path: other/path/to/cert/within/container + # Paths are relative to the container. $config-file-root expands to the + # directory containing postal.yml (/config in the standard installation). + # tls_certificate_path: $config-file-root/smtp.cert + # tls_private_key_path: $config-file-root/smtp.key + # An OpenSSL cipher list to restrict the ciphers offered # tls_ciphers: + # The OpenSSL SSL/TLS version to use (SSLv23 negotiates the best available) # ssl_version: SSLv23 ``` -You will need to run `postal restart` if you change the configuration or your key/certificate. +Once enabled, the SMTP server advertises `STARTTLS` in its `EHLO` response and clients can upgrade the connection. Whether a message was received over TLS is recorded on the message (`received_with_ssl`) and in the `postal_smtp_server_tls_connections_total` [metric](/features/health-metrics). + +The certificate and key are read once when the SMTP server starts, so you will need to run `postal restart` if you change the configuration or renew your key/certificate. + +## Verifying + +You can check your certificate from the outside with: + +```bash +openssl s_client -connect postal.yourdomain.com:25 -starttls smtp +``` diff --git a/content/3.features/spam-and-virus-checking.md b/content/3.features/spam-and-virus-checking.md index 2bac085..e70bcf5 100644 --- a/content/3.features/spam-and-virus-checking.md +++ b/content/3.features/spam-and-virus-checking.md @@ -1,17 +1,49 @@ --- title: Spam & Virus Checking -description: '' +description: 'Integrate SpamAssassin, rspamd and ClamAV to scan messages passing through your mail servers.' category: Features --- -Postal can integrate with SpamAssassin and ClamAV to automatically scan incoming and outgoing messages that pass through mail servers. +Postal can integrate with SpamAssassin, rspamd and ClamAV to automatically scan incoming and outgoing messages that pass through mail servers. ::callout{icon="i-heroicons-exclamation-triangle" color="amber"} This functionality is disabled by default. :: +## How it works + +When a message is inspected, Postal sends it to each enabled inspector and collects the results: + +* **Spam checking** is performed by either rspamd or SpamAssassin. Each check returns a list of rules that matched along with a score for each; the message's spam score is the sum of these. If both `rspamd` and `spamd` are enabled, **only rspamd is used**. +* **Virus checking** is performed by ClamAV. This sets a "threat" flag on the message (and records the name of the threat) but does **not** contribute to the spam score. A detected threat does not, on its own, cause a message to be held or failed - it is exposed to you through the `X-Postal-Threat` header, the web interface and the API so that your application can decide what to do. + +Inspection results are shown on the **Spam Checks** tab of each message in the web interface. + +**Incoming messages** are always inspected when at least one inspector is enabled. + +**Outgoing messages** are only inspected when an administrator has set an **Outbound spam threshold** for the server (found under the server's **Advanced Settings**). If the score is greater than or equal to this threshold, the message is failed with the details "Message is likely spam". Leave this blank to disable outgoing inspection entirely. + +## Setting up rspamd + +rspamd is the recommended spam checker. Postal communicates with rspamd's HTTP worker (normally on port 11334) using the `/checkv2` endpoint. Install and configure rspamd following [its own documentation](https://rspamd.com/doc/), then enable it in your `postal.yml` and restart Postal. + +```yaml +rspamd: + enabled: true + host: 127.0.0.1 + port: 11334 + # Use HTTPS when connecting to rspamd + ssl: false + # If your rspamd controller requires a password + password: + # Any flags to pass in the Flags header (see the rspamd documentation) + flags: +``` + +When scanning outgoing messages, Postal tells rspamd that the message is outbound so that checks which are not relevant to locally submitted mail are skipped. + ## Setting up SpamAssassin -By default, Postal will talk to SpamAssassin's `spamd` using an TCP socket connection (port 783). You'll need to install SpamAssassin on your server and then enable it within Postal. +By default, Postal will talk to SpamAssassin's `spamd` using a TCP socket connection (port 783). You'll need to install SpamAssassin on your server and then enable it within Postal. ### Installing SpamAssassin @@ -20,17 +52,17 @@ sudo apt install spamassassin ``` #### Systemd systems -On systems that use systemd (e.g. Debian Bookworm), you will need to enable the SpamAssassin timer. It is used to udpate the spam rules (which can be done manually using `sa-update`). +On systems that use systemd (e.g. Debian Bookworm), you will need to enable the SpamAssassin timer. It is used to update the spam rules (which can be done manually using `sa-update`). ```shell systemctl enable --now spamassassin-maintenance.timer ``` #### Other systems -On other systems, you will need to open up `/etc/default/spamassassin` and change `ENABLED` to `1` and `CRON` to `1`. On some systems (such as Ubuntu 20.04 or newer), you might need to enable the SpamAssassin daemon with the following command. +On other systems, you will need to open up `/etc/default/spamassassin` and change `ENABLED` to `1` and `CRON` to `1`. On some systems (such as Ubuntu 20.04 or newer), you might need to enable the SpamAssassin daemon with the following command. ```bash -update-rc.d spamassassin enable +update-rc.d spamassassin enable ``` Then you should restart SpamAssassin. @@ -41,7 +73,7 @@ sudo systemctl restart spamassassin ### Enabling in Postal -To enable spam checking, you'll need to add the following to your `postal.yml` configuration file and restart. If you have installed SpamAssassin on a different host to your Postal installation you can change the host here but be sure to ensure that the `spamd` is listening on your external interfaces. +To enable spam checking, you'll need to add the following to your `postal.yml` configuration file and restart. If you have installed SpamAssassin on a different host to your Postal installation you can change the host here but be sure to ensure that `spamd` is listening on your external interfaces. ```yaml spamd: @@ -55,12 +87,37 @@ postal stop postal start ``` -### Classifying Spam +When scanning outgoing messages, the `NO_RECEIVED`, `NO_RELAYS`, `ALL_TRUSTED`, `FREEMAIL_FORGED_REPLYTO`, `RDNS_DYNAMIC` and `CK_HELO_GENERIC` rules are ignored as they are not meaningful for locally submitted mail. + +## Setting up ClamAV -The spam system is based on a numeric scoring system and different scores are assigned to different issues which may appear in a message. You can configure different thresholds which define when a message is treated as spam. We recommend starting at 5 and updating this once you've seen how your incoming messages are classified. +Postal connects to the `clamd` daemon over TCP (using the `INSTREAM` command). Install ClamAV and ensure `clamd` is configured with a `TCPSocket` (the port is `3310` in most distributions' default configuration - note that Postal's default is `2000`, so set the port to match your `clamd.conf`). + +```yaml +clamav: + enabled: true + host: 127.0.0.1 + port: 3310 +``` + +## Classifying spam + +The spam system is based on a numeric scoring system and different scores are assigned to different issues which may appear in a message. Each mail server has two thresholds which can be changed from the **Settings → Spam** page for the server. The defaults for new servers are set by the `postal.default_spam_threshold` (default `5`) and `postal.default_spam_failure_threshold` (default `20`) configuration options. + +* **Spam threshold** - a message with a score **greater than** this is treated as spam. We recommend starting at 5 and updating this once you've seen how your incoming messages are classified. +* **Spam failure threshold** - a message with a score **greater than or equal to** this is failed immediately and will not be delivered to any route or endpoint. This happens before the route is considered. + +The following headers are appended to every inspected incoming message so that your application can make its own decisions: + +```text +X-Postal-Spam: yes +X-Postal-Spam-Threshold: 5.0 +X-Postal-Spam-Score: 7.3 +X-Postal-Threat: no +``` -You have three options which can be configured on a per route basis which defines how spam messages are treated: +You then have three options which can be configured on a per-route basis which define how messages identified as spam (over the spam threshold but under the failure threshold) are treated: -* **Mark** - messages will be sent through to your endpoint but the spam information will be made available to you. -* **Quarantine** - the message will placed into your hold queue and you'll need to release them if you wish them to be passed to your application. They will only remain here for 7 days, +* **Mark** - messages will be sent through to your endpoint but the spam information will be made available to you in the headers above and the `spam_status` field of HTTP payloads and webhooks. +* **Quarantine** - the message will be placed into your held queue and you'll need to release it if you wish it to be passed to your endpoint. Held messages expire after the number of days set by `postal.default_maximum_hold_expiry_days` (default 7). * **Fail** - the message will be marked as failed and will only be recorded in your message history without being sent. diff --git a/content/3.features/users-and-permissions.md b/content/3.features/users-and-permissions.md new file mode 100644 index 0000000..aac4455 --- /dev/null +++ b/content/3.features/users-and-permissions.md @@ -0,0 +1,43 @@ +--- +title: Users & Permissions +description: 'Global administrators, organization members and what each can do.' +category: Features +--- + +Postal has a simple permission model with two kinds of user. + +## Global administrators + +Administrators have full access to every organization, server and setting in the installation. They are the only users who can: + +* See and manage **all** organizations (non-admins only see organizations they have been added to). +* Create and delete organizations. +* Manage users (**Users** in the top navigation): create users, edit their details, grant or revoke admin status and choose which organizations a non-admin user belongs to. An administrator cannot remove their own admin status or delete their own user. +* Manage [IP pools](/features/ip-pools), IP addresses and organization pool assignments. +* Change a server's **Advanced Settings** (send limit, allow sender header, privacy mode, SMTP data logging, outbound spam threshold and retention) and suspend or unsuspend servers. See [Mail server settings](/features/mail-server-settings#advanced-settings-administrators-only). +* Add domains without verifying them - domains added by an administrator are marked as verified immediately. + +The first administrator is created from the command line during installation with: + +```bash +postal make-user +``` + +This prompts for an e-mail address, first name, last name and password, and always creates an administrator. Run it again at any time to create additional administrators, for example if you have locked yourself out. + +## Organization users + +Everyone else is an ordinary user who belongs to one or more organizations. Within an organization they have full access to *all* of its mail servers and domains - there is no per-server or read-only access. This includes creating and deleting servers, managing domains, routes, endpoints, credentials, webhooks and the non-admin server settings, and viewing every message. + +Organizations record which user created them as the **owner**, but this does not currently grant any additional permissions. + +Users are added to organizations by an administrator from the **Users** page: edit the user and tick the organizations they should be able to access. Removing the last organization from a non-admin user leaves them able to log in but with nothing to see. + +## Accounts and passwords + +* Users log in with their e-mail address and password. Passwords must be at least 8 characters long. +* Password resets are available from the login page and are sent using the `smtp` section of the Postal configuration, so make sure that is set up (you can test it with `postal test-app-smtp`). +* From **My Settings** a user can change their name, e-mail address, time zone and (after confirming their current password) their password. Times throughout the interface are shown in the user's time zone, which defaults to UTC. +* After logging in you are asked whether you would like to stay logged in. Choosing **Remember me** keeps the session alive across browser restarts; otherwise it ends when the browser is closed. + +If [OpenID Connect](/features/oidc) is enabled, users can be created without a password and are linked to their identity provider account on first login. Local logins can be disabled entirely with `oidc.local_authentication_enabled: false`. diff --git a/content/4.developer/1.api.md b/content/4.developer/1.api.md index 80f68da..9633ce0 100644 --- a/content/4.developer/1.api.md +++ b/content/4.developer/1.api.md @@ -1,79 +1,273 @@ --- title: Using the API -description: '' +description: 'Send messages and retrieve message details using the Postal HTTP API.' --- -The HTTP API allows you to send messages to us using JSON over HTTP. You can either talk to the API using your current HTTP library or you can use one of the pre-built libraries. -[Full API documentation](https://apiv1.postalserver.io) +The HTTP API allows you to send messages to Postal using JSON over HTTP and to retrieve information about messages that have been sent. You can either talk to the API using your current HTTP library or you can use one of the pre-built [client libraries](/developer/client-libraries). ::callout{icon="i-heroicons-information-circle"} -This API does not support managing all the functions of Postal. There are plans to introduce a new v2 API which will have more functionality and significantly better documentation. We do not have an ETA for this. Additionally, we will not be accepting any pull requests to extend the current API to have any further functionality than it currently does. +This API does not support managing all the functions of Postal. There are plans to introduce a new v2 API which will have more functionality. We do not have an ETA for this. Additionally, we will not be accepting any pull requests to extend the current API to have any further functionality than it currently does. :: -## General API Instructions +## General API instructions -* You should send POST requests to the URLs shown below. -* Parameters should be encoded in the body of the request and `application/json` should be set as the `Content-Type` header. -* The response will always be provided as JSON. The status of a request can be determined from the `status` attribute in the payload you receive. It will be `success` or `error`. Further details can be found in the `data` attribute. +* The API is available on your Postal web hostname under `/api/v1/`, for example `https://postal.yourdomain.com/api/v1/send/message`. +* Requests can be sent using `POST`, `PUT`, `PATCH` or `GET`. We recommend using `POST`. +* Parameters should be encoded as JSON in the body of the request with `application/json` set as the `Content-Type` header. Alternatively, you may send a form-encoded request with a single field named `params` containing the JSON document. +* The HTTP status of the response is **always `200`**, even for errors. You must inspect the `status` attribute in the response body to determine the outcome. -An example response body is shown below: +### Authentication -```javascript +To authenticate to the API you'll need to create a credential with the type **API** for your mail server through the web interface (**Credentials** in the server menu). This is a random string which is unique to your server. + +Pass this key in the `X-Server-API-Key` HTTP header on every request. Credentials of type **SMTP** cannot be used with the API. + +### Response format + +The response will always be provided as JSON with the following attributes. + +```json { - "status":"success", - "time":0.02, - "flags":{}, - "data":{"message_id":"xxxx"} + "status": "success", + "time": 0.02, + "flags": {}, + "data": { "message_id": "xxxx" } } ``` -To authenticate to the API you'll need to create an API credential for your mail server through the web interface. This is a random string which is unique to your server. +* `status` is one of: + * `success` - the request was processed and `data` contains the result. + * `error` - the request failed. `data.code` contains an error code (listed for each endpoint below) and `data` may contain further details such as a `message`. + * `parameter-error` - a parameter was missing or invalid. `data.message` contains a description. There is no `code` for this status. +* `time` is the time taken to process the request in seconds. +* `flags` is always an empty object. -To authenticate a request to the API, you need to pass this key in the `X-Server-API-Key` HTTP header. +The following errors may be returned by any endpoint: -## Sending a message +| Code | Meaning | +|---|---| +| `AccessDenied` | No `X-Server-API-Key` header was provided. | +| `InvalidServerAPIKey` | The key provided did not match an API credential. `data.token` contains the key you sent. | +| `ServerSuspended` | The mail server that owns the credential has been suspended. | -There are two ways to send a message - you can either provide each attribute needed for the e-mail individually or you can craft your own RFC 2822 message and send this instead. - -Full details about these two methods can be found in our API documentation: +## Sending a message -* [Sending a message](https://postalserver.github.io/postal-api/controllers/send/message) -* [Sending an RFC 2822 message](https://postalserver.github.io/postal-api/controllers/send/raw) +There are two ways to send a message - you can either provide each attribute needed for the e-mail individually (`send/message`) or you can craft your own RFC 2822 message and send this instead (`send/raw`). -For both these methods, the API will return the same information as the result. It will contain the `message_id` of the message that was sent plus a `messages` hash with the IDs of the messages that were sent by the server to each recipient. +For both methods, the API will return the same information as the result. It contains the `message_id` of the message that was sent plus a `messages` object with the ID and token of the message created for each recipient. -```javascript +```json { - "message_id":"message-id-in-uuid-format@rp.postal.yourdomain.com", - "messages":{ - "john@example.com":{"id":37171, "token":"a4udnay1"}, - "mary@example.com":{"id":37172, "token":"bsfhjsdfs"} + "message_id": "message-id-in-uuid-format@rp.postal.yourdomain.com", + "messages": { + "john@example.com": { "id": 37171, "token": "a4udnay1kwsd8b3c" }, + "mary@example.com": { "id": 37172, "token": "bsfhjsdfsrt3qwe1" } } } ``` -## GET Message +Messages are queued and delivered asynchronously by the Postal workers. Use the `messages/message` and `messages/deliveries` endpoints, or [webhooks](/developer/webhooks), to track their progress. + +### `POST /api/v1/send/message` + +Sends a message by providing its attributes individually. Postal will construct the message for you. + +| Parameter | Type | Description | +|---|---|---| +| `to` | array of strings | Recipient addresses for the `To` header. A comma-separated string is also accepted. Maximum 50. | +| `cc` | array of strings | Recipient addresses for the `Cc` header. Maximum 50. | +| `bcc` | array of strings | Recipient addresses that will receive the message but not appear in headers. Maximum 50. | +| `from` | string | **Required.** The address for the `From` header, e.g. `Sales `. The domain must be a verified domain on the server or its organization. | +| `sender` | string | The address for the `Sender` header. If the server has **Allow sender header** enabled, this may be used to authenticate the message instead of `from`. | +| `subject` | string | The subject of the message. | +| `tag` | string | A tag for the message. This is stored with the message and returned in webhooks. | +| `reply_to` | string | The address for the `Reply-To` header. | +| `plain_body` | string | The plain text body. At least one of `plain_body` or `html_body` is required. | +| `html_body` | string | The HTML body. | +| `headers` | object | A hash of additional headers to add to the message, e.g. `{"X-Custom": "value"}`. | +| `attachments` | array of objects | Each attachment must have a `name` and `data` (Base64-encoded content). `content_type` is optional and defaults to `application/octet-stream`. | +| `bounce` | boolean | Set to `true` if this message is a bounce. Bounce messages are sent with an empty `MAIL FROM`. | + +At least one recipient in `to`, `cc` or `bcc` is required. A separate message (with its own ID and token) is created for each unique recipient across all three lists. + +Errors returned by this endpoint (only the first error encountered is returned): + +| Code | Meaning | +|---|---| +| `NoRecipients` | No `to`, `cc` or `bcc` addresses were provided. | +| `TooManyToAddresses` / `TooManyCCAddresses` / `TooManyBCCAddresses` | More than 50 addresses in the given list. | +| `NoContent` | Neither `plain_body` nor `html_body` was provided. | +| `FromAddressMissing` | No `from` address was provided. | +| `UnauthenticatedFromAddress` | The domain of the `from` address (or `sender` address, if permitted) is not a verified domain on this server or its organization. | +| `AttachmentMissingName` | An attachment does not have a `name`. | +| `AttachmentMissingData` | An attachment does not have any `data`. | + +Example: + +```bash +curl -X POST https://postal.yourdomain.com/api/v1/send/message \ + -H 'X-Server-API-Key: YOUR_API_KEY' \ + -H 'Content-Type: application/json' \ + -d '{ + "to": ["john@example.com"], + "from": "Sales ", + "subject": "Welcome", + "plain_body": "Hello John!", + "html_body": "

Hello John!

", + "tag": "welcome", + "attachments": [ + {"name": "terms.txt", "content_type": "text/plain", "data": "SGVsbG8gd29ybGQh"} + ] + }' +``` + +### `POST /api/v1/send/raw` + +Sends a message which you have constructed yourself as a full RFC 2822 message. + +| Parameter | Type | Description | +|---|---|---| +| `mail_from` | string | **Required.** The address to use as the SMTP envelope sender. | +| `rcpt_to` | array of strings | **Required.** The recipient addresses. This must be a JSON array, even for a single recipient. There is no limit on the number of recipients. | +| `data` | string | **Required.** The full message (headers and body) encoded with Base64. | +| `bounce` | boolean | Set to `true` if this message is a bounce. | -To retrieve a message and its contents, use the `GET` method with the `id` (received when sending the message) and `_expansions` parameters (if you need more information than the basics) for the message from Postal. For more details, refer to the [Postal API documentation](https://postalserver.github.io/postal-api/controllers/messages/message.html). +The `From` header (and the `Sender` header if the server has **Allow sender header** enabled) is parsed from the message and must contain a verified domain for the server or its organization. The `mail_from` value itself is not checked against your domains. + +Errors returned by this endpoint: + +| Status | Code / message | +|---|---| +| `parameter-error` | `` `rcpt_to` parameter is required but is missing `` | +| `parameter-error` | `` `mail_from` parameter is required but is missing `` | +| `parameter-error` | `` `data` parameter is required but is missing `` | +| `error` | `UnauthenticatedFromAddress` | + +## Retrieving a message + +### `POST /api/v1/messages/message` + +Returns details about a message using the `id` that was returned when the message was sent. + +| Parameter | Type | Description | +|---|---|---| +| `id` | integer or string | **Required.** The ID of the message. | +| `_expansions` | `true` or array of strings | Which additional sections to include in the response. Pass `true` to include everything, or an array containing any of `status`, `details`, `inspection`, `plain_body`, `html_body`, `attachments`, `headers`, `raw_message` and `activity_entries`. | + +Without any expansions, the response only contains `id` and `token`. + +```bash +curl -X POST https://postal.yourdomain.com/api/v1/messages/message \ + -H 'X-Server-API-Key: YOUR_API_KEY' \ + -H 'Content-Type: application/json' \ + -d '{"id": 14, "_expansions": ["status", "details"]}' +``` + +The full response with all expansions is shown below. ```json { "id": 14, - "_expansions": true + "token": "a4udnay1kwsd8b3c", + "status": { + "status": "Sent", + "last_delivery_attempt": 1477945177.12, + "held": false, + "hold_expiry": null + }, + "details": { + "rcpt_to": "john@example.com", + "mail_from": "sales@yourdomain.com", + "subject": "Welcome", + "message_id": "81026759-68fb-4872-8c97-6dd2901cb33a@rp.postal.yourdomain.com", + "timestamp": 1477945170.52, + "direction": "outgoing", + "size": "1522", + "bounce": false, + "bounce_for_id": 0, + "tag": "welcome", + "received_with_ssl": true + }, + "inspection": { + "inspected": true, + "spam": false, + "spam_score": 0.1, + "threat": false, + "threat_details": "No threats found" + }, + "plain_body": "Hello John!", + "html_body": "

Hello John!

", + "attachments": [ + { + "filename": "terms.txt", + "content_type": "text/plain", + "data": "SGVsbG8gd29ybGQh\n", + "size": 12, + "hash": "d3486ae9136e7856bc42212385ea797094475802" + } + ], + "headers": { + "from": ["Sales "], + "to": ["john@example.com"], + "subject": ["Welcome"] + }, + "raw_message": "RnJvbTogU2FsZXMgPHNhbGVzQHlvdXJkb21haW4uY29tPg0K...", + "activity_entries": { + "loads": [ + {"ip_address": "185.22.208.2", "user_agent": "Mozilla/5.0 ...", "timestamp": "2016-11-01T19:39:37.000Z"} + ], + "clicks": [ + {"url": "https://yourdomain.com", "ip_address": "185.22.208.2", "user_agent": "Mozilla/5.0 ...", "timestamp": "2016-11-01T19:40:02.000Z"} + ] + } } ``` -### Example cURL Request +* `status.status` is `Pending` for a message that has not yet been processed, otherwise it reflects the status of the most recent delivery: `Sent`, `SoftFail`, `HardFail`, `Held`, `HoldCancelled`, `Bounced`, `Processed` or `Error`. +* `details.direction` is `outgoing` or `incoming`. `details.size` is a string. +* `headers` is an object of lower-cased header names to arrays of values. +* `attachments[].hash` is the SHA1 hex digest of the attachment content. -You can use the following cURL command to make the request: +Errors returned by this endpoint: -```bash -curl --location 'https:///api/v1/messages/message' \ ---header 'X-Server-API-Key: $yourAPIKeyFromTheCredentialsPage' \ ---header 'Content-Type: application/json' \ ---data '{ - "id": 14, - "_expansions": true -}' -``` \ No newline at end of file +| Status | Code / message | +|---|---| +| `parameter-error` | `` `id` parameter is required but is missing `` | +| `parameter-error` | `` `id` parameter must be a string or integer `` | +| `error` | `MessageNotFound` - no message with this ID exists on the server. `data.id` contains the ID you sent. | + +### `POST /api/v1/messages/deliveries` + +Returns every delivery attempt for a message, oldest first. + +| Parameter | Type | Description | +|---|---|---| +| `id` | integer or string | **Required.** The ID of the message. | + +```json +[ + { + "id": 1, + "status": "SoftFail", + "details": "No SMTP servers were available for example.com. Tried mx1.example.com.", + "output": "", + "sent_with_ssl": false, + "log_id": "abc123", + "time": 2.13, + "timestamp": 1477945100.12 + }, + { + "id": 2, + "status": "Sent", + "details": "Message sent by SMTP to aspmx.l.google.com (2a00:1450:400c:c0b::1b)", + "output": "250 2.0.0 OK 1477944899 ly2si31746747wjb.95 - gsmtp", + "sent_with_ssl": true, + "log_id": "abc124", + "time": 0.22, + "timestamp": 1477945177.12 + } +] +``` + +`status` is one of `Sent`, `SoftFail`, `HardFail`, `Held`, `Bounced`, `Processed`, `HoldCancelled` or `Error`. The same errors as `messages/message` apply. diff --git a/content/4.developer/3.http-payloads.md b/content/4.developer/3.http-payloads.md index 2193de5..41dfb19 100644 --- a/content/4.developer/3.http-payloads.md +++ b/content/4.developer/3.http-payloads.md @@ -1,77 +1,144 @@ --- title: Receiving e-mail by HTTP -description: '' +description: 'Deliver incoming messages to your own application over HTTP.' --- One of the most useful features in Postal is the ability to have incoming messages delivered to your own application as soon as they arrive. To receive incoming messages from Postal you can set it up to pass them to an HTTP URL of your choosing. -Each endpoint has an HTTP URL (we highly recommend making use of HTTPS where possible) as well as a set of rules which defines how data is sent to you. +To do this you create an **HTTP endpoint** on your mail server and then create a **route** which delivers mail for a given address (or a whole domain) to that endpoint. See [Routing incoming e-mail](/features/routing-incoming-email) for details of routes. -* You can choose whether data is encoded as normal form data or whether Postal sends JSON as the body of the request. -* You can choose whether to receive the raw message (raw) or have it as a JSON dictionary (processed). -* You can choose whether you'd like replies and signatures to be separated from the plain body of the message. +## HTTP endpoint options -Your server should accept Postals incoming request and reply within 5 seconds. If it takes longer than this, Postal will assume it has failed and the request will be retried. Your server should send a `200 OK` status to signal to Postal that you've received the request. +Each endpoint has an HTTP URL (we highly recommend making use of HTTPS where possible) as well as a set of options which define how data is sent to you. -Messages will be tried up to 18 times with an exponential back-off until a successful response is seen except in the case of `5xx` statuses which will fail immediately. +| Option | Values | Description | +|---|---|---| +| **Encoding** | `Sent in the body as JSON` (default) or `Sent as form data` | Whether the payload is sent as a JSON document with `Content-Type: application/json`, or as `application/x-www-form-urlencoded` form fields. | +| **Format** | `Delivered as a hash` (default) or `Delivered as the raw message` | Whether you receive the message parsed into its component parts (the *processed* payload) or the full RFC 2822 message (the *raw* payload). | +| **Strip replies** | on/off (default off) | When enabled, Postal will attempt to separate quoted replies and signatures from the plain body. Only applies to the processed payload. | +| **Include attachments** | on/off (default on) | Whether attachment data is included in the processed payload. The raw payload always contains attachments as part of the message. | +| **Timeout** | 5 to 60 seconds (default 5) | How long Postal will wait for your server to respond before treating the attempt as failed. | -When a message permanently fails to be delivered to your endpoint (i.e. the server returned a 5xx status code or it wasn't accepted after 18 attempts), the recipient will be sent a bounce message. +## Delivery behaviour -You can view the attempts (along with debugging information) on the message page in the web interface. +Your server should accept Postal's request and reply within the configured timeout. Your server should send a `2xx` status to signal to Postal that you've received the request. Redirects are not followed. + +What happens next depends on the response: + +| Response | Result | +|---|---| +| `2xx` | The message is marked as **Sent**. | +| `5xx`, a timeout, a connection error or an SSL certificate error | The delivery is a **soft fail** and will be retried later. | +| `429 Too Many Requests` | The delivery is a **hard fail**. The message will not be retried and **no bounce** is sent to the sender. | +| Any other status (including `3xx` and other `4xx`) | The delivery is a **hard fail**. The message will not be retried and a bounce message is sent to the original sender. | + +Soft failures are retried with an exponential back-off. The delay before attempt *n* is approximately `5 minutes × 1.3ⁿ` (5 min, 6.5 min, 8.5 min, 11 min...). The maximum number of attempts is set by the `postal.default_maximum_delivery_attempts` configuration option (default `18`), which works out to roughly 31 hours of retries. When the maximum is reached, the message is marked as a hard fail and a bounce message is sent to the sender. + +Bounces are only sent when the incoming message itself is not a bounce and has a non-empty `MAIL FROM`. + +You can view the attempts (along with the response received from your server) on the message page in the web interface. Each delivery attempt also triggers the [`MessageSent`, `MessageDelayed` or `MessageDeliveryFailed` webhook](/developer/webhooks#message-status-events) if you have webhooks configured. + +## Request headers + +Every request includes the following headers. + +* `User-Agent: Postal/{version}` +* `Content-Type` - `application/json` or `application/x-www-form-urlencoded` depending on the encoding option. +* `X-Postal-Signature-256`, `X-Postal-Signature` and `X-Postal-Signature-KID` - signatures of the request body which you can use to verify the request genuinely came from your Postal installation. These are identical to the headers used for webhooks; see [Verifying signatures](/developer/webhooks#verifying-signatures). ## The processed payload -When you chose to receive the message as JSON (processed), you'll receive a payload with the following attributes. +When you choose to receive the message as a hash (processed), you'll receive a payload with the following attributes. -```javascript +```json { - "id":12345, - "rcpt_to":"sales@awesomeapp.com", - "mail_from":"test@example.com", - "token":"rtmuzogUauKN", - "subject":"Re: Welcome to AwesomeApp", - "message_id":"81026759-68fb-4872-8c97-6dd2901cb33a@rp.postal.yourdomain.com", - "timestamp":1478169798.924355, - "size":"822", - "spam_status":"NotSpam", - "bounce":false, - "received_with_ssl":false, - "to":"sales@awesomeapp.com", - "cc":null, - "from":"John Doe ", - "date":"Thu, 03 Nov 2016 10:43:18 +0000", - "in_reply_to":null, - "references":null, - "plain_body":"Hello there!", - "html_body":"

Hello there!

", - "auto_submitted":"auto-reply", - "attachment_quantity":1, - "attachments":[ + "id": 12345, + "rcpt_to": "sales@awesomeapp.com", + "mail_from": "test@example.com", + "token": "rtmuzogUauKN", + "subject": "Re: Welcome to AwesomeApp", + "message_id": "81026759-68fb-4872-8c97-6dd2901cb33a@rp.postal.yourdomain.com", + "timestamp": 1478169798.924355, + "size": "822", + "spam_status": "NotSpam", + "bounce": false, + "received_with_ssl": false, + "to": "sales@awesomeapp.com", + "cc": null, + "from": "John Doe ", + "date": "Thu, 03 Nov 2016 10:43:18 +0000", + "in_reply_to": null, + "references": null, + "reply_to": ["John Doe "], + "auto_submitted": "auto-reply", + "plain_body": "Hello there!", + "replies_from_plain_body": null, + "html_body": "

Hello there!

", + "attachment_quantity": 1, + "attachments": [ { - "filename":"test.txt", - "content_type":"text/plain", - "size":12, - "data":"SGVsbG8gd29ybGQh" + "filename": "test.txt", + "content_type": "text/plain", + "size": 12, + "data": "SGVsbG8gd29ybGQh\n" } ] } ``` -* You will only have the `attachments` attribute if you have enabled it. -* The `data` attribute for each attachment is Base64 encoded. +* `rcpt_to` and `mail_from` are the SMTP envelope addresses. `to`, `cc`, `from`, `date`, `in_reply_to`, `references` and `auto_submitted` are the values of the corresponding message headers (the last value if the header appears more than once). `reply_to` is an array of all `Reply-To` header values. +* `size` is a string containing the size of the message in bytes. +* `spam_status` is one of `NotChecked`, `Spam` or `NotSpam`. +* `replies_from_plain_body` is only present when **Strip replies** is enabled. It contains the text that was removed from `plain_body`, or `null` if nothing was removed. +* You will only have the `attachments` attribute if **Include attachments** is enabled. +* The `data` attribute for each attachment is Base64 encoded (with line breaks every 60 characters). + +When using the **form data** encoding, the same keys are sent as form fields. Attachments are flattened into `attachments[0][filename]`, `attachments[0][content_type]`, `attachments[0][size]` and `attachments[0][data]` (and so on for each attachment). ## The raw message payload When you choose to receive the full message, you will receive the following attributes. -```javascript +```json { - "id":12345, - "message":"REtJTS1TaWduYXR1cmU6IHY9MTsgYT1yc2Etc2hhMjU2Oy...", - "base64":true, - "size":859 + "id": 12345, + "rcpt_to": "sales@awesomeapp.com", + "mail_from": "test@example.com", + "message": "REtJTS1TaWduYXR1cmU6IHY9MTsgYT1yc2Etc2hhMjU2Oy...", + "base64": true, + "size": 859 } ``` -* The `base64` attribute specifies whether or not the `message` attribute is encoded with Base64. This is likely to be true all the time. +* The `base64` attribute specifies whether or not the `message` attribute is encoded with Base64. This is always `true`. +* The **Strip replies** and **Include attachments** options have no effect on the raw payload. + +## Blocked destinations + +To prevent server-side request forgery, Postal refuses to send HTTP endpoint and webhook requests to URLs which resolve to any of the following address ranges. If a hostname resolves to multiple addresses and *any* of them is blocked, the request is refused. + +| IPv4 | IPv6 | +|---|---| +| `0.0.0.0/8` | `::/128` | +| `10.0.0.0/8` | `::1/128` | +| `100.64.0.0/10` | `::ffff:0:0/96` (IPv4-mapped, checked against the IPv4 list) | +| `127.0.0.0/8` | `fc00::/7` | +| `169.254.0.0/16` | `fe80::/10` | +| `172.16.0.0/12` | `ff00::/8` | +| `192.0.0.0/24` | | +| `192.168.0.0/16` | | +| `198.18.0.0/15` | | +| `224.0.0.0/4` | | +| `240.0.0.0/4` | | + +If you need to deliver to an application on a private network, add the hostname, IP address or CIDR range to the `postal.allowed_request_destinations` configuration option (environment variable `POSTAL_ALLOWED_REQUEST_DESTINATIONS`, comma-separated). Hostname entries are matched case-insensitively against the hostname in the URL; IP and CIDR entries are matched against the resolved address. + +```yaml +postal: + allowed_request_destinations: + - app.internal + - 10.0.5.0/24 +``` + +A request to a blocked destination is recorded as a soft failure and will be retried, so if you see repeated `SoftFail` deliveries for an internal endpoint, check this setting. diff --git a/content/4.developer/4.webhooks.md b/content/4.developer/4.webhooks.md index 921abb8..2ed1c79 100644 --- a/content/4.developer/4.webhooks.md +++ b/content/4.developer/4.webhooks.md @@ -1,74 +1,136 @@ --- title: Webhooks -description: '' +description: 'Receive HTTP notifications when events occur during the lifecycle of a message.' --- Postal supports sending webhooks over HTTP when various events occur during the lifecycle of a message. -This page lists all the different types of event along with an example JSON payload that you'll receive. In many cases, only a small amount of information will be sent, if you require more information you should use the API to obtain it. +Webhooks are configured per mail server from the **Webhooks** section in the web interface. Each webhook has a URL (`http://` or `https://`) and either receives **all events** or a chosen subset of events. + +This page lists all the different types of event along with an example JSON payload that you'll receive. In many cases, only a small amount of information will be sent; if you require more information you should use the [API](/developer/api) to obtain it. + +## How webhooks are delivered + +* Requests are sent as `POST` with a `Content-Type: application/json` body. +* The `User-Agent` header is `Postal/{version}` (for example `Postal/3.3.7`). +* Your endpoint must respond within **5 seconds**. Any `2xx` response is treated as success. Redirects are **not** followed, so a `3xx` response counts as a failure. +* Every request is signed (see [Verifying signatures](#verifying-signatures)). +* If a request fails, it will be retried up to a maximum of **6 attempts** in total. The delay before each retry is 2, 3, 6, 10 and 15 minutes respectively. After the final failed attempt the request is discarded. A webhook is never automatically disabled because of failures. +* Disabling a webhook stops new requests from being created for it. Any requests already queued will still be delivered. +* A history of every attempt (including the request payload and the response received) is stored for **10 days** and can be viewed from the **Webhooks → History** page for the server. +* Webhook URLs that resolve to private, loopback, link-local or otherwise reserved IP addresses are blocked unless they are listed in the `postal.allowed_request_destinations` configuration option. See [Receiving e-mail by HTTP](/developer/http-payloads#blocked-destinations) for the list of blocked ranges. + +## Payload envelope + +Every webhook request has the same outer structure. The event-specific data described in the rest of this page is found in the `payload` key. + +```json +{ + "event": "MessageSent", + "timestamp": 1477945177.12994, + "uuid": "820b47a4-4dfd-42e4-ae6a-1e5bed5a33fd", + "payload": { + "...": "event specific data" + } +} +``` + +* `event` - the name of the event (see below). +* `timestamp` - the UNIX time (with fractional seconds) when the event was queued for delivery. +* `uuid` - a unique identifier for this webhook request. This is the same across all delivery attempts for the request and can be used to de-duplicate. +* `payload` - the event-specific data. + +## The `message` object + +Most events include a `message` object describing the message that the event relates to. It always has the following attributes. + +```json +{ + "id": 12345, + "token": "abcdef123", + "direction": "outgoing", + "message_id": "5817a64332f44_4ec93ff59e79d154565eb@app34.mail", + "to": "test@example.com", + "from": "sales@awesomeapp.com", + "subject": "Welcome to AwesomeApp", + "timestamp": 1477945177.12994, + "spam_status": "NotSpam", + "tag": "welcome" +} +``` + +* `direction` is either `outgoing` or `incoming`. +* `to` and `from` are the SMTP envelope addresses (`RCPT TO` and `MAIL FROM`), not the values of the `To` and `From` headers. +* `spam_status` is one of `NotChecked`, `Spam` or `NotSpam`. +* `tag` may be `null`. ## Message Status Events -These events are triggered on various events in an e-mail's lifecycle. The payload format is identical for all messages however the `status` attribute may change. The following statuses may be delivered. +These events are triggered when a delivery attempt is made for a message. The payload format is identical for all of them. The `status` attribute reflects the delivery status. + +* `MessageSent` - when a message is successfully delivered to a recipient/endpoint. `status` will be `Sent`. +* `MessageDelayed` - when a message's delivery has been delayed. This will be sent each time Postal attempts a delivery and the message is delayed further. `status` will be `SoftFail`. +* `MessageDeliveryFailed` - when a message cannot be delivered and will not be retried. `status` will be `HardFail`. +* `MessageHeld` - when a message is held. `status` will be `Held`. See [Mail server settings](/features/mail-server-settings#held-messages) for the reasons a message may be held. -* `MessageSent` - when a message is successfully delivered to a recipient/endpoint. -* `MessageDelayed` - when a message's delivery has been delayed. This will be sent each time Postal attempts a delivery and a message is delayed further. -* `MessageDeliveryFailed` - when a message cannot be delivered. -* `MessageHeld` - when a message is held. +These events are sent for both outgoing messages (delivered by SMTP) and incoming messages (delivered to HTTP, SMTP or address endpoints). -```javascript +```json { - "status":"Sent", - "details":"Message sent by SMTP to aspmx.l.google.com (2a00:1450:400c:c0b::1b) (from 2a00:67a0:a:15::2)", - "output":"250 2.0.0 OK 1477944899 ly2si31746747wjb.95 - gsmtp", - "time":0.22, - "sent_with_ssl":true, - "timestamp":1477945177.12994, - "message":{ - "id":12345, - "token":"abcdef123", - "direction":"outgoing", - "message_id":"5817a64332f44_4ec93ff59e79d154565eb@app34.mail", - "to":"test@example.com", - "from":"sales@awesomeapp.com", - "subject":"Welcome to AwesomeApp", - "timestamp":1477945177.12994, - "spam_status":"NotSpam", - "tag":"welcome" + "status": "Sent", + "details": "Message sent by SMTP to aspmx.l.google.com (2a00:1450:400c:c0b::1b) (from 2a00:67a0:a:15::2)", + "output": "250 2.0.0 OK 1477944899 ly2si31746747wjb.95 - gsmtp", + "time": 0.22, + "sent_with_ssl": true, + "timestamp": 1477945177.12994, + "message": { + "id": 12345, + "token": "abcdef123", + "direction": "outgoing", + "message_id": "5817a64332f44_4ec93ff59e79d154565eb@app34.mail", + "to": "test@example.com", + "from": "sales@awesomeapp.com", + "subject": "Welcome to AwesomeApp", + "timestamp": 1477945177.12994, + "spam_status": "NotSpam", + "tag": "welcome" } } ``` +* `details` and `output` are truncated (to 250 and 512 characters respectively). +* `time` is the time taken for the delivery attempt in seconds. + ## Message Bounces -If Postal receives a bounce message for a message that was previously accepted, you'll receive the `MessageBounced` event. +If Postal receives a bounce message for a message that was previously accepted, you'll receive the `MessageBounced` event. The `bounce` object describes the incoming bounce message that was received. -```javascript +```json { - "original_message":{ - "id":12345, - "token":"abcdef123", - "direction":"outgoing", - "message_id":"5817a64332f44_4ec93ff59e79d154565eb@app34.mail", - "to":"test@example.com", - "from":"sales@awesomeapp.com", - "subject":"Welcome to AwesomeApp", - "timestamp":1477945177.12994, - "spam_status":"NotSpam", - "tag":"welcome" + "original_message": { + "id": 12345, + "token": "abcdef123", + "direction": "outgoing", + "message_id": "5817a64332f44_4ec93ff59e79d154565eb@app34.mail", + "to": "test@example.com", + "from": "sales@awesomeapp.com", + "subject": "Welcome to AwesomeApp", + "timestamp": 1477945177.12994, + "spam_status": "NotSpam", + "tag": "welcome" }, - "bounce":{ - "id":12347, - "token":"abcdef124", - "direction":"incoming", - "message_id":"5817a64332f44_4ec93ff59e79d154565eb@someserver.com", - "to":"abcde@psrp.postal.yourdomain.com", - "from":"postmaster@someserver.com", - "subject":"Delivery Error", - "timestamp":1477945179.12994, - "spam_status":"NotSpam", - "tag":null + "bounce": { + "id": 12347, + "token": "abcdef124", + "direction": "incoming", + "message_id": "5817a64332f44_4ec93ff59e79d154565eb@someserver.com", + "to": "abcde@psrp.postal.yourdomain.com", + "from": "postmaster@someserver.com", + "subject": "Delivery Error", + "timestamp": 1477945179.12994, + "spam_status": "NotSpam", + "tag": null } } ``` @@ -77,46 +139,46 @@ If Postal receives a bounce message for a message that was previously accepted, If you have click tracking enabled, the `MessageLinkClicked` event will tell you that a user has clicked on a link in one of your e-mails. -```javascript +```json { - "url":"https://atech.media", - "token":"VJzsFA0S", - "ip_address":"185.22.208.2", - "user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36", - "message":{ - "id":12345, - "token":"abcdef123", - "direction":"outgoing", - "message_id":"5817a64332f44_4ec93ff59e79d154565eb@app34.mail", - "to":"test@example.com", - "from":"sales@awesomeapp.com", - "subject":"Welcome to AwesomeApp", - "timestamp":1477945177.12994, - "spam_status":"NotSpam", - "tag":"welcome" + "url": "https://atech.media", + "token": "VJzsFA0S", + "ip_address": "185.22.208.2", + "user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36", + "message": { + "id": 12345, + "token": "abcdef123", + "direction": "outgoing", + "message_id": "5817a64332f44_4ec93ff59e79d154565eb@app34.mail", + "to": "test@example.com", + "from": "sales@awesomeapp.com", + "subject": "Welcome to AwesomeApp", + "timestamp": 1477945177.12994, + "spam_status": "NotSpam", + "tag": "welcome" } } ``` ## Message Loaded/Opened Event -If you have open tracking enabled, the `MessageLoaded` event will tell you that a user has opened your e-mail (or, at least, have viewed the tracking pixel embedded within it.) +If you have open tracking enabled, the `MessageLoaded` event will tell you that a user has opened your e-mail (or, at least, has viewed the tracking pixel embedded within it). -```javascript +```json { - "ip_address":"185.22.208.2", - "user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36", - "message":{ - "id":12345, - "token":"abcdef123", - "direction":"outgoing", - "message_id":"5817a64332f44_4ec93ff59e79d154565eb@app34.mail", - "to":"test@example.com", - "from":"sales@awesomeapp.com", - "subject":"Welcome to AwesomeApp", - "timestamp":1477945177.12994, - "spam_status":"NotSpam", - "tag":"welcome" + "ip_address": "185.22.208.2", + "user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36", + "message": { + "id": 12345, + "token": "abcdef123", + "direction": "outgoing", + "message_id": "5817a64332f44_4ec93ff59e79d154565eb@app34.mail", + "to": "test@example.com", + "from": "sales@awesomeapp.com", + "subject": "Welcome to AwesomeApp", + "timestamp": 1477945177.12994, + "spam_status": "NotSpam", + "tag": "welcome" } } ``` @@ -125,24 +187,81 @@ If you have open tracking enabled, the `MessageLoaded` event will tell you that Postal regularly monitors domains it knows about to ensure that your SPF/DKIM/MX records are correct. If you'd like to be notified when the checks fail, you can subscribe to the `DomainDNSError` event. -```javascript +This event is only triggered by the automatic hourly DNS checks (not when you press **Check** in the web interface) and only for domains that belong to the mail server itself. Domains owned by the organization do not trigger this event. + +```json { - "domain":"example.com", - "uuid":"820b47a4-4dfd-42e4-ae6a-1e5bed5a33fd", - "dns_checked_at":1477945711.5502, - "spf_status":"OK", - "spf_error":null, - "dkim_status":"Invalid", - "dkim_error":"The DKIM record at example.com does not match the record we have provided. Please check it has been copied correctly.", - "mx_status":"Missing", - "mx_error":null, - "return_path_status":"OK", - "return_path_error":null, - "server":{ - "uuid":"54529725-8807-4069-ab29-a3746c1bbd98", - "name":"AwesomeApp Mail Server", - "permalink":"awesomeapp", - "organization":"atech" + "domain": "example.com", + "uuid": "820b47a4-4dfd-42e4-ae6a-1e5bed5a33fd", + "dns_checked_at": 1477945711.5502, + "spf_status": "OK", + "spf_error": null, + "dkim_status": "Invalid", + "dkim_error": "The DKIM record at example.com does not match the record we have provided. Please check it has been copied correctly.", + "mx_status": "Missing", + "mx_error": null, + "return_path_status": "OK", + "return_path_error": null, + "server": { + "uuid": "54529725-8807-4069-ab29-a3746c1bbd98", + "name": "AwesomeApp Mail Server", + "permalink": "awesomeapp", + "organization": "atech" + } +} +``` + +The possible values for each `_status` field are documented on the [Sending domains](/features/sending-domains#dns-checks) page. + +## Send Limit Events + +When a mail server has a send limit configured, two further events are triggered. These events cannot be selected individually in the web interface; they are only delivered to webhooks configured to receive **all events**. Each is sent at most once per hour per server. + +* `SendLimitApproaching` - the server has sent 90% or more of its hourly send limit. +* `SendLimitExceeded` - the server has reached its send limit and further outgoing messages are being held. + +```json +{ + "server": { + "uuid": "54529725-8807-4069-ab29-a3746c1bbd98", + "name": "AwesomeApp Mail Server", + "permalink": "awesomeapp", + "organization": "atech" }, + "volume": 950, + "limit": 1000 } ``` + +## Verifying signatures + +Every webhook request is signed using the private key configured in `postal.signing_key_path` (the `signing.key` file created during installation). The following headers are included on every request. + +| Header | Description | +|---|---| +| `X-Postal-Signature-256` | Base64-encoded RSA-SHA256 signature of the raw request body. This is the header you should verify. | +| `X-Postal-Signature` | Base64-encoded RSA-SHA1 signature of the raw request body. Retained for backwards compatibility. | +| `X-Postal-Signature-KID` | The key ID (JWK thumbprint) of the key used to sign the request. | + +The public key can be retrieved from your Postal installation as a JSON Web Key Set at `https://postal.yourdomain.com/.well-known/jwks.json`. Find the key whose `kid` matches the `X-Postal-Signature-KID` header and use it to verify the signature over the exact bytes of the request body. + +An example in Ruby using the `jwt` gem to parse the JWK: + +```ruby +require "openssl" +require "base64" +require "json" +require "net/http" +require "jwt" + +jwks = JSON.parse(Net::HTTP.get(URI("https://postal.yourdomain.com/.well-known/jwks.json"))) +jwk = jwks["keys"].find { |k| k["kid"] == request.headers["X-Postal-Signature-KID"] } +public_key = JWT::JWK.import(jwk).public_key + +signature = Base64.decode64(request.headers["X-Postal-Signature-256"]) +valid = public_key.verify(OpenSSL::Digest.new("SHA256"), signature, request.raw_body) +``` + +The signature is a standard RSA PKCS#1 v1.5 signature so it can be verified with any language's standard cryptography library. + +The same headers are added to requests made to [HTTP endpoints](/developer/http-payloads) for incoming messages. diff --git a/content/5.other/5.workers-and-background-tasks.md b/content/5.other/5.workers-and-background-tasks.md new file mode 100644 index 0000000..6c0b6af --- /dev/null +++ b/content/5.other/5.workers-and-background-tasks.md @@ -0,0 +1,64 @@ +--- +title: Workers & Background Tasks +description: 'How Postal workers process the queue, coordinate with each other and run scheduled maintenance tasks.' +--- + +All mail processing in Postal happens in **worker** processes (`postal worker`). The web and SMTP servers only accept messages and place them on a queue in the main database; workers take them from the queue, deliver them, and run a set of scheduled maintenance tasks. Since Postal v3 there is no message broker - the queue lives in the `queued_messages` table and workers coordinate purely through the database. + +You can run as many worker processes as you need, on as many hosts as you need, provided they all share the same database. + +## Worker threads + +Each worker process starts a number of **work threads** (set by `worker.threads`, default `2`) plus one **tasks thread**. On start up the worker checks that the database connection pool (`main_db.pool_size`) is at least `threads + 3` and increases it automatically if not, logging a warning. + +Each work thread loops over the two jobs below. If a job found work to do, the thread immediately loops again; if neither job found anything, the thread sleeps for 5 seconds before checking again. The worker shuts down cleanly on `SIGINT`/`SIGTERM`, finishing any message it is in the middle of processing first. + +### Processing queued messages + +The `ProcessQueuedMessagesJob` atomically claims **one** message from the queue by writing a lock (`locked_by`, `locked_at`) to a row that is not locked and is ready - that is, it has no `retry_after` or its `retry_after` is at least 30 seconds in the past. Only messages with no allocated IP address, or whose allocated IP address is present on the worker's host, are considered (see [IP pools](/features/ip-pools#how-addresses-are-allocated)). + +The message is then processed: + +* **Outgoing** messages are checked against the server's suspension, send limit, credential hold, suppression list and development mode; optionally inspected for spam; parsed for click/open tracking; DKIM signed; and delivered by SMTP to the recipient's MX servers (or your configured relays). +* **Incoming** messages are matched to their bounce originals, inspected for spam, and delivered according to their [route](/features/routing-incoming-email). + +If delivery fails temporarily, the message is unlocked with a `retry_after` of `5 minutes × 1.3^attempts` and `attempts` is incremented. Once `attempts` reaches `postal.default_maximum_delivery_attempts` (default 18) the message is hard failed. The time between a message being queued and a worker picking it up is exported as the `postal_message_queue_latency` metric. + +### Batching + +When `postal.batch_queued_messages` is enabled (the default), a worker that has claimed a message also claims up to 100 further ready messages with the same **batch key** and IP address. The batch key is the destination domain for outgoing mail, or the route and endpoint for incoming mail. All messages in a batch are delivered over a single SMTP connection (or to the same HTTP endpoint), and if the connection cannot be established the whole batch is soft-failed at once rather than each message retrying the connection. Set `postal.batch_queued_messages: false` to process strictly one message at a time. + +### Delivering webhooks + +The `ProcessWebhookRequestsJob` claims one pending webhook request at a time in the same way and delivers it. See [Webhooks](/developer/webhooks#how-webhooks-are-delivered) for the retry schedule. + +### Stale locks + +If a worker crashes or is killed while holding a lock, the message stays locked. Once an hour (at :45) the `TidyQueuedMessagesTask` deletes any queued message whose lock is older than `postal.queued_message_lock_stale_days` (default 1 day). These messages are removed from the queue **without** being retried or recording a delivery, so if you see messages disappearing check for worker crashes in the logs. + +## Scheduled tasks + +Housekeeping is performed by scheduled tasks which run inside the worker. To ensure each task runs only once regardless of how many workers you have, workers hold an election for the `tasks` role using the `worker_roles` table: every 60 seconds each worker's tasks thread tries to acquire the role, which succeeds if it already holds it, if nobody holds it, or if the current holder has not renewed it for 5 minutes (i.e. it has died). Only the holder runs due tasks. The acquisition is logged (`acquired task role by creating it` / `by stealing it from a lazy worker`), and the role is released on clean shutdown. + +The next run time of each task is stored in the `scheduled_tasks` table so that schedules survive restarts. Times are in UTC. + +| Task | Schedule | What it does | +|---|---|---| +| `SendNotificationsScheduledTask` | Every minute | Sends send-limit approaching/exceeded e-mails and webhooks for servers that have recently crossed a threshold (at most once per hour each). | +| `CheckAllDNSScheduledTask` | Hourly at :15 | Re-checks SPF/DKIM/MX/return path for domains last checked over an hour ago, and CNAMEs for tracking domains. Triggers `DomainDNSError` webhooks. | +| `ExpireHeldMessagesScheduledTask` | Hourly at :15 | Cancels the hold on held messages whose hold expiry (`postal.default_maximum_hold_expiry_days`) has passed. | +| `ActionDeletionsScheduledTask` | Hourly at :15 | Permanently destroys organizations and servers that have been deleted from the interface, including dropping their message databases. | +| `CleanupAuthieSessionsScheduledTask` | Hourly at :15 | Removes expired web login sessions. | +| `PruneWebhookRequestsScheduledTask` | Hourly at :45 | Deletes webhook request history older than 10 days from each server's database. | +| `TidyQueuedMessagesTask` | Hourly at :45 | Removes queued messages with stale locks (see above). | +| `ProcessMessageRetentionScheduledTask` | Daily at 03:00 | Applies each server's [retention settings](/features/mail-server-settings#retention), dropping old raw message tables and deleting old message metadata. | +| `PruneSuppressionListsScheduledTask` | Daily at 03:00 | Removes expired entries from each server's suppression list. | + +Each task's run time is exported as the `postal_worker_task_runtime` metric. + +## Scaling + +* To handle more mail, increase `worker.threads` and/or run more worker processes (see [the postal command](/getting-started/postal-command#number-of-workers) for the standard installation). Watch `postal_message_queue_latency` to see whether messages are waiting. +* Workers on different hosts are fine, but remember the IP pool affinity rule: a message allocated to an IP address is only processed by a worker on a host that has that address. +* MariaDB is the coordination point. Each worker holds `threads + 3` connections, so size `max_connections` accordingly. +* Workers can be restarted safely with `SIGTERM` (which `docker stop` sends): they finish the message they are processing before exiting. If a worker is killed abruptly, any message it had locked stays locked and is eventually removed by the stale lock task rather than retried, so avoid `SIGKILL` where possible.