Split deployment

The self-hosting guide puts everything on one machine, which is the right answer until it is not. This page is the other shape: the control plane on a container host, the worker fleet on machines you own, and the durable pieces in a cloud region next to the control plane.

It is the deployment SendSets's own architecture describes. The control plane owns relational state and runs where a container host is convenient. The execution plane is a fleet of machines you rent, because a worker's value is being a distinct machine somewhere.

container host          machines you own        cloud region
──────────────          ────────────────        ────────────
backend                 worker                  Postgres
consumer                worker                  KMS key
realtime                nats + redis            S3 bucket
tracking                                        SES
forms
web / admin

Everything here is env-var configuration and manifests in deploy/split-cloud/. No code changes, no fork.

What goes where, and why

The consumer is the one people put in the wrong place. It is not a worker: it updates relational state, and it needs the database DSN a worker is deliberately never given. Running it beside the backend saves a database connection crossing the internet for every event.

Keep the three regions close

The backend reaches Postgres and Redis on every request, so the round trips between the three providers are the latency floor of the whole product. Pick a container-host region, a cloud region, and a datacentre for the machines that are near each other, and confirm it before you build anything on top.

Two things that are not optional

Blobs have to be object storage. A worker reads the message body the backend wrote. With BLOB_PROVIDER=filesystem it has neither the disk nor the permissions, and the failure arrives at the last step of a send with everything else looking healthy. sendsets join warns about this and starts the node anyway; the warning is the whole warning you get.

Nodes carry no cloud credentials

A machine in the fleet needs two privileged things: opening the sealed data key for an organization, and reading and writing message bodies. Both would normally mean an AWS credential on every box, in a file sendsets join rewrites on each run.

It does not work that way. When the control plane runs KMS_PROVIDER=aws or BLOB_PROVIDER=s3, a joining node is handed the brokered form instead:

A brokered provider authenticates with the internal API token the node already holds and asks the instance to perform the one privileged operation: unwrap this key, sign this blob operation. The instance token is scoped to one deployment and revocable from it, which an IAM access key on a rented machine is not.

Blob bytes are not proxied. The control plane signs a URL and the node transfers directly against the object store, so a mailbox sync costs the backend one small request per object rather than the bytes.

The cost is one HTTPS call per DEK open, which Redis caches, and one per blob operation. If you would rather a node talk to AWS directly, put KMS_PROVIDER, BLOB_PROVIDER and the credentials in /etc/sendsets/node.local.env, which a re-join does not overwrite.

Two limits are worth knowing. A signed URL covers one verb on one key, and only for the prefixes a node reaches: transport bodies, campaign attachments, and the mailbox bodies a sync stores. Avatars, form assets and workspace export archives cannot be signed for at all, so the credential a node holds is not a credential for the bucket.

That credential is NODE_BROKER_TOKEN. It falls back to INTERNAL_API_TOKEN, which is what a single-machine install uses, but set it to its own value here: the tracking and forms services are internet-facing and carry the shared token, and there is no reason for what they hold to be enough to open a data key. The same token covers the third broker route, which mints a provider access token for a mailbox SendSets Cloud manages.

Building it

scripts/aws-bootstrap.sh --domain example.com --region eu-central-1

curl -fsSLO https://raw.githubusercontent.com/AddisonHoff/SendSets/main/deploy/split-cloud/bus/setup.sh
sh setup.sh --domain bus.example.com

PRIMARY_DB=postgres://sendsets:...@db.us-east-1.rds.amazonaws.com:5432/sendsets?sslmode=verify-full&sslrootcert=/etc/ssl/rds/global-bundle.pem
REDIS=rediss://:<password>@bus.example.com:6380
NATS_URL=tls://<token>@bus.example.com:4222
ENCRYPTED_KEYS_BACKEND_URL=https://api.example.com
BLOB_PROVIDER=s3

sendsetsctl fleet join-token

# on each machine, as root
curl -fsSL https://api.example.com/join.sh | sh -s -- \
  --url https://api.example.com \
  --token <join-token> \
  --role worker \
  --region eu-central

The dashboard and admin panel on a static host

web and admin are static builds, so they do not need a container. Serving them from a CDN is cheaper and faster than running two nginx containers on your container host.

The one thing to know: both read window.__SENDSETS_ENV__ from /config.js, which the container entrypoint renders from environment at start. A static host has no container start, so the same script renders it at build time instead.

Build command:      pnpm build:pages
Build output:       dist
Root directory:     web        (or admin)

build:pages runs the normal build and then the app's own entrypoint with SENDSETS_CONFIG_OUT=dist/config.js. One definition of the key set serves both paths, so the container and the static host cannot drift apart.

Set these as build environment variables on the host:

Whichever origin you serve them from has to be in the backend's CORS_ALLOW_ORIGINS, or the dashboard loads and every API call fails preflight.

Anything the control plane cannot know

sendsets join writes /etc/sendsets/node.env from the control plane's answer and rewrites it on every join. Next to it, /etc/sendsets/node.local.env is created once and never written again, and the container reads it second, so a name repeated there wins.

That is the place for a value this instance does not hold: a DSN kept in a secret manager rather than the environment, credentials for infrastructure of your own, a per-machine tuning knob.

printf 'PRIMARY_DB=%s\n' "postgres://..." >> /etc/sendsets/node.local.env
systemctl restart sendsets-consumer

Sizing the fleet

A worker defaults to a planning target of 100 assigned mailboxes. Every assigned mailbox counts as one, regardless of provider or whether warmup is enabled.

The target is a local planning value, not a sending quota or a tested maximum. Google and Microsoft enforce their sending and API limits per mailbox, tenant, or API project. Set SENDSETS_WORKER_CAPACITY in a machine's /etc/sendsets/node.local.env when benchmarks show that host should carry more or less than the conservative default, then restart the worker. Placement spreads mailbox counts, limits provider and tenant concentration, and keeps a mailbox on its existing worker when capacity allows so its provider sees a stable login address.

Give NATS and Redis their own machine once you have more than one worker. Until then the control plane depends on a box that also runs a worker, and restarting that worker takes the bus with it.

What it costs

A starting instance, with one machine running the bus and the first worker:

Two things move that number. A cloud account under twelve months old gets the smallest database instance free, which takes roughly $14 off. Turning on a standby doubles the database line, so leave it off at the start and take snapshots instead.

The line that grows quietly is egress. Every query result and every message body a node reads crosses the internet, billed past the first 100 GB a month. It is invisible at launch and becomes the second-largest cloud line once mailbox sync is moving real volume.

Where things run out

One bus machine is one failure domain. Nothing sends while it is down. Snapshots and a documented rebuild are the honest answer at this size; NATS clustering is the answer when the instance is worth more than the afternoon it costs.

The database is reachable from the internet. That is what makes a container host and a fleet able to share it. Restrict the security group to the addresses that need it, force TLS, and treat the master password as the credential it is.

Auto-update covers nodes, not the control plane. The backend is what tells every node which version to be, so it upgrades the way the rest of your container host does. Upgrade it, and the fleet follows.