SendSetsDocs

Data control

Where every store lives on disk, what each retention window governs, how backups work, and how to move an instance to another host.

Self-hosting SendSets means holding mailbox credentials, message bodies and contact records on your own disk. This page is the whole answer to where that data is, how long it stays, and how you get it off this machine.

Three questions, and they have different answers:

QuestionAnswer
Where is it?Six stores, each pointed at by one variable
How long does it stay?Retention windows, all editable in the admin panel
How do I move it?sendsetsctl backup for the instance, workspace export for one workspace

Where the data sits

Every store's location is one variable in .env. Compose reads a source starting with / as a bind mount and anything else as a named volume, so the same variable covers both and there is no second compose file.

VariableHoldsDefault
SENDSETS_PG_DATAPostgres: organizations, users, mailboxes (credentials sealed), contacts, campaigns, the audit trail<data root>/postgres
SENDSETS_BLOBSMessage bodies, attachments, avatars, logos and email body images<data root>/blobs
SENDSETS_NATS_DATAThe event bus's JetStream state<data root>/nats
SENDSETS_REDIS_DATACache and rate-limit counters. Disposable<data root>/redis
SENDSETS_WORKER_STATEA worker's own id and sync cursors. Disposable<data root>/worker
SENDSETS_UPDATER_STATEThe last update job's log. Disposable<data root>/updater

An install from install.sh points all six under one data root, so /opt/sendsets/data is the whole of it and you can rsync that path. A clone-and-build install keeps Docker named volumes unless you set the same variables.

Only the first two carry anything you cannot rebuild. The other four are state a fresh container reconstructs, which is why a backup does not include them.

Encryption

Two keys, and they are not interchangeable.

KeyOpensRead by
CREDENTIALS_ENCRYPTION_KEYMailbox SMTP and IMAP credentials, and OAuth tokensBackend and workers, without an organization context
KMS_LOCAL_MASTER_KEYThe per-organization data keys, which in turn open everything elseBackend and consumer

Both are unrecoverable. A database backup without them restores an instance whose mailboxes authenticate against nothing, and there is no way back from that. They are in .env, and install.sh also writes them to keys-backup.txt next to the install, which is on the same disk as the database and therefore not a backup either. Copy them somewhere else.

External stores

Postgres, Redis and blob storage each accept an external target, set at install time or by editing .env:

PRIMARY_DB=postgres://user:pass@db.internal:5432/sendsets?sslmode=require
REDIS=redis://cache.internal:6379
BLOB_PROVIDER=s3
BLOB_BUCKET=sendsets
AWS_ENDPOINT_URL_S3=https://<account>.r2.cloudflarestorage.com

Filesystem blobs stop working when workers run off-host

A remote worker writes bodies to its own disk, so the dashboard on the control-plane host finds nothing. Any deployment with workers on more than one machine needs S3-compatible blob storage. This is the one storage choice that is not just a preference.

What is kept, and for how long

Every window below lives in the database, not the environment, and is edited under Instance > Configuration > Settings in the admin panel. A sweep runs a few times a day and reads the current value on every pass, so a change takes effect without a restart.

Mailbox import and sync

SettingDefaultRangeWhat it governs
sync.backfill_days901 to 730How far back the initial import reaches when a mailbox is connected, newest first
sync.backfill_messages5,0001 to 100,000The most messages that import stores per mailbox
sync.daily_messages_per_mailbox2,0001 to 100,000New mail one mailbox may store per UTC day
sync.daily_messages_per_org25,0001 to 2,000,000New plus imported mail across one workspace per UTC day

Mail over a daily budget is deferred, never dropped: the provider cursor is held and the mail is re-offered on the next pass. Replies to the mailbox's own outreach ride a separate budget of the same size and keep landing regardless.

Automatic inbox tagging

Automatic inbox tagging is off unless both TYPESAFE_API_KEY and INBOX_TAGGING_ENABLED=true are set. When enabled, the consumer sends the inbound subject and bounded plain-text body, plus the previous outbound message in the thread when one exists, to TypeSafe for classification. The result and raw probabilities are stored in inbox_tag_results. Completed results travel with the Inbox group in a workspace export; temporary processing claims do not. See Automatic inbox tagging.

Event history

SettingDefaultWhat it holds
retention.engagement_event_days365Per-event open and click logs: client, device, approximate location
retention.form_event_days180Form funnel events: views, starts, field-level drop-off
retention.audit_log_days90The audit trail: actor, IP address, user agent, change payload

Each is between 1 and 3,650 days. These are the three settings a retention or privacy policy applies to, because each window is also how long the personal data in that log is held.

None of them change a number anyone reads. Campaign progress keeps its own summary of opens and clicks that outlives the per-event log, so counts, filters and branching are unaffected by shortening any of these. What gets shorter is what a contact's timeline can show, how far a funnel report reaches, and how far back an admin can audit.

Shortening a window deletes on the next sweep

There is no grace period and no copy. Take a backup first if you are not sure.

The minimal retention preset in the admin panel and in the installer sets all three to 30 days.

Warmup standing by address

Warmup keeps the standing of every penalised mailbox (score, state, and any block) against its address as well as its row, so removing the mailbox or dropping out of the pool and coming back does not start it clean. The copy is cleared as soon as the mailbox recovers, and once the mailbox is gone it is kept for 90 days after the block ends or after the removal, whichever is later; a block that requires review is kept until an operator lifts it. A mailbox in good standing has no such copy. The window is fixed rather than a setting, and the rows are purged by the same sweep.

Pending inbox verification

Mail whose warmup verification is temporarily unavailable waits in the unibox_pending_emails Postgres table. It holds the sync event, including message metadata and bounded searchable body text, until verification succeeds or the mailbox is deleted. The consumer retries it automatically after restarts. Full bodies remain in blob storage. This pending queue is included in instance backups but excluded from workspace exports, whose destination re-syncs provider mail.

Disconnecting a mailbox

Disconnecting a mailbox, or deleting the workspace holding it, removes its rows from Postgres straight away through the database's own cascades. Two things cannot be done in that instant and are queued instead, then worked off within about a minute:

OwedWhat happens
The OAuth grantA Gmail mailbox's refresh token is sent to Google's revocation endpoint, which invalidates it and removes the instance from the customer's third-party access list. Microsoft publishes no per-application revocation endpoint, so an Outlook mailbox's tokens are destroyed locally and the customer removes the app themselves
The message bodiesEverything the mailbox wrote under users/<user>/emails/<mailbox>/ in the blob store is deleted, whether that is a filesystem path or an S3 bucket

While an erasure is outstanding its row holds a copy of the mailbox's OAuth refresh token, still sealed under CREDENTIALS_ENCRYPTION_KEY, because the row it was read from goes with the mailbox and the token is what the revocation sends. It is destroyed with the rest of the row when the erasure finishes.

The queue is the mailbox_erasures table, and it holds only work still owed: a finished erasure is removed rather than kept, because the row names the address the customer asked to have forgotten. A failure is retried with a widening backoff, capped at six hours, and never abandoned, so an unreachable provider or store delays the erasure instead of losing it. Anything still outstanding after a day is reported by the mailbox_erasure job, visible under scheduled jobs in the admin panel.

This runs on the control plane, which is the only place holding both the credential key and a credential for the blob store. A worker cannot do it: it signs one object at a time through the control plane and holds no bucket credential at all.

Set them at install time

The installer writes the answers into .env as one document, applied on the first boot of a fresh database:

SENDSETS_SETTINGS_BOOTSTRAP={"sync":{"backfill_days":30},"retention":{"audit_log_days":30}}

It is read only while the settings row has never been written. From the first save in the admin panel onwards the panel is authoritative, so leaving the line in .env never undoes a later edit.

Backups

sendsetsctl backup writes one bundle holding the three things that only restore together:

  • the database, as a pg_dump
  • the blob root, when blobs are on the filesystem
  • the encryption keys, unless you pass --no-keys
docker compose -p sendsets exec backend sendsetsctl backup --out /data/blobs/sendsets.tar.gz
docker compose -p sendsets cp backend:/data/blobs/sendsets.tar.gz ./sendsets.tar.gz \
  && docker compose -p sendsets exec -T backend rm -f /data/blobs/sendsets.tar.gz

/data/blobs is a hand-off, not a destination: it is the one path the container and the host both see. The && matters twice over. backup leaves its own output out of the archive, but a bundle left there is swept into the next run, so it has to be deleted; and a cp that failed must not be followed by deleting the only copy that exists.

The bundle is written 0600 and holds every mailbox credential on the instance plus the keys that open them. Treat the file as you would the instance itself.

An install from install.sh --wizard can schedule this for you: backup.sh next to the install, a systemd timer, a retention count, and an optional aws s3 cp to somewhere off the host. A backup that only exists on the machine it backs up is not one.

Restore

On the destination host, with the same keys in place:

docker compose -p sendsets exec backend sendsetsctl restore --file /data/blobs/sendsets.tar.gz
docker compose -p sendsets restart

The restore empties the schema and replays the dump, so it replaces everything currently on that instance and asks you to type restore first.

Before it does anything it compares the bundle's CREDENTIALS_ENCRYPTION_KEY and KMS_LOCAL_MASTER_KEY against the destination's and refuses to continue when they differ, printing the two lines to put in .env. That check is the point of the command: without it a restore looks like it worked and every mailbox fails to authenticate days later, with no error that names the cause.

Moving an instance

Two ways, and they answer different questions.

The whole instance, to a new host

Every workspace, every user, the platform admins, the API keys.

Install SendSets on the new host

curl -fsSL https://sendsets.com/install.sh | sh -s -- --host <new-hostname>

Put the old keys in the new .env

Copy CREDENTIALS_ENCRYPTION_KEY and KMS_LOCAL_MASTER_KEY from the old install, then recreate the containers so they take:

docker compose -p sendsets up -d

Restore the bundle

docker compose -p sendsets exec backend sendsetsctl restore --file /data/blobs/sendsets.tar.gz
docker compose -p sendsets restart

Check it

docker compose -p sendsets exec backend sendsetsctl status

Mailboxes should be connected, not needing a reconnect. If they need one, the keys did not match.

The rsync alternative works too and is sometimes simpler, as long as the stack is stopped first: a running Postgres data directory copied file by file is not a consistent snapshot and can restore as a corrupt cluster. Stop it, copy the data root and the .env to the new host, start it there. It moves the same bytes; the bundle exists because it is the version that survives a different host layout, a different Postgres, a live instance, and a partial copy.

One workspace, to another instance

Workspace export and import moves a single organization's data between two running instances, re-sealing its secrets for the destination's keys. That is the per-customer tool; the bundle here is the instance-level one. They are not interchangeable: a bundle cannot be applied to one workspace, and a workspace archive cannot restore an instance.

Outbound calls

A self-hosted SendSets makes no outbound call of its own except one, and it is off with a single setting.

CallWhenTurn it off
GitHub releases APIEvery 30 minutes, to tell the admin panel a newer version existsUPDATE_CHECK_ENABLED=false

Everything else is you: mail through the mailboxes you connect, DNS lookups for the domains you check, and whatever integrations you configure. There is no telemetry, no phone-home, and no license check.

Error reporting

An instance reports errors nowhere unless you point it somewhere. Every service reads its own credentials and none of them ship with any, so a default install sends no crash, no stack trace and no browser error to anybody, including us.

Two backends are supported and you choose either, both, or neither. PostHog is the default one; Sentry works exactly as it always has.

ServicePostHogSentry
Backend, consumer, workerPOSTHOG_KEYSENTRY_DSN
Forms servicePOSTHOG_KEYSENTRY_DSN
Tracking servicePOSTHOG_KEYSENTRY_DSN
Realtime servicePOSTHOG_KEYSENTRY_DSN
Dashboard and admin containersSENDSETS_POSTHOG_KEYSENDSETS_SENTRY_DSN
Public form pagesSENDSETS_POSTHOG_KEY on the forms serviceSENDSETS_SENTRY_DSN on the forms service

POSTHOG_HOST decides which PostHog receives them, including one you run yourself, and POSTHOG_ERROR_TRACKING=false turns error reporting off while leaving the key configured for the product analytics below. A Sentry DSN names whatever Sentry Cloud project, self-hosted Sentry or Sentry-compatible server you like.

Leave them unset, which is the default the installer writes, and nothing leaves the process: there is no host to contact and nothing to opt out of. Errors still reach that service's own log, as they always did. The dashboard, the admin panel and form pages go further and load no reporting code at all, so there is not even a script to block.

What a reported error carries is the exception, its stack trace, the service, the environment and the build, plus enough context to answer it:

  • Which workspace and user. A browser exception carries the organization id and user id of whoever hit it, and a server-side one carries them for the request that panicked, along with its route and request id. That is what makes "this customer says campaigns are broken" a search rather than a guess. On the server side that is a property and nothing more: no person is identified. In the dashboard and the admin panel, with a browser key set, the signed-in user is identified to PostHog (a person profile with their email and name, and an identifier in local storage and a cookie), so the exception belongs to that person; see usage analytics below.
  • What led to it. The last routes visited and the last API calls that failed, with their method, path, status and request id.
  • Nothing else about the request. No headers, no bodies, no query strings. Session replay exists only in the dashboard and the admin panel, only with a browser key set, and SENDSETS_POSTHOG_SESSION_REPLAY=false turns it off; the marketing site and public form pages are never recorded.

Server-side reports that belong to no request, a scheduled job or a worker, are attributed to the process that raised them and to nobody.

The sendsets CLI runs on your own machine and reports nowhere, ever. It has nothing to set.

Usage analytics

There are none. A self-hosted instance loads no analytics script and sends no usage data, not aggregated, not anonymised, not "to help us improve the product".

The hosted service at sendsets.com does measure its own marketing site, dashboard and admin panel, including session replay in the two apps, and the code for that ships in the same images you run. It is inert without a key: the installer never asks for one, the .env template does not mention it, and a build without one contains no analytics script for you to block. If you want to check, grep posthog the served assets of your own dashboard.

If you do want product analytics on your own instance, the variables are on configuration and they accept a self-hosted PostHog. The server-side POSTHOG_KEY carries the backend's error tracking and its product events; the browser-side SENDSETS_POSTHOG_KEY is what turns on the dashboard's and the admin panel's analytics, session replay and browser errors, and SENDSETS_POSTHOG_ERROR_TRACKING and SENDSETS_POSTHOG_SESSION_REPLAY switch those last two off separately. That is your decision to make, and nothing about it points at us.

See also

On this page