Instance health

SendSets checks its own deployment and tells you what is wrong in the words of the thing that is wrong, with the command that fixes it. A silent instance and a healthy instance look identical from the outside, so the checks exist to make the difference visible.

Setup and health

The page lives in the admin panel on :5174, under Instance > Setup and health, on its Findings tab; the Services tab next to it holds the per-service probes and the platform mail card. It is operator tooling, so it is not in the customer-facing dashboard, and every route behind it is gated on a platform admin permission.

The same list is the top panel of the admin overview, and the sidebar entry carries a badge with the number of open findings, because nobody visits a health page speculatively.

Only findings are listed. A check that passes returns nothing, so an empty page means the instance is clean.

Each row carries a title, the full message including the fix, and a link to the section of these docs that explains it. Every check that ships today is instance-wide, so no row is scoped to a single mailbox, worker or campaign.

Every check that runs is listed below. If a check is not on this page, it does not exist.

Running the checks from a shell

make doctor

That runs sendsetsctl status, which prints this instance's state (accounts, platform admins, registration mode, mail transport, APP_URL) followed by a Checks section holding the same findings, and exits non-zero when anything is at error severity. It is the right thing to put at the end of a deploy script, and it is what to reach for when the backend is unhealthy enough that the admin panel will not load.

Every command works in any runtime:

docker compose -p sendsets exec backend sendsetsctl status   # docker compose
docker exec -it sendsets-backend sendsetsctl status          # plain docker
kubectl exec -it deploy/sendsets-backend -- sendsetsctl status
sendsetsctl status                                          # bare binary

status is one of nine commands. The rest, including the ones that create an account and reset a password when nobody can sign in, are on the sendsetsctl reference.

Health endpoints

There are two, and they answer different questions.

Two neighbouring endpoints round out the surface, both under Instance in the panel:

Security and secrets

secret_published_default

One or more of AUTH_SECRET, KMS_LOCAL_MASTER_KEY, CREDENTIALS_ENCRYPTION_KEY, INTERNAL_API_TOKEN or SECRET_KEY_BASE still holds the value shipped in docker-compose.yml. Those values are published in this repository, so anyone can forge a session token or unwrap every organization key.

Generate real values and restart. See secrets. The backend refuses to boot on a published default once APP_ENV is not dev, but only the backend does that check: the consumer and every worker start happily on one, which is why this check exists.

allow_insecure_defaults

ALLOW_INSECURE_DEFAULTS=true overrides the boot refusal above. Remove it once you have generated real values.

credentials_key_unset

CREDENTIALS_ENCRYPTION_KEY is empty, so mailbox SMTP and IMAP passwords are stored without being sealed. Set a 64 hex character key (openssl rand -hex 32) before connecting any mailbox, and back it up: losing it makes connected mailboxes unrecoverable. See encryption.

internal_token_unset

INTERNAL_API_TOKEN is empty, so every request to /api/v1/internal/ is rejected. Workers cannot fetch organization keys and the tracking service cannot resolve click links. Nothing fails at boot, only at runtime. See workers.

trusted_proxies_unset

The request that loaded this page carried an X-Forwarded-For header while TRUSTED_PROXIES is empty, so SendSets is recording your proxy's address as the client address. The per IP login limiter, session records, audit rows and API key IP allowlists all read the wrong address. Set TRUSTED_PROXIES to your proxy's CIDR. See network and proxy.

captcha_misconfigured

CAPTCHA_PROVIDER=turnstile with an empty TURNSTILE_SECRET fails every verification, which means nobody can sign in. Set the secret or set CAPTCHA_PROVIDER=none. See captcha.

turnstile_bypass_set

TURNSTILE_BYPASS_TOKEN is set. It is only honoured when APP_ENV=dev, so on a production deployment it does nothing except confuse the next person. Remove it.

dev_mode_public

APP_ENV is not prod while APP_URL is not a loopback address. Dev mode allows the published default secrets and enables debug logging. Set APP_ENV=prod. See deployment.

unsafe_webhook_urls

SENDSETS_ALLOW_UNSAFE_WEBHOOK_URLS=true lets customer webhooks point at http:// and private addresses, which means any workspace member can make the backend reach into your internal network. Development only. See webhooks.

tls_verification_off

SMTP_TLS_INSECURE_SKIP_VERIFY or MAIL_TLS_INSECURE is on, so TLS certificate verification is disabled for platform mail or for mailbox connections. Only legitimate for a relay using a private certificate authority.

URLs and reachability

app_url_unset

APP_URL and FRONTEND_BASE_URL are both empty, so password reset, invitation and setup links are being built against https://app.sendsets.com. Those links go to the hosted service, not to your instance, and a live reset token in one of them leaves your deployment. Set APP_URL to your dashboard origin. See addresses.

app_url_insecure

APP_URL uses http:// on a non-loopback host. Browsers refuse WebAuthn outside a secure context, so passkeys are disabled, and session cookies travel in the clear. Put the dashboard behind HTTPS.

app_url_host_mismatch

You reached the panel on one host while APP_URL names another. Emailed links are built from APP_URL, so they will point somewhere other than where people actually reach this instance.

cors_missing_origin

An origin that talks to the API is not in the resolved CORS list, so the browser blocks its requests. In practice the app loads and then every request fails. Add it to CORS_ALLOW_ORIGINS.

api_public_url_unset_oidc

OIDC_ISSUER_URL is set while both API_PUBLIC_URL and OIDC_REDIRECT_URL are empty, so there is no redirect URL and the single sign-on path is disabled. Set API_PUBLIC_URL to this backend's public base. See single sign-on.

oidc_discovery_failed

Discovery against the configured issuer failed at boot, so the single sign-on button is not shown. The message carries the underlying error. Check the issuer URL, its TLS chain, and that the backend can reach it.

app_origin_wildcard

Neither APP_ORIGIN nor a usable APP_URL is set, so the mailbox OAuth callback page posts the authorization code back to the dashboard with a wildcard target origin. Set APP_URL to your dashboard origin, or APP_ORIGIN if the dashboard is served somewhere else. Setting APP_URL alone clears this: the callback page derives its target from it.

tracking_domain_unreachable

TRACKING_DOMAIN is set but a probe from the backend did not get an answer, so open pixels and click links in campaign mail will not record. Recipients still receive the mail.

Platform mail

mail_transport_log

MAIL_TRANSPORT=log writes every message to the backend log instead of delivering it. Login codes, password resets, team invitations and notification digests will never arrive.

Invitations still work: copy the invite link from Settings > Members and send it yourself. See without a mail relay. To fix it properly, point MAIL_TRANSPORT at a relay. See platform mail.

mail_preflight_failed

The transport is meant to deliver, and the relay did not accept a connection. The message carries the raw SMTP dialogue. Nobody can reset a password or receive an invitation until this is fixed. This check stays quiet when mail_transport_log has already fired, so one incident is one row.

mail_identity_unset

EMAIL_ADDRESS or EMAIL_NAME is missing. The backend refuses to start without them, but the consumer only warns and silently disables all notification and digest email, so this instance can look healthy while sending nothing.

mail_from_domain_mismatch

Platform mail is sent from a domain that differs from the dashboard's. Mailbox providers may treat that as a mismatch. Only a problem if you did not intend them to differ.

login_code_demoted

AUTH_LOGIN_CODE=always was demoted to new_device at boot because the transport does not deliver. Otherwise nobody could ever complete a login. See login codes.

Updates

update_available

A newer SendSets exists: a release on the configured channel is newer than the running build, or the checkout is behind its branch. The message names both versions and how to apply it. With the updater running, the version pill in the top bar has an Update and restart button; without it, git pull && make up. See Updates.

updater_unreachable

UPDATER_URL is set but nothing answers there, so the update button cannot work. Start the updater (the updater compose profile, or the systemd unit on a bare-metal host), fix the address, or remove the variable to update by hand. See enabling the updater.

Accounts and access

registration_mode

Always reported, so the mode is never a surprise. invite_only means nobody creates an account from the sign-up form and people join through an invitation. true means signups are closed and invitations do not work either. See registration modes.

no_sign_in_method

DISABLE_PASSWORD_LOGIN=true with no OIDC, Google or Apple provider configured leaves no way to sign in to this instance at all. Set DISABLE_PASSWORD_LOGIN=false or configure OIDC_ISSUER_URL. A provider counts as configured only when every value it needs is set, which is what the next two checks report.

google_sign_in_incomplete

Some of the Google sign-in configuration is set and some is missing, so the button is not shown. GOOGLE_CLIENT_ID and GOOGLE_CLIENT_SECRET are both required, and the callback needs a redirect URI, which derives from API_PUBLIC_URL when GOOGLE_REDIRECT_URI is unset. The same check warns when the redirect URI points at the dashboard: it is served by the API. See sign-in methods.

apple_sign_in_incomplete

The same for Sign in with Apple, which needs APPLE_APP_ID (the Services ID), APPLE_TEAM_ID, APPLE_KEY_ID and APPLE_KEY_SECRET together, plus an HTTPS redirect URI. Apple refuses a plain-http return URL, so Sign in with Apple cannot run against a local http install.

single_platform_admin

Exactly one account holds platform admin permissions. If you lose access to it there is no way to grant admin from inside the product. Add a second from Accounts > Admins, or with sendsetsctl user grant-admin. See platform admins.

bootstrap_password_still_set

SENDSETS_BOOTSTRAP_PASSWORD is still in this deployment's environment. It is read only while the users table is empty, so it now does nothing except leave a plaintext password where docker inspect can read it. Remove it. See first owner.

setup_link_outstanding

The instance has no accounts and a single-use setup link is live. Find it with make claim, or print a fresh one with sendsetsctl setup-link. See first run.

expired_invitations

Some invitations have expired. They are no longer visible in the dashboard but still hold their address. Re-inviting the same address replaces the expired row. See invitations.

Workers

no_worker_heartbeat

No worker has checked in for more than five minutes while mailboxes are assigned to workers, so nothing is being sent or synced.

Check the worker process is running, then check ENCRYPTED_KEYS_BACKEND_URL and ENCRYPTED_KEYS_WORKER_TOKEN: an empty value lets the worker start, subscribe and never register, with no log line to tell you. See workers.

tracking_domain_shares_brand

TRACKING_DOMAIN is on the same registered domain as APP_URL or API_PUBLIC_URL. Every campaign's links and unsubscribe pages carry that host, so complaints against any one workspace can get it listed on a URL blocklist, and the listing reaches the domain that also serves your marketing site and your platform mail.

Use a separate registered domain for tracking, and move workspaces onto their own verified tracking domains so their link reputation is theirs rather than shared.

fleet_infra_unreachable

Nodes are checking in from more than one machine, and NATS_URL, REDIS or ENCRYPTED_KEYS_BACKEND_URL still names a host that only resolves here, such as a compose service name or loopback. A joining node inherits those values verbatim, so it enrols, keeps heartbeating over HTTP, and reaches neither the bus nor the cache. The fleet looks healthy the whole time, which is why this is a check rather than a log line. See split deployment.

More than one machine is read as more than one distinct address among live nodes, so scaled replicas on a single host do not trigger it.

codec_registry

CODEC_PROVIDER is avro and SCHEMA_REGISTRY_URL is empty. Avro resolves every event against a registry, so there is nothing to encode against. Set it, or set CODEC_PROVIDER=json, which needs no registry. See event bus.

Database

migrations_dirty

The schema_migrations row is marked dirty. The backend applies migrations at boot; a dirty row means one failed halfway and must be resolved before the instance is used. The message carries the version it stopped at.

warmup_pools_missing

Fewer than two warmup pools exist, so warmup cannot place a mailbox and every warmup task fails. Migration 000156 creates the free and premium pools on every instance and nothing else does; a data-only restore or a manual delete removes them, and the migration will not run a second time. Put them back under their fixed ids:

INSERT INTO warmup_pools (id, pool_type, name)
VALUES ('77777777-aaaa-0000-0000-000000000001', 'free', 'Free warmup pool'),
       ('77777777-aaaa-0000-0000-000000000002', 'premium', 'Premium warmup pool')
ON CONFLICT (id) DO NOTHING;

Redis

redis_unreachable

Redis is not reachable. Rate limits, the organization key cache, the realtime bridge and the first-run setup token are all down. See cache.

Realtime

websocket_unreachable

The backend turned WEBSOCKET_URL into a /health URL and the realtime service did not answer it, so the dashboard will not update live and presence stays empty. Anything below a 500 counts as an answer, so an auth wall does not trip this.

A reachable realtime service that still leaves the dashboard dead has a different cause, and no check reports it: AUTH_SECRET on the backend and JWT_SECRET on the realtime service must be identical or every token is rejected. Instance > Configuration > Environment shows a four character fingerprint for each sensitive value so you can compare the two without either being disclosed.

Storage

blob_root_missing

BLOB_PROVIDER=filesystem and BLOB_FS_ROOT is unset, missing or not writable, so email bodies, attachments and avatars cannot be stored. The check writes and deletes a probe file to prove the path is usable. The backend, the consumer and every worker on the host must share that path. See storage.

fleet_blobs_not_shared

BLOB_PROVIDER=filesystem while nodes check in from more than one machine. A worker reads the message body the backend wrote, so a node that does not share this filesystem cannot send, and nothing says so until the last step of a send. Move to BLOB_PROVIDER=s3 with a bucket both sides reach. See split deployment.

What each service needs to be healthy

See also

• Configuration reference for every variable named here, including the realtime and tracking variables no check reads
• Accounts and access for the access checks
• sendsetsctl for status and every other operator command
• Troubleshooting for symptoms that do not come from a check