# Theirside docs

> Theirside watches the third-party APIs a Node backend calls and the webhooks it receives, learns what normal looks like for each endpoint, and alerts when a provider starts failing. Self-hosted: one Docker image plus Postgres.

These docs are for Theirside 0.6.3. Source pages: https://theirside.dev/docs/

<!-- https://theirside.dev/docs/getting-started -->

# Getting started

Theirside watches the third-party APIs your backend calls (and the webhooks they send you), learns what normal looks like for each endpoint, and alerts you when one starts failing, with a summary you can paste into a support ticket. It runs in your infrastructure: one Docker image and Postgres.

## 1. Run the collector

You need Docker. No clone, no configuration:

```sh
mkdir theirside && cd theirside
curl -fsSLO https://theirside.dev/compose.yml
docker compose -f compose.yml up -d --pull always --wait
docker compose -f compose.yml logs collector | grep -m1 "/setup"
```

The last command prints a one-time link: `Open http://localhost:4318/setup?token=… to create the first account`. `--pull always` fetches the newest release of the `0.6` line each time you run it; the collector listens on `127.0.0.1:4318`, this machine only.

Running it on a server rather than your laptop? Set `THEIRSIDE_PUBLIC_URL` in `compose.yml` to the address people will use, and open the link straight away: until you do, whoever has it can create the first account. See [Self-hosting](https://theirside.dev/docs/self-hosting).

## 2. Create your account

Open the link and create your account. The dashboard then shows **Connect your app**: your project API key and the `init()` line with this collector's URL filled in. Admins find the key again under **Settings → API keys**.

## 3. Add the SDK

```sh
npm install @theirside/sdk
```

Call `init()` once, before your app starts making requests. Keep the key out of your code:

```ts
import { init } from '@theirside/sdk';

init({
  apiKey: process.env.THEIRSIDE_API_KEY,
  endpoint: process.env.THEIRSIDE_COLLECTOR_URL, // http://localhost:4318 for now
  service: 'orders-api',
  environment: process.env.NODE_ENV,
});
```

Where exactly `init()` goes depends on your setup: [Node servers](https://theirside.dev/docs/node-servers) (Express, Fastify, NestJS) or [Serverless](https://theirside.dev/docs/serverless) (AWS Lambda, Workers, Deno, Bun). Without `apiKey` or `endpoint` the SDK stays off, so it is safe to deploy before the collector is ready.

`init()` uses the values it gets when it runs: `process.env.THEIRSIDE_API_KEY` is read at that moment. If your app loads its settings later (SSM, Secrets Manager, Vault, a config service), call `init()` with those values right after loading them: [Config loaded at runtime](https://theirside.dev/docs/node-servers#config-loaded-at-runtime). The SDK and the collector are versioned separately; any recent pair works together ([Versions](https://theirside.dev/docs/self-hosting#versions)).

## 4. See your first provider

Start your app and make it call an external API. Within a minute the **Providers** page lists it: every third-party host is grouped into a provider (`api.payco.test` and `files.payco.test` become `payco.test`; well-known APIs get their name), with each endpoint's calls per minute, errors and latency. A provider whose first calls are still in the current minute shows as **receiving** until that minute is complete. Your own infrastructure (localhost, private networks, internal names) is never recorded.

The SDK records calls made with `fetch` and undici, `http`/`https`, Axios, and libraries built on them (most HTTP clients and API SDKs, discord.js for example). Calls your cloud provider's SDK makes (AWS, Google Cloud, Azure) are grouped as **infrastructure**: monitored, with alerts off by default.

For the first day each endpoint shows **learning**: it is building its normal. Real outages still alert while it learns.

Nothing showing? See [Troubleshooting](https://theirside.dev/docs/troubleshooting#a-provider-doesnt-show-up).

## 5. Get alerts in Slack

1. In Slack, create an incoming webhook for the channel you want (Slack: **Apps → Incoming Webhooks**, or a Slack app with incoming webhooks turned on) and copy its URL.
2. In the dashboard: **Settings → Alert channels → Add channel**, choose **Slack (incoming webhook)**, paste the URL, save.
3. Press **Send test alert**. It must say "Sent" and arrive in the channel; if it doesn't, the error says why.

Email and signed webhooks work the same way (email needs SMTP settings: [Configuration](https://theirside.dev/docs/configuration#email)).

## Next

- [Webhooks](https://theirside.dev/docs/webhooks): the webhooks providers send you are monitored too.
- [Expected outcomes](https://theirside.dev/docs/expected-outcomes): when a provider answers a normal case with an error status.
- [Privacy](https://theirside.dev/docs/privacy): what the SDK sends, and what it never does.

<!-- https://theirside.dev/docs/node-servers -->

# Node servers

The SDK works on Node 18.18 and later, in any HTTP framework. It records the webhooks your server receives and the outgoing calls made with:

- `fetch` and undici, and libraries built on undici (discord.js, for example);
- `http`/`https`, and everything built on them: Axios, `@nestjs/axios`, got, node-fetch and most API SDKs.

Calls your cloud provider's SDK makes (AWS, Google Cloud, Azure) are recorded too, grouped as **infrastructure**: monitored, with alerts off by default. Calls are recorded from the moment `init()` runs, including through HTTP clients created before it; calls made before it are not.

## Where `init()` goes

### ES modules (`import`)

In ES modules every `import` runs before the code of the file that imports it, so an `init()` call below your imports misses any call made while those modules load. Put it in a file of its own and import that file first:

```ts
// instrumentation.ts
import { init } from '@theirside/sdk';

init({
  apiKey: process.env.THEIRSIDE_API_KEY,
  endpoint: process.env.THEIRSIDE_COLLECTOR_URL,
  service: 'orders-api',
  environment: process.env.NODE_ENV ?? 'production',
});
```

```ts
// server.ts
import './instrumentation.js'; // first line, before anything else
import express from 'express';

const app = express();
app.get('/health', (_req, res) => {
  res.send('ok');
});
app.listen(3000);
```

You can also load it from the command line, with no change to your entry file: `node --import ./dist/instrumentation.js dist/server.js`.

### CommonJS (`require`)

`require` runs in order, so `init()` at the top of the entry file is enough:

```js
// server.js
const { init } = require('@theirside/sdk');

init({
  apiKey: process.env.THEIRSIDE_API_KEY,
  endpoint: process.env.THEIRSIDE_COLLECTOR_URL,
  service: 'orders-api',
  environment: process.env.NODE_ENV || 'production',
});

const express = require('express');

const app = express();
app.get('/health', (_req, res) => {
  res.send('ok');
});
app.listen(3000);
```

TypeScript compiled to CommonJS (`"module": "commonjs"`) keeps your import order, so `import { init } …; init(…);` at the top of the entry file works there too. When in doubt, use the separate file: it works in both.

## Config loaded at runtime

`init()` uses the values it gets when it runs: in the examples above, `process.env.THEIRSIDE_API_KEY` is read at that moment. If your app loads its settings later, from SSM, Secrets Manager, Vault or a config service, those variables are still empty then and the SDK stays off. Call `init()` with explicit values right after the config is loaded, before your app creates its HTTP clients and starts making calls:

```ts
// server.ts
import { init } from '@theirside/sdk';
import express from 'express';

const config = await loadConfig(); // your loader: SSM, Secrets Manager, Vault, a config service…

init({
  apiKey: config.theirsideApiKey,
  endpoint: config.theirsideCollectorUrl,
  service: 'orders-api',
  environment: config.environment,
});

const app = express(); // created after init(), like any HTTP clients
app.get('/health', (_req, res) => {
  res.send('ok');
});
app.listen(3000);
```

Calls made before `init()` (the config loader's own, for example) aren't recorded; everything after is. On AWS Lambda, do the same at cold start: [Serverless](https://theirside.dev/docs/serverless#config-loaded-at-cold-start).

## Express

As above: import `./instrumentation.js` first in the file that creates the app. Nothing else to add. Webhooks your routes receive are picked up automatically ([Webhooks](https://theirside.dev/docs/webhooks)).

## Fastify

```ts
// server.ts
import './instrumentation.js';
import Fastify from 'fastify';

const app = Fastify();
app.get('/health', async () => 'ok');
await app.listen({ port: 3000, host: '0.0.0.0' });
```

## NestJS

Import the instrumentation file at the top of `main.ts`, before `NestFactory`:

```ts
// main.ts
import './instrumentation.js';
import { NestFactory } from '@nestjs/core';
import { AppModule } from './app.module.js';

const app = await NestFactory.create(AppModule);
await app.listen(3000);
```

```ts
// app.module.ts: your module as it is; nothing to add for Theirside
import { Module } from '@nestjs/common';

@Module({})
export class AppModule {}
```

No shutdown code is needed: the SDK sends what is queued on `SIGTERM` by itself (at most 2 s), also when the app calls `app.enableShutdownHooks()`. With SDK 0.4.1 or older and `enableShutdownHooks()`, add `await shutdown()` in an `onApplicationShutdown()` hook: Nest ends the process before those versions have sent.

A Nest project that compiles to CommonJS (the default `nest new` setup) can call `init()` at the very top of `main.ts` instead of using a separate file.

## Service and environment names

- `service` names the app in the dashboard, alerts and SDK health: one per deployable (`orders-api`, `billing-worker`). Several instances of the same app share it.
- `environment` keeps staging traffic from mixing with production: each environment learns its own normal and has its own incidents. Default: `production`.
- Use the same names everywhere a service runs; renaming one starts a new series.

## Shutting down

On `SIGTERM` and when the event loop empties, the SDK sends what is queued (at most 2 s) and lets your app exit as before. If another `SIGTERM` handler ends the process itself (re-raising the signal or calling `process.exit()`, as NestJS does with `enableShutdownHooks()`), the SDK holds that until it has sent, then exits the same way. Nothing else is needed: the SDK never keeps a process alive and never blocks a request.

## Check it works

```ts
import { getStats } from '@theirside/sdk';

// undefined: the SDK is off (no apiKey or endpoint). Otherwise `sent` grows and
// lastFlushError stays undefined once the app has made a few calls.
console.log(getStats());
```

The service then shows under **Settings → SDK health** in the dashboard, and its providers on the **Providers** page within a minute. Anything else: [Troubleshooting](https://theirside.dev/docs/troubleshooting).

<!-- https://theirside.dev/docs/serverless -->

# Serverless

A function's process can be frozen the moment it returns, so the SDK can't send in the background there. It sends at the end of each invocation instead, within a time cap, and never makes a failing collector your problem: if the collector is slow or down, the invocation waits at most the cap.

## AWS Lambda

Wrap the handler with `wrap()`. It runs your handler, then sends what the invocation recorded (at most 300 ms, and never past the invocation's deadline), then returns your result or rethrows your error unchanged:

```ts
// handler.ts
import { init, wrap } from '@theirside/sdk';

init({
  apiKey: process.env.THEIRSIDE_API_KEY,
  endpoint: process.env.THEIRSIDE_COLLECTOR_URL,
});

export const handler = wrap(async (event: unknown) => {
  return chargeCard(event); // your code, calling providers as usual
});
```

- Call `init()` at the top level of the module, outside the handler: it runs once per cold start.
- `service` defaults to the function name; the SDK notices it is on Lambda and switches to serverless mode by itself.
- Set `THEIRSIDE_API_KEY` and `THEIRSIDE_COLLECTOR_URL` in the function's environment (or read them from your secrets store and pass them in). The collector must be reachable from the function: same VPC, or over HTTPS.
- It works the same behind **API Gateway** (REST and HTTP APIs), an **Application Load Balancer** and a **function URL**, and for non-HTTP triggers (SQS, EventBridge, schedules).

### Config loaded at cold start

`init()` uses the values it gets when it runs. If the function loads its settings at cold start (SSM Parameter Store, Secrets Manager, a config service) instead of having them in its environment, call `init()` right after loading them, still at the top level, before the handler runs:

```ts
// handler-config.ts
import { init, wrap } from '@theirside/sdk';

const config = await loadConfig(); // once per cold start

init({
  apiKey: config.theirsideApiKey,
  endpoint: config.theirsideCollectorUrl,
  inbound: true,
});

export const handler = wrap(async (event: unknown) => chargeCard(event));
```

That needs an ES module handler (top-level `await`). A CommonJS handler can initialise on its first invocation instead; outgoing calls are recorded from then on, but that one invocation's own webhook delivery isn't, because `wrap()` looks for it before your handler runs:

```js
// handler.js
const { init, wrap } = require('@theirside/sdk');

let ready;
exports.handler = wrap(async (event) => {
  ready ??= loadConfig().then((config) =>
    init({ apiKey: config.theirsideApiKey, endpoint: config.theirsideCollectorUrl, inbound: true }),
  );
  await ready;
  return chargeCard(event);
});
```

### When events are sent

Each invocation sends its events right after your handler returns, before Lambda freezes the process. A reachable collector adds about 1 ms; the cap is 300 ms and can be changed:

```ts
// handler-with-cap.ts
import { init, wrap } from '@theirside/sdk';

init({ apiKey: process.env.THEIRSIDE_API_KEY, endpoint: process.env.THEIRSIDE_COLLECTOR_URL });

export const handler = wrap(async (event: unknown) => chargeCard(event), { flushTimeoutMs: 150 });
```

Events that didn't make it within the cap stay queued and go with the next invocation of the same container.

If your handler already flushes in its own `finally`, keep `wrap()` around it and set `flushTimeoutMs: 0`: `wrap()` then adds nothing, except after a webhook delivery (below), which it must send itself.

### Webhooks on Lambda

Webhook deliveries that reach your function through API Gateway, an ALB or a function URL are recorded through `wrap()` ([Webhooks](https://theirside.dev/docs/webhooks)). Add `inbound: true`, so a cold start records them at once instead of waiting for the collector to confirm it stores webhooks:

```ts
// webhooks.ts
import { init, wrap } from '@theirside/sdk';

init({
  apiKey: process.env.THEIRSIDE_API_KEY,
  endpoint: process.env.THEIRSIDE_COLLECTOR_URL,
  inbound: true,
});

export const handler = wrap(async (event: { body?: string }) => {
  if (!verifySignature(event)) return { statusCode: 401, body: '' };
  await handleWebhook(event.body);
  return { statusCode: 200, body: '' };
});
```

The status recorded is the one the sender got: your `statusCode`; a thrown error counts as 500 on HTTP APIs and function URLs and 502 on REST APIs and ALB. A delivery is recorded when your handler returns, so it is always sent by `wrap()`, after any flush of your own.

## Cron jobs and scripts

Wrap the main function, or call `await flush(1000)` before the process exits:

```ts
// sync.ts
import { init, wrap } from '@theirside/sdk';

init({ apiKey: process.env.THEIRSIDE_API_KEY, endpoint: process.env.THEIRSIDE_COLLECTOR_URL });

await wrap(async () => {
  await syncInvoices();
}, { flushTimeoutMs: 1000 })();
```

Google Cloud Functions, Azure Functions and Netlify Functions: wrap the handler the same way. Outgoing calls are recorded there; incoming webhooks are not (yet).

## Cloudflare Workers, Deno, Bun

These runtimes get a fetch-only build of the SDK automatically (through the package's export conditions; `@theirside/sdk/edge` if you need to name it). It records the `fetch` calls your code makes and sends with `fetch` too. Use `wrapFetch()`: it returns your response first and sends afterwards with `waitUntil`, so the user never waits for it:

```ts
// worker.ts
import { init, wrapFetch } from '@theirside/sdk';

init({ apiKey: 'tsd_live_…from a secret', endpoint: 'https://theirside.example.test' });

export default {
  fetch: wrapFetch(async (request: Request) => {
    const res = await fetch('https://api.payco.test/v1/charges', { method: 'POST' });
    return new Response(String(res.status));
  }),
};
```

On Workers, read the key from a secret binding and keep `init()` at the top level. On Vercel, pass its `waitUntil`: `wrapFetch(handler, { waitUntil })` with `waitUntil` from `@vercel/functions`.

**Not covered on these runtimes:** incoming webhooks, and `markExpected()` (there's no `AsyncLocalStorage`; `expecting()` covers calls started synchronously inside it). On Workers, events go out only when a request flushes.

<!-- https://theirside.dev/docs/webhooks -->

# Webhooks

The SDK also watches the webhooks your app receives: on any Node HTTP server (Express, Fastify, NestJS, plain `http`) with no extra code, and on AWS Lambda through `wrap()` with `inbound: true` ([Serverless](https://theirside.dev/docs/serverless#webhooks-on-lambda)). Each delivery is one event: the sender, the route, the status your app answered, how long your handler took and, when the sender signs one, when it was sent. Nothing else from the request is read.

## What alerts

Each sender and route learns its own normal; the **Webhooks** page shows it per route.

| Incident | When | What it usually means |
|---|---|---|
| **Silence** | No deliveries for longer than this sender's rate for this hour of the week allows ("No PayCo webhooks for 2h 10m; normally ~40/hour at this time") | The sender stopped, or your endpoint became unreachable |
| **Rejections** | Your app answers non-2xx to more deliveries than normal (401/403 named separately) | A rotated signing secret, a deploy that broke the handler |
| **Slow handling** | Your handler takes longer than normal | Your side, before the sender starts retrying or timing out |
| **Delivery lag** | Deliveries arrive long after the sender signed them | The sender's queue is backed up |

Silence alerts start once a route has three weeks of history, and only for regular senders: one whose normal gaps are too irregular to judge (traffic from tests or batch jobs) is marked so, with silence alerts off unless you switch them on for that route. When every route goes quiet at once, silence alerts are held: that looks like your receiver, not every sender at the same time.

## Known senders

Recognised by their signature headers, nothing to configure: Stripe, GitHub, GitLab, Shopify, Slack, SendGrid, Twilio, Checkout.com, Paddle, PayPal, Square, HubSpot, Zoom, Twitch, Calendly, Typeform, Razorpay, Linear, Vercel, Svix and Standard Webhooks.

## Senders the SDK doesn't know: discovery

A POST or PUT that looks like a delivery from a sender the SDK can't identify is reported as a **discovered route**, at most once a minute per route, with a delivery count and a suggested name. It looks like one when it has any of:

- a route word: `/webhooks/…`, `/hooks/…`, `/callbacks/…` (a final `/events`, `/notify` or `/notifications` too, when there's no `Authorization` or `X-Api-Key` header);
- a header whose name contains `signature` or `hmac`;
- CloudEvents headers (`ce-specversion` with `ce-source` or `ce-type`);
- a host whose first label names webhooks (`webhook.example.test`).

Requests from browsers (cookies, `Origin`) are never candidates. The suggested name comes from the CloudEvents source's domain, the word next to the route word (`/webhooks/payco`), or the word after a version (`/v1/payco`).

Discovered routes appear at the top of the **Webhooks** page under **Discovered, not monitored yet**, and once in your alert channels. Nothing alerts on them until an admin confirms one:

- **Link to an existing provider**: the route belongs to a company you already monitor for outgoing calls. Its outgoing endpoints and webhooks then share one provider, with one alerts switch and one support contact. The best match is preselected with the reason ("same domain as payco.test").
- **Create new provider**: for a sender you don't call.

A confirmed route moves into the monitored list at once and is judged like any other from its first delivery. **Ignore** routes that aren't webhooks; the SDK stops reporting them.

## Declaring routes

Some senders can't be recognised from headers (they sign inside the body), and some platforms send the same headers for many customers. Name those routes yourself; declared routes win over detection:

```ts
import { init } from '@theirside/sdk';

init({
  apiKey: process.env.THEIRSIDE_API_KEY,
  endpoint: process.env.THEIRSIDE_COLLECTOR_URL,
  webhooks: [
    { path: '/webhooks/payco', provider: 'PayCo' },
    { path: '/hooks/:tenant/notifyco', provider: 'NotifyCo' },
  ],
});
```

Use your framework's route syntax (`:param` segments). `webhooks: false` turns inbound monitoring off; `webhookDiscovery: false` keeps known senders and declared routes but stops discovery.

## Why wasn't a webhook picked up?

Start the receiving service with `THEIRSIDE_DEBUG=inbound`, or pass `debug: 'inbound'` to `init()` if its settings don't come from the environment:

```ts
import { init } from '@theirside/sdk';

init({
  apiKey: process.env.THEIRSIDE_API_KEY,
  endpoint: process.env.THEIRSIDE_COLLECTOR_URL,
  debug: 'inbound',
});
```

It logs one line per incoming request saying what became of it and why, and one line per flush saying what was sent and the collector's answer:

```
[theirside] inbound POST /v1/payco: candidate /v1/payco, suggested name payco (signature header, cloudevents; name from ce-source)
[theirside] inbound POST /v1/orders: ignored, no webhook signal
[theirside] inbound POST /v1/cart: ignored, browser or front-end request (cookie header)
[theirside] inbound flush: 1 candidate to /v1/ingest/inbound: 202
```

The lines name rules, header names and routes only (IDs as `:id`), never a header value or a body, and go to your app's own console. `getStats().inbound` has the counters: `{ support, held, sent, dropped, candidates: { seen, sent } }`. More in [Troubleshooting](https://theirside.dev/docs/troubleshooting#a-webhook-isnt-discovered).

<!-- https://theirside.dev/docs/expected-outcomes -->

# Expected outcomes

Some APIs answer a normal business case with an error status: a consent check answers `400` when the user hasn't consented, a lookup answers `404` for "no such customer yet". Counted as failures, these make the endpoint's normal noisy, and a change in how often users hit them can open an incident. Tell Theirside which answers are normal; those calls then count as successful.

## In code: `markExpected` (recommended)

Call it after your code has read the answer and decided it's a normal case:

```ts
// consent.ts
import { markExpected } from '@theirside/sdk';

export async function hasConsent(userId: string): Promise<boolean> {
  const res = await fetch(`https://api.consentco.test/v1/consent/${userId}`);
  if (res.status === 400) {
    const body = (await res.json()) as { code?: string };
    if (body.code === 'consent_required') {
      markExpected('consent_required'); // a normal answer: the user hasn't consented yet
      return false;
    }
  }
  if (!res.ok) throw new Error(`consent check failed: ${res.status}`);
  return true;
}
```

`markExpected(reason)` tags the latest outgoing call that ended in the same incoming request, if it answered with a 4xx in the last 5 seconds. It works in any Node HTTP server (Express, Fastify, NestJS). Only the reason label is sent (letters, digits and `_ . : -`, up to 64 characters); the SDK never reads the response itself.

## In code: `expecting`

When you know up front, wrap the call instead. 4xx answers to the calls made inside, also from async work it starts, are tagged:

```ts
// consent-check.ts
import { expecting } from '@theirside/sdk';

export async function check(userId: string) {
  return expecting('consent_required', () => consentClient.check(userId));
}
```

Use `expecting()` outside an incoming request (queue consumers, cron jobs), and when several calls run at once in one request: `markExpected()` tags whichever ended last.

## Why only the calls you tag

Untagged calls with the same status stay failures. So if the provider changes its contract and starts answering `400` to everything, your code doesn't recognise those answers, doesn't tag them, and you still get an alert. Only 4xx answers can be expected; a 5xx, timeout or connection error never is. Tagged calls show as their own series per reason on the endpoint page.

## From the dashboard

Without a code change, mark a status as expected for a whole endpoint: from an incident ("Mark 404 as expected for POST /v1/lookup"), from the endpoint row, or in the provider's settings. Members and admins can do it, and an open incident it explains is resolved with a note. Slack alerts about 401/403, 404 or 400/409/422 have a **Mark as expected** button.

That is coarser: every answer with that status from that endpoint counts as fine. A **safety net** still opens an incident when the marked status takes over: above 80% of calls, or halfway from its normal share to every call if that is higher, with confidence, for about 10 minutes ("400 (marked expected) now 95% of calls, normally 30%").

## Which to use

| | In code | From the dashboard |
|---|---|---|
| Precision | Only the answers your code read as normal | Every answer with that status |
| Contract change ("400 to everything") | Alerts as usual | Alerts only once the status takes over |
| Effort | A line where you handle the answer | A click |
| Works on | Node servers; `expecting()` anywhere | Any SDK version |

Start from the dashboard to quiet an endpoint today; move to `markExpected` where you handle the answer when you next touch that code.

## Checking it

`getStats().expected` counts `tagged` calls and `missed` marks (a `markExpected` that found no 4xx to tag); with `debug: true` the SDK says why a mark missed.

<!-- https://theirside.dev/docs/self-hosting -->

# Self-hosting

The collector is one Docker image (`ghcr.io/theirside/collector`, amd64 and arm64) plus Postgres 13 or later. It receives events from your apps, runs detection and alerts, and serves the dashboard and status page on port 4318. Your data never leaves your infrastructure.

**Recommended size: 1 vCPU and 2 GB of memory**, one instance. That is plenty for most teams; the examples below use it.

## Docker Compose

```sh
mkdir theirside && cd theirside
curl -fsSLO https://theirside.dev/compose.yml
docker compose -f compose.yml up -d --pull always --wait
docker compose -f compose.yml logs collector | grep -m1 "/setup"
```

`compose.yml` runs Postgres and the collector; every setting in it is optional and commented. `--pull always` fetches the newest release of the minor line in the file (`:0.6`) each time. The collector listens on `127.0.0.1:4318`, so only that machine reaches it: on a server, put a TLS reverse proxy in front, or change the `ports` line to `'4318:4318'` to reach it from your network ([Behind a load balancer](#behind-a-load-balancer)). Put your settings in a `.env` file next to it (Compose reads it), for example:

```sh
THEIRSIDE_PUBLIC_URL=https://theirside.internal.example.test
POSTGRES_PASSWORD=change-me
THEIRSIDE_SECRET_KEY=output-of-openssl-rand-base64-32
THEIRSIDE_SMTP_URL=smtps://user:password@smtp.example.test:465
THEIRSIDE_SMTP_FROM=alerts@example.test
```

Then `docker compose -f compose.yml up -d --pull always --wait` again. Every variable: [Configuration](https://theirside.dev/docs/configuration#collector-environment-variables).

## First account and API key

On its first start, the collector creates your organisation and project, and tells you in its log how to create the first account. Find the line with `grep -m1 "/setup"` in the logs (`docker compose … logs collector`, `kubectl logs`, or the service's log group).

- **Nothing preset** (the default): it generates the project API key and logs a one-time link, `Open https://…/setup?token=… to create the first account`. Open it, create the account, and copy the key from **Connect your app**.
- **`THEIRSIDE_BOOTSTRAP_API_KEY` set** (your own key, `tsd_live_` + at least 16 characters, e.g. kept in your secrets store), or `THEIRSIDE_ADMIN_TOKEN` set: it logs `No user accounts yet. Open https://…/#/setup and create the first admin with setup code …`. Open that address and enter the code, valid 24 hours, or the admin token. Your apps use the key you set.

Until the first account exists, whoever has the link or code can create it: do it right after the first start.

## ECS Fargate

Run the image as a service in private subnets with RDS for PostgreSQL, behind an **internal** load balancer. Keep `DATABASE_URL` (`postgres://user:password@<instance>.rds.amazonaws.com:5432/theirside?sslmode=verify-full`) and the other secrets in Secrets Manager:

```json
{
  "family": "theirside-collector",
  "requiresCompatibilities": ["FARGATE"],
  "networkMode": "awsvpc",
  "cpu": "1024",
  "memory": "2048",
  "executionRoleArn": "arn:aws:iam::123456789012:role/ecsTaskExecutionRole",
  "containerDefinitions": [
    {
      "name": "collector",
      "image": "ghcr.io/theirside/collector:0.6",
      "essential": true,
      "portMappings": [{ "containerPort": 4318, "protocol": "tcp" }],
      "environment": [
        { "name": "THEIRSIDE_PUBLIC_URL", "value": "https://theirside.internal.example.test" },
        { "name": "THEIRSIDE_DATABASE_CA_FILE", "value": "/app/certs/rds-global-bundle.pem" }
      ],
      "secrets": [
        { "name": "DATABASE_URL", "valueFrom": "arn:aws:secretsmanager:eu-west-1:123456789012:secret:theirside/database-url" },
        { "name": "THEIRSIDE_SECRET_KEY", "valueFrom": "arn:aws:secretsmanager:eu-west-1:123456789012:secret:theirside/secret-key" }
      ],
      "healthCheck": {
        "command": ["CMD", "node", "-e", "fetch('http://127.0.0.1:4318/healthz').then(r=>process.exit(r.ok?0:1)).catch(()=>process.exit(1))"],
        "interval": 15,
        "timeout": 5,
        "retries": 3,
        "startPeriod": 30
      }
    }
  ]
}
```

- **RDS**: end `DATABASE_URL` with `?sslmode=verify-full` and set `THEIRSIDE_DATABASE_CA_FILE=/app/certs/rds-global-bundle.pem`, as in the task definition above: the image includes AWS's RDS CA bundle, so the database's certificate is checked with nothing else to install ([Database TLS](#database-tls)).
- **Load balancer target group**: health check on `/readyz`. One task is enough; more are fine (migrations are locked and one task runs the jobs).
- Can't pull from ghcr.io? Copy the image into ECR once per version (`docker pull`, `docker tag`, `docker push`).

## Kubernetes

A minimal setup with your own Postgres. The `theirside` secret holds `DATABASE_URL` and `THEIRSIDE_SECRET_KEY`:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: theirside-collector
spec:
  replicas: 1
  selector:
    matchLabels: { app: theirside-collector }
  template:
    metadata:
      labels: { app: theirside-collector }
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
      containers:
        - name: collector
          image: ghcr.io/theirside/collector:0.6
          ports:
            - containerPort: 4318
          env:
            - name: THEIRSIDE_PUBLIC_URL
              value: https://theirside.internal.example.test
          envFrom:
            - secretRef:
                name: theirside
          readinessProbe:
            httpGet: { path: /readyz, port: 4318 }
            periodSeconds: 10
          livenessProbe:
            httpGet: { path: /healthz, port: 4318 }
            initialDelaySeconds: 30
            periodSeconds: 15
          resources:
            requests: { cpu: '1', memory: 2Gi }
            limits: { memory: 2Gi }
---
apiVersion: v1
kind: Service
metadata:
  name: theirside-collector
spec:
  selector: { app: theirside-collector }
  ports:
    - port: 4318
      targetPort: 4318
```

```sh
kubectl create secret generic theirside \
  --from-literal=DATABASE_URL='postgres://theirside:password@postgres.db.svc:5432/theirside' \
  --from-literal=THEIRSIDE_SECRET_KEY="$(openssl rand -base64 32)"
kubectl apply -f theirside.yaml
kubectl logs deploy/theirside-collector | grep -m1 "/setup"
```

Apps in the cluster then use `THEIRSIDE_COLLECTOR_URL=http://theirside-collector.<namespace>.svc:4318`. `/readyz` answers 503 while the database is unreachable; `/healthz` stays 200, so a database blip doesn't restart the pod.

## Database and user

The collector needs one database it owns, on Postgres 13 or later. On Postgres 16 or RDS, connected as your admin user:

```sql
CREATE ROLE theirside WITH LOGIN PASSWORD 'change-me';
GRANT theirside TO CURRENT_USER; -- needed on RDS: the admin user isn't a superuser
CREATE DATABASE theirside OWNER theirside;
```

Then `DATABASE_URL=postgres://theirside:change-me@<host>:5432/theirside?sslmode=verify-full`. The collector creates its tables on first start; there's nothing else to run.

## Database TLS

To encrypt the connection to Postgres and check that it really is your database, end `DATABASE_URL` with `?sslmode=verify-full`. The certificate must then chain to a trusted CA and name the host in the URL.

- **AWS RDS**: set `THEIRSIDE_DATABASE_CA_FILE=/app/certs/rds-global-bundle.pem`. The image includes AWS's global RDS CA bundle at that path.
- **Another provider, or your own CA** (Cloud SQL, Azure, a Postgres with your company's CA): mount its CA certificate file into the container and point `THEIRSIDE_DATABASE_CA_FILE` at it.
- **A certificate from a public CA**: `?sslmode=verify-full` alone is enough.

`?sslmode=no-verify` also encrypts the connection but skips the certificate check, so anything on the network path could pose as your database: use it for local testing only. In PostgreSQL's own terms `require`, `prefer` and `allow` don't verify the certificate either; ask for `verify-full`.

The collector checks this at startup and never blocks on it. When the database is on another machine (anything but `localhost`, `127.0.0.1`, `::1`, a Unix socket or a single-label name like the Compose service `postgres`; private IPs count as another machine) and the certificate isn't verified, it logs one warning with the exact fix, and admins see the same notice under **Settings → SDK health**.

## Upgrades

1. Read the version's release notes: they say what changes and when to upgrade the collector before your apps' SDK.
2. Change the image tag, or keep the minor tag (`:0.6`) and pull: Compose with `docker compose -f compose.yml up -d --pull always --wait`, Kubernetes and ECS by rolling out the new image.
3. Database migrations run when the new version starts, before it serves traffic; there's nothing to run by hand.

With one instance, a deploy leaves a short gap while the new one starts. Your apps aren't affected: the SDK keeps events in memory (up to 10,000 per process) and sends them once the collector answers again. Run two instances if you also want the dashboard up during deploys.

A minor tag (`0.6`) moves to each patch release; a new minor (`0.7`) needs the tag changed.

### Versions

The SDK (`@theirside/sdk` on npm) and the collector (the image) are versioned separately. **SDK 0.4 and later work with collector 0.5 and later**, and older SDKs keep working with newer collectors: upgrade the collector first. Each release's notes say if a new SDK feature needs a newer collector.

## Backups

Everything is in Postgres: back it up like any other database (managed snapshots, or `pg_dump`). With Compose:

```sh
docker compose -f compose.yml exec -T postgres pg_dump -U theirside theirside | gzip > theirside-$(date +%F).sql.gz
```

Also keep `THEIRSIDE_SECRET_KEY` somewhere safe: without it, two-factor codes can't be checked after a restore.

## Behind a load balancer

- **Keep it private.** The dashboard and every `/v1/*` path (ingest, the APIs, sign-in) belong on a private network: an internal load balancer reachable from your apps and your office or VPN. Only `/status/*`, `/embed.js` and `/fonts/*` may ever be public (the status page, below).
- **TLS** at the load balancer; point apps' `THEIRSIDE_COLLECTOR_URL` at the HTTPS address, so API keys never cross the network in clear text.
- **`THEIRSIDE_PUBLIC_URL`**: the address people use, for links in alerts and secure cookies.
- **`THEIRSIDE_TRUST_PROXY`**: leave it unset unless your load balancer reaches the collector from a public address, or you want to trust only its subnets. By default the collector believes `X-Forwarded-For` from proxies on private and loopback addresses, which fits most setups.
- **Health check**: `/readyz`.
- **Company sign-in**: an identity proxy or an AWS ALB with OIDC can sign people in (`THEIRSIDE_AUTH_PROXY`, [Configuration](https://theirside.dev/docs/configuration#sign-in-and-security)).

## A public status page

The status page shows your providers' health to your own customers. To publish it without exposing anything else, run a second instance of the same image on the same database with `THEIRSIDE_PUBLIC_PATHS_ONLY=true` behind an internet-facing load balancer or CDN. It serves only `/status/*`, `/embed.js`, `/fonts/*` and health checks, answers 404 to everything else, and runs no jobs, so a misconfigured listener rule can't expose the dashboard. Choose what it shows under **Settings → Status page** and in each provider's settings.

<!-- https://theirside.dev/docs/configuration -->

# Configuration

Both tables are generated from the source code, so they always match the release you are reading.

## SDK: `init()` options

Only `apiKey` and `endpoint` are needed; everything else has a default that suits most apps.

```ts
import { init } from '@theirside/sdk';

init({
  apiKey: process.env.THEIRSIDE_API_KEY,
  endpoint: process.env.THEIRSIDE_COLLECTOR_URL,
  service: 'orders-api',
  environment: 'production',
  providers: [{ name: 'PayCo', hosts: ['api.payco.test'], routes: ['/v1/charges/:chargeId'] }],
  internalHosts: ['*.corp.test'],
});
```


| Option | Type | Default | What it does |
|---|---|---|---|
| `apiKey` | `string` | required | Project API key (`tsd_live_…`) with the `ingest` scope. Without it the SDK stays off, so it is safe to ship before the collector is set up. |
| `endpoint` | `string` | required | Collector base URL, e.g. `http://collector:4318`. Without it the SDK stays off. |
| `service` | `string` | the function name on AWS Lambda, Google Cloud Functions and Azure Functions, else `unknown-service` | Name of this app in the dashboard and alerts. |
| `environment` | `string` | `production` | Environment name, e.g. `production`, `staging`. Each environment learns its own normal. |
| `providers` | `ProviderConfig[]` | `[]` | Providers you want to name yourself: `{ name, hosts, routes? }`. Host patterns: `api.x.com`, `*.x.io` (any subdomain, not the apex), `localhost:4010`. Optional: every third-party host is recorded anyway (`autoDiscover`). |
| `autoDiscover` | `boolean` | `true` | Record calls to every third-party host, not only `providers`. Loopback, private networks, cloud metadata, internal names, telemetry hosts, the collector itself and `internalHosts` are never recorded. |
| `internalHosts` | `string[]` | `[]` | Your own services (host patterns): never recorded. |
| `remoteConfig` | `boolean` | `true` | Poll the collector every 60 s for dashboard settings (hidden hosts, routes, path privacy). Failures keep the last good settings. |
| `pathPrivacy` | `'strict' \| 'balanced'` | `'strict'` | How plain words in paths are treated (docs/privacy.md). `strict`: only known API words (the built-in vocabulary, or a provider's allowed words) are sent as-is, everything else becomes `:id`. `balanced`: any word seen 5 times at a position with at most 16 distinct values; only for paths that never carry personal data. The dashboard can set it per provider. |
| `getCorrelationId` | `() => string \| number \| null \| undefined` | none | Returns your request or correlation ID (e.g. from AsyncLocalStorage), sent with each call so you can find it in your logs. Max 128 characters. |
| `requestIdHeaders` | `string[]` | `['x-request-id', 'request-id', 'x-amzn-requestid', 'cf-ray']` | Response headers whose value is kept as the provider's request ID (for support tickets). `[]` disables. |
| `sampleRate` | `number` | `1` | Fraction of successful calls kept (0..1]. Errors are always kept; the collector re-weights the counts. |
| `flushIntervalMs` | `number` | `5000` (ms) | How often queued events are sent. A full batch is sent at once. |
| `maxBatch` | `number` | `500` | Events per request to the collector (max 5000). |
| `maxQueue` | `number` | `10000` | Events kept in memory; when full, the oldest are dropped and counted (`getStats().dropped`). The SDK never blocks your app. |
| `webhooks` | `boolean \| WebhookRouteConfig[]` | `true` | Inbound webhook monitoring (Node servers, AWS Lambda through `wrap()`). `true`: known senders detected by their signature headers. A list of `{ path, provider }` declares routes by hand (they win over detection). `false` turns inbound capture off. Requests that aren't webhooks are never recorded. |
| `inbound` | `boolean` | `false` | Record webhook deliveries from the first one, without waiting for the collector's remote config to say it stores them. Set it on AWS Lambda, where each cold start would otherwise lose the deliveries before that answer. A collector older than 0.4.0 rejects them and the SDK stops recording deliveries. Without it, a long-running server holds the first 500 deliveries (at most 60 s) until the remote config answers, and discards them if it says no or doesn't answer. |
| `webhookDiscovery` | `boolean` | `true` | Webhook discovery: also report POST/PUT requests that look like deliveries from a sender the SDK can't identify (a route like `/webhooks/…`, a signature or HMAC header name, CloudEvents headers, a webhook host). They appear in the dashboard to confirm or ignore, and never alert until confirmed. `false` turns it off; known senders and declared routes are still recorded. |
| `debug` | `boolean \| 'inbound'` | `false` | `true`: log SDK problems (bad config, flush failures, template overflow) to `console.warn`. `'inbound'`: log what became of each incoming request and of each flush's webhook deliveries (the same as `THEIRSIDE_DEBUG=inbound`, for apps whose settings don't reach `process.env`). |
| `enabled` | `boolean` | `true` | `false` makes `init()` do nothing (e.g. in tests). |
| `shutdownTimeoutMs` | `number` | `2000` (ms) | Cap for the flush done by `shutdown()` and on process exit. |
| `handleProcessExit` | `boolean` | `true` (`false` in serverless mode: no exit to hook) | Flush on `beforeExit` and `SIGTERM` (the default `SIGTERM` exit is kept). |
| `serverless` | `boolean` | detected from AWS Lambda, Google Cloud Functions, Azure Functions, Vercel, Netlify, Deno Deploy and Cloudflare Workers environment variables | Short-lived or frozen processes (functions, edge, cron, CLI): no exit hooks, no heartbeat, no keep-alive sockets; send with `flush()` or `wrap()` at the end of each invocation. |


The SDK also reads one environment variable itself: `THEIRSIDE_DEBUG=inbound` turns on the webhook debug lines, like `debug: 'inbound'` ([Webhooks](https://theirside.dev/docs/webhooks#why-wasnt-a-webhook-picked-up)).

## Collector: environment variables

None are needed for a first start with the [compose file](https://theirside.dev/docs/self-hosting#docker-compose): it sets `DATABASE_URL` and nothing else is required. Set them in `compose.yml`, your task definition or your Kubernetes manifest.


### Basics

| Variable | Default | What it does |
|---|---|---|
| `DATABASE_URL` | required | Postgres 13+ connection string, `postgres://user:pass@host:5432/db`. Migrations run on start (safe with several instances). |
| `THEIRSIDE_DATABASE_CA_FILE` | – | PEM file with the CA certificates the database’s TLS certificate must chain to; with it, TLS is on and the host name is checked too (use `sslmode=verify-full` in `DATABASE_URL`). The image includes AWS RDS’s global bundle: `/app/certs/rds-global-bundle.pem`. For another provider, mount its CA file and point this at it. |
| `PORT` | `4318` | Port the collector listens on. |
| `HOST` | `0.0.0.0` | Address the collector listens on. |
| `THEIRSIDE_PUBLIC_URL` | – | Where people open the dashboard. Used for links in alerts (none when unset) and the session cookie’s `Secure` flag. |
| `LOG_LEVEL` | `info` | `debug`, `info`, `warn` or `error`. |

### First run

| Variable | Default | What it does |
|---|---|---|
| `THEIRSIDE_BOOTSTRAP_API_KEY` | – | Project API key created on the first start (`tsd_live_` + at least 16 characters). With neither this nor `THEIRSIDE_ADMIN_TOKEN`, the first start generates a key and logs a one-time setup link. |
| `THEIRSIDE_ADMIN_TOKEN` | – | Completes the setup page and is the break-glass sign-in (also `Authorization: Bearer` for scripts). Unset it to turn it off. |
| `THEIRSIDE_BOOTSTRAP_ORG_NAME` | `Default org` | Organisation name created on the first start. |
| `THEIRSIDE_BOOTSTRAP_PROJECT_NAME` | `Default project` | Project name created on the first start. |

### Sign-in and security

| Variable | Default | What it does |
|---|---|---|
| `THEIRSIDE_SECRET_KEY` | – | Encrypts two-factor secrets; required for two-factor sign-in. At least 32 characters (`openssl rand -base64 32`). Back it up. |
| `THEIRSIDE_SECRET_KEY_PREVIOUS` | – | Old keys (comma-separated) still accepted while you rotate `THEIRSIDE_SECRET_KEY`. |
| `THEIRSIDE_TRUST_PROXY` | private ranges | Leave unset unless your load balancer reaches the collector from a public address, or you want to trust only its subnets. Default: `X-Forwarded-*` from proxies on private and loopback addresses is believed. Otherwise your load balancers’ CIDRs, a hop count, `false`, or `true` (any; logged as a warning). |
| `THEIRSIDE_SESSION_MAX_AGE_HOURS` | `168` | Dashboard session lifetime. |
| `THEIRSIDE_SESSION_IDLE_HOURS` | `24` | Dashboard session idle timeout. |
| `THEIRSIDE_AUTH_PROXY` | `off` | `header` (oauth2-proxy, Pomerium, Cloudflare Access…) or `alb` (AWS ALB with OIDC): sign people in from the identity proxy in front. |
| `THEIRSIDE_AUTH_PROXY_TRUSTED_IPS` | – | Required with `THEIRSIDE_AUTH_PROXY`: the proxy’s addresses or CIDRs, comma-separated. |
| `THEIRSIDE_AUTH_PROXY_HEADER` | `x-forwarded-email` | Header carrying the signed-in email (`header` mode). |
| `THEIRSIDE_AUTH_PROXY_ALB_ARN` | – | Required in `alb` mode: the load balancer’s ARN, checked against the signed header. |
| `THEIRSIDE_AUTH_PROXY_AUTO_CREATE` | `false` | Create an account (member) for a proxy user who has none. |
| `THEIRSIDE_ADMIN_EMAILS` | – | Comma-separated emails that become admins when the proxy creates their account. |

### Email

| Variable | Default | What it does |
|---|---|---|
| `THEIRSIDE_SMTP_URL` | – | Email alerts, invites and password resets: `smtp://user:pass@host:587` (STARTTLS) or `smtps://user:pass@host:465` (TLS). URL-encode special characters in the password. |
| `THEIRSIDE_SMTP_HOST` | – | Instead of the URL: the SMTP server… |
| `THEIRSIDE_SMTP_PORT` | `587` (`465` with `SECURE`) | …its port… |
| `THEIRSIDE_SMTP_SECURE` | `false` | …TLS from the start (port 465) instead of STARTTLS… |
| `THEIRSIDE_SMTP_USER` | – | …user… |
| `THEIRSIDE_SMTP_PASS` | – | …and password. |
| `THEIRSIDE_SMTP_REQUIRE_TLS` | `true` with credentials | Refuse to send credentials over plain text. `false` allows it. |
| `THEIRSIDE_SMTP_FROM` | `theirside@localhost` | Sender address; most providers only accept a verified sender or domain. |

### Data

| Variable | Default | What it does |
|---|---|---|
| `THEIRSIDE_RAW_RETENTION_DAYS` | `30` | Days raw call events are kept (daily partitions). |
| `THEIRSIDE_ROLLUP_RETENTION_DAYS` | `90` | Days per-minute rollups (charts, baselines, the status page) are kept. |
| `THEIRSIDE_PARTITIONS_AHEAD_DAYS` | `3` | Daily event partitions created ahead of time. |

### Advanced

| Variable | Default | What it does |
|---|---|---|
| `THEIRSIDE_PUBLIC_PATHS_ONLY` | `false` | `true` for a public status-page instance: serves only `/status/*`, `/embed.js`, `/fonts/*` and health checks, and runs no jobs. |
| `THEIRSIDE_STATUS_CACHE_SECONDS` | `30` | How long status pages may be cached by browsers and CDNs. |
| `THEIRSIDE_SCHEDULER_ENABLED` | `true` | Run the incident, alert and maintenance jobs here. With several instances one is elected; `false` keeps an instance ingest-only. |
| `THEIRSIDE_CATALOG_OVERRIDE` | – | Path to a JSON file with extra provider catalog entries (`{ "providers": [{ "name", "category", "domains": [] }] }`), e.g. to name an internal partner API. |
| `THEIRSIDE_LICENSE_KEY` | – | Unlocks paid features. Never needed for ingest, detection or alerts. |
| `THEIRSIDE_DESIGN_GALLERY` | `false` | Opens the dashboard’s `#/design` page (design tokens, no data) without sign-in. Leave off in production. |

<!-- https://theirside.dev/docs/privacy -->

# Privacy

The SDK runs inside your app and sends only what's needed to tell whether a provider is healthy. Everything goes to your own collector; nothing is sent anywhere else.

## What is recorded

For each outgoing call to a third-party host, one small event:

- the host and HTTP method;
- the **endpoint template**, built inside your app before anything is queued (`/v1/customers/cus_8x2K/cards` becomes `/v1/customers/:id/cards`);
- the status code, duration and outcome (ok, 5xx, timeout, connection error…);
- optionally the value of a provider's request-ID response header (`x-request-id` and similar, for support tickets) and a correlation ID your code supplies;
- the reason label, when your code marks an answer as expected (`consent_required`).

For each webhook delivery your app receives from a recognised sender or a declared route: the sender, the route template, the status your app answered, how long it took, and the sender's signed send time when there is one. For a [discovered route](https://theirside.dev/docs/webhooks#senders-the-sdk-doesnt-know-discovery): the route template, a suggested sender name and a delivery count.

## What is never captured

- request or response **bodies**;
- **query strings** and fragments;
- **headers**: no `Authorization`, cookies or API keys; response headers only for the request-ID value above. For webhooks the SDK looks at header *names*, and reads only the sender's timestamp and, for discovery, the `Host`, `Content-Type` and CloudEvents source and time values;
- raw paths, error messages, stack traces, client IP addresses;
- anything about calls to your own infrastructure: loopback, private networks, cloud metadata, internal names (`redis`, `*.svc`, `*.internal`), telemetry endpoints, hosts you list in `internalHosts` or hide in the dashboard.

## Paths without personal data

Paths are where personal data hides (`/v2/people/jane-doe`). By default (`pathPrivacy: 'strict'`):

- IDs become `:id`: numbers, UUIDs, prefixed IDs (`cus_123`), card numbers, emails, long tokens;
- a plain word is kept only if it's a known API word (a built-in vocabulary of words real APIs use, with personal names removed); any other word becomes `:id`;
- a position that sees more than 16 different values becomes `:id`.

So `/v2/people/jane-doe` is sent as `/v2/people/:id`. When a provider's own resource names show as `:id`, add them as that provider's allowed words in the dashboard, or give the provider explicit `routes` in `init()`. The one deliberate exception: a discovered webhook route keeps one plain lower-case word that names the sender (`/v1/payco`), at most 16 such words per process.

## Where data lives

In your collector's Postgres. Raw events are kept 30 days and per-minute rollups 90 days by default (`THEIRSIDE_RAW_RETENTION_DAYS`, `THEIRSIDE_ROLLUP_RETENTION_DAYS`). Alerts go only to the channels you configure.

The full, field-by-field description is `docs/privacy.md` in the Theirside repository, kept in step with the code.

<!-- https://theirside.dev/docs/troubleshooting -->

# Troubleshooting

Two tools answer most questions: `getStats()` in your app, and the SDK's debug output (`debug: true` for SDK problems, `debug: 'inbound'` for webhooks).

```ts
import { getStats } from '@theirside/sdk';

const stats = getStats();
if (!stats) console.log('the SDK is off: no apiKey or endpoint, or enabled: false');
else console.log(stats.sent, stats.queued, stats.dropped, stats.lastFlushError);
```

## A provider doesn't show up

1. **Is the SDK on?** `getStats()` returns `undefined` when `apiKey` or `endpoint` is missing: check `THEIRSIDE_API_KEY` and `THEIRSIDE_COLLECTOR_URL` reach the process.
2. **Does it send?** `sent` should grow after the app has made a few calls, and `lastFlushError` stay undefined. If not, see [The collector is unreachable](#the-collector-is-unreachable).
3. **Is `init()` early enough?** In ES modules it must run in a file imported before the rest of your app ([Node servers](https://theirside.dev/docs/node-servers#where-init-goes)).
4. **Is the host your own?** Loopback, private networks, internal names (`*.internal`, `*.svc`, single-label hosts) and telemetry endpoints are never recorded. A partner API on a private address needs `providers: [{ name, hosts }]`.
5. **Hidden?** Check **Settings → Providers** for a hidden provider or host.
6. **Right project and environment?** The API key decides the project; the environment picker at the top of the dashboard filters by `environment`.

Turn on `debug: true` to see bad options, flush failures and template overflows in your app's console. **Settings → SDK health** shows each service and process the collector has heard from.

## A webhook isn't discovered

1. Start the receiving service with `THEIRSIDE_DEBUG=inbound` (or `debug: 'inbound'`) and send one delivery. The line for it says what happened: `recorded as a delivery from …`, `candidate …`, or `ignored, …` with the reason.
2. **No line at all?** The request didn't reach a process with the SDK, or `init()` ran after the server was created. On AWS Lambda the handler must be wrapped with `wrap()` ([Serverless](https://theirside.dev/docs/serverless#webhooks-on-lambda)); other function platforms and Workers don't record incoming requests.
3. **`ignored, no webhook signal`**: the request has no route word, signature header, CloudEvents headers or webhook host. Declare the route: `webhooks: [{ path, provider }]` ([Webhooks](https://theirside.dev/docs/webhooks#declaring-routes)).
4. **`ignored, browser or front-end request`**: it carries cookies or `Origin`; real webhooks don't.
5. **Recorded, but not in the dashboard?** The flush line says whether it was sent (`… to /v1/ingest: 202`), `held` (waiting for the collector to confirm it stores webhooks) or `not sent: timeout`. On Lambda, add `inbound: true`. `getStats().inbound` shows `held`, `sent` and `dropped`.
6. **Ignored in the dashboard earlier?** Restore it under **Settings → Providers → Discovered webhook routes**.

## Alerts are too noisy

- **The first day**: each endpoint learns its normal for 24 hours and uses cautious defaults meanwhile. Webhook silence alerts need three weeks of history.
- **An error status that is a normal answer** (a `400` for "no consent yet"): mark it as expected, in code or from the incident ([Expected outcomes](https://theirside.dev/docs/expected-outcomes)).
- **A provider you don't need alerts for**: turn its alerts off on its page or in **Settings → Providers**; it stays monitored on the dashboard.
- **Too sensitive in general, or for one provider**: **Settings → Thresholds**, or the provider's settings, adjust the thresholds.
- **Test or staging traffic mixed in**: give it its own `environment` so it learns separately.

## The collector is unreachable

Your app is never affected: the SDK keeps events in memory up to `maxQueue` (10,000), drops the oldest beyond that and counts them, and retries with backoff. `getStats().lastFlushError` says why sending fails:

| `lastFlushError` | Meaning |
|---|---|
| `ECONNREFUSED`, `ENOTFOUND`, `ETIMEDOUT` | Wrong `THEIRSIDE_COLLECTOR_URL`, or the network doesn't reach it (security group, DNS, VPN) |
| `http_401` | The API key is wrong or revoked: **Settings → API keys** |
| `http_413` | A batch is too large: lower `maxBatch` |
| `http_429` | The project is over its plan's limit |
| `http_503` | The collector can't reach its database |

On the collector side: `GET /readyz` answers 200 when it is ready and 503 while its database is unreachable; its log says why.

<!-- https://theirside.dev/docs/ai-setup -->

# Set up with an AI coding agent

**Before you start:** a collector must be running, from the [local Quick start](https://theirside.dev/docs/getting-started#1-run-the-collector) or [Prompt 2](#prompt-2-add-the-collector-to-your-infrastructure) below. The API key and collector URL come from the dashboard's **Connect your app** screen, shown after you create the first account.

## Prompt 1: add the SDK to your app

Paste this into your coding agent (Claude Code, Cursor, Copilot…) at the root of your Node project. It adds the SDK in the right place for your stack and changes nothing else. Review the diff as you would any change.

```text
Add the Theirside SDK (@theirside/sdk) to this project. Change only what these steps need.

1. Detect the stack. Find the deployable entry points: a long-running Node HTTP server
   (Express, Fastify, NestJS, Koa, plain http) and/or AWS Lambda handlers. Note whether
   the code is ES modules or CommonJS, and which package manager the repo uses.
2. Install @theirside/sdk with that package manager, as a runtime dependency.
3. Check how the project loads its configuration, then call init() once per process.
   init() uses the values it gets when it runs, so pick the pattern that fits:
   - Settings in process.env at startup (platform env vars, dotenv loaded first):
       init({ apiKey: process.env.THEIRSIDE_API_KEY, endpoint: process.env.THEIRSIDE_COLLECTOR_URL,
              service: '<a short name for this deployable>' })
     before the app loads anything else. ES modules: in its own file (e.g. instrumentation.ts)
     imported first by the entry point; CommonJS: at the very top of the entry file; NestJS:
     the first import in main.ts; Lambda: at module top level, outside the handler.
   - Settings loaded at runtime (SSM, Secrets Manager, Vault, a config service): call init()
     with the loaded values right after the config is loaded and before HTTP clients are
     created. Lambda: at cold start, right after loading config (top-level await in ES
     modules; on the first invocation in CommonJS).
   - Never hard-code the API key or URL, and never commit their values.
4. Lambda handlers that receive webhooks over HTTP (API Gateway, ALB, function URL):
   wrap the exported handler with wrap() from @theirside/sdk, and add inbound: true to
   init(). Other Lambda handlers: wrap them with wrap() too, so events are sent before
   the function freezes.
5. Add the API key and collector URL where this project already declares its settings
   (.env.example, a config schema, the secrets the config loader reads, serverless.yml,
   template.yaml, Terraform/CDK, Helm values, docker-compose…), with placeholder values,
   following the existing pattern. Do not invent a new config mechanism.
6. Do not refactor, reformat, rename or upgrade anything else. Do not add tests unless
   the project's checks fail without them.
7. Run the project's type-check/build to confirm it still compiles.

Finish by printing: the files you changed and why (one line each), the env vars to set,
and how to verify: deploy or run the app with THEIRSIDE_API_KEY and
THEIRSIDE_COLLECTOR_URL set, make it call an external API, and its provider appears on
the Theirside dashboard's Providers page within a minute (getStats() from
@theirside/sdk returns undefined if the SDK is off).
```

After it's done: put the API key in your secrets store, set both variables in each environment, and check **Settings → SDK health** once the app runs. Details for each stack: [Node servers](https://theirside.dev/docs/node-servers), [Serverless](https://theirside.dev/docs/serverless).

## Prompt 2: add the collector to your infrastructure

Paste this at the root of the repository that describes your infrastructure. It adds the collector and its database the way that repository already deploys services, keeps it private, and changes nothing else. Review the plan or diff before applying it.

```text
Add the Theirside collector to this repository's infrastructure. Change only what these steps need.

1. Find how this repository already deploys services: docker-compose files, Terraform
   (ECS/Fargate), Kubernetes manifests or Helm charts, AWS CDK, or Pulumi. Follow that
   pattern and its conventions only. If there is none, stop and suggest the Docker Compose
   quick start instead: https://theirside.dev/docs/getting-started
2. Add one service running the image ghcr.io/theirside/collector:0.6 (it follows 0.6.x):
   - container port 4318; it runs as a non-root user (uid 1000); one instance, 1 vCPU, 2 GB;
   - health checks: GET /readyz for the load balancer or readiness (503 while its database
     is unreachable), GET /healthz for liveness or the container health check;
   - env vars: DATABASE_URL (required), THEIRSIDE_PUBLIC_URL (the dashboard's internal URL),
     THEIRSIDE_SECRET_KEY (32+ characters, for two-factor sign-in). Everything else is
     optional: https://theirside.dev/docs/configuration and /docs/self-hosting
3. Postgres 13+: if the repository already has a managed Postgres (RDS, Cloud SQL, Azure),
   add a new database and user on it, not a new server. Otherwise add Postgres the way the
   repository runs other stateful services. On RDS: ?sslmode=verify-full on DATABASE_URL
   and THEIRSIDE_DATABASE_CA_FILE=/app/certs/rds-global-bundle.pem (built into the image);
   another managed Postgres: mount its CA file and point that variable at it. Never use
   sslmode=no-verify outside local testing.
4. Keep it private: an internal load balancer, or no public ingress at all. Your apps reach
   it at its internal address. Do not add the optional public status-page instance (a second
   instance with THEIRSIDE_PUBLIC_PATHS_ONLY=true); only mention it in your summary.
5. Store DATABASE_URL and THEIRSIDE_SECRET_KEY the way this repository already stores secrets
   (SSM Parameter Store, Secrets Manager, Kubernetes Secrets, an uncommitted .env). Never
   write real values: placeholders and references only.
6. Change nothing else. Run the repository's validation if it has one (terraform validate,
   helm lint, kubectl apply --dry-run=client, docker compose config) and fix what it reports.

Finish by printing: the files you changed and why (one line each), the secrets to create and
how, and how to create the first account after deploying: on its first start the collector
logs one line containing "/setup" (docker compose logs collector, kubectl logs deploy/<name>,
or the service's log group; grep for /setup). By default it is a one-time link
(/setup?token=…): open it, create the account, copy the API key and URL from "Connect your
app" for Prompt 1. If THEIRSIDE_BOOTSTRAP_API_KEY is set (your own key from the secrets
store), it is an address ending in /#/setup plus a setup code valid 24 hours instead.
```

After it's applied: check that only your apps and your network can reach the collector, back up its database like any other, and keep `THEIRSIDE_SECRET_KEY` safe. Details: [Self-hosting](https://theirside.dev/docs/self-hosting), [Configuration](https://theirside.dev/docs/configuration).