What Leaves Your Process: The Life of a Monitoring Event

Here is a sentence that should make any engineer a little nervous: the SDK you generated and handed to strangers now reports its errors back to a server you do not run.

That is what Octri Monitoring is. It is also, if you squint, a small data pipeline that starts inside somebody else's production and ends on a disk in a data centre. So before you switch it on, you are entitled to some blunt answers. What exactly is in one of those events? Who can read it? How long does it sit there? And when a user of your app invokes their right to be forgotten, what happens, precisely, and how fast?

We spent the last stretch answering those questions in code rather than in a policy page, and then wrote the policy page to match. This post walks one event from the moment it is born to the moment it is gone. It is long, because the honest version is long.

The event, as it leaves

Picture a browser SDK we generated for your API. A call to POST /charges comes back with a 402 that your code did not expect. The SDK builds an event. Before we talk about what is in it, here is what is not, in any runtime, ever:

• The request body and the response body.
• Headers and cookies, including Authorization.
• The query string.
• The IP address of whoever sent it. Ingest reads it for rate limiting and drops it; it is not on the record.

Those are not filtered out. They are never collected in the first place, which is a much stronger property. There is no field for them.

What is in the event, by default: the error's name, message and stack; the operation that failed (method, templated path, status); the SDK's own version; and a random eventId. Everything that could describe a person is behind a gate.

The browser and Python SDKs carry a consent gate that is closed at rest. Until your code calls setLoggingConsent(true), an event is the error and the operation and nothing else. Open it and three more things travel: the user object you set (with its identifiers already redacted, more on that shortly), the device language, screen size and page title, and a breadcrumb trail of the last thirty SDK calls as method, path and status. The gate is a function rather than a banner because you already have a consent flow, and the point is to wire into it, not to add a second one.

Server-side and mobile runtimes have no browser session to attach context from, so they have no gate. They send a user only if your code set one.

If you would rather see this than read it, there is a "More about Telemetry" button on the monitoring page. It draws the flow and shows a sample event with the redactions highlighted.

Pass one: inside your process

The first scrubber runs in the SDK, on the payload, right before the HTTP request. This matters more than anything we do on our side, because by the time an event reaches us the sensitive values are already [REDACTED], and you can open the generated source and read the code that did it.

The old version of this was a fixed list of 69 keys. password, email, api_key, the usual suspects, compared after lower-casing and swapping dashes for underscores. It was fine until you looked at a real payload, which is full of keys like billingEmail, customer_phone_number and stripeSecretKey. None of those is on a list of whole words. A list of whole words is a list of the keys the author happened to think of.

So we changed how a key is read. Every key is normalised first: camel case is split, everything is lower-cased, and anything that is not a letter or a digit becomes an underscore. stripeSecretKey, API-Key and first name become stripe_secret_key, api_key and first_name. Then two checks run. Is the whole normalised key on the list? If not, does the key contain one of 43 terms as a run of whole words?

function normalizePiiKey(key: string): string {
  return key
    .replace(/([a-z0-9])([A-Z])/g, "$1_$2")
    .toLowerCase()
    .replace(/[^a-z0-9]/g, "_");
}

function isPiiKey(key: string): boolean {
  const normalized = normalizePiiKey(key);
  if (PII_KEYS.has(normalized)) return true;
  const padded = `_${normalized}_`;
  return PII_SEGMENTS.some((segment) => padded.includes(segment));
}

The padding is the whole trick. Each term is stored pre-padded, so email is really _email_, and _billing_email_ contains it while _emailed_at_ does not. Word runs, not substrings.

That distinction is why the list is split in two. Some words are safe to match anywhere: nobody has an innocent key containing passport or client_secret. Some are not. zip would take zipFile. url would take avatarUrl. query would take queryTimeMs, and then you would open an issue to find the one number you wanted replaced with a placeholder. Those short words match only as the whole key. The line between the two lists was drawn by hand, key by key, with a test that pins both sides: the keys that must be caught and the keys that must be left alone.

Then we did it ten times, because Octri generates SDKs in ten languages and every one of them has to make the same decision about the same key. The pattern above is easy in TypeScript and Python. It gets more interesting in Rust, where the regex you would reach for is not in the standard library, and in Swift, where the string API has opinions. The test suite feeds each runtime the same fixture and expects byte-identical output, which is how we found that one port had quietly collapsed a + and another had a $ in a Kotlin template that Kotlin read as interpolation. Both are fixed. Both are the kind of thing you only find by refusing to trust a port.

The standalone backend packages (@octri/node, octri on PyPI, octri-monitoring on crates.io, and their siblings) go one step further, because a server exception message is where a secret usually leaks. They also scan free text for bearer tokens, JSON Web Tokens, card numbers that pass a Luhn check, and email addresses, and mask them wherever they appear, stack frames included. The id you set on the user survives; its email, phone and names do not. Version 1.2.0 of each package adds the identifier keys to that list, and is rolling out to the registries now.

Those packages expose two hooks. addScrubFields("accountNumber", "otp") adds your domain's keys to the list. setBeforeSend(hook) runs your function on every payload; return null to drop the event. Redaction runs after your hook, so a hook cannot put a credential back, on purpose or by accident.

Pass two: at the door

Every payload that reaches ingest is scrubbed again, unconditionally, before anything is written. Not because we distrust the SDK, but because not every event comes from one. A generic HTTP client posting to the endpoint. A build from before client-side filtering existed. A consumer who set filterPii: false because they were pointing at a private endpoint and then pointed it at us.

The ingest pass matches keys the same way the SDKs do, against a superset of their list. It also scans values, and this is where it gets fun, because values are where the weird stuff lives.

A JWT is three base64url segments joined by dots and starting with eyJ. A Bearer token announces itself. AWS access keys start with AKIA and are exactly twenty characters. GitHub tokens are ghp_, gho_, ghu_, ghs_ or ghr_ and then a long tail. Stripe keys are sk_, rk_ or pk_ followed by live or test. Slack tokens are xox and a letter. Google API keys are AIza and 35 more characters. A private key is bracketed by -----BEGIN and -----END. Card numbers are 13 to 19 digits that pass a Luhn check, which rules out order numbers and most phone numbers.

Then there is the assignment pattern: password = "hunter2", apiKey: sk_live_..., client_secret => '...'. Somebody wrote that into an exception message once and it will happen again. The scrubber recognises a credential-shaped name, an assignment operator and a value, and replaces only the value. The first draft of that regex also matched tokensUsed = 1234, which is what a billing service logs about a language model, and we would have redacted a number that is not a secret in every event from that service. The fix is that the value has to be at least six characters and not purely digits, which is a fair description of a secret and an unfair description of a counter. dbPassword = ... still matches, because the name may have a prefix.

In every case only the match is replaced. The rest of the message survives, so Failed to charge card 4242 4242 4242 4242 for order 8813 becomes Failed to charge card [REDACTED] for order 8813, which is exactly the message you want to see grouped with its siblings.

Email addresses get different treatment: masked, not removed. jane@acme.com becomes j***@acme.com, anywhere it appears, in a field, a message or a request path. The shape survives so you can still recognise a user and so an issue can still be grouped per affected person, but the address is not on disk. When the pipeline computes an issue's grouping fingerprint it goes further and templates addresses to <email>, along with ids, numbers and quoted literals, so two users hitting the same bug land in the same issue rather than one each.

Transport

The wire is HTTPS only. The public ingest hostname sits behind Cloudflare's edge, which terminates TLS, runs a web application firewall and forwards to our server over TLS again. Cloudflare does not store the payload.

Every request carries a token our API mints when you enable monitoring. It is an HMAC-signed claim naming your project and a version number. The signing secret never leaves our API server, and the token cannot be used to read anything. The rule that makes tenancy work is small and absolute: ingest takes the project from the token, never from the payload. An SDK holding one project's token cannot write into another project no matter what it sends.

Enabling monitoring again increments the version, and every token carrying an older version is refused from that moment. That is how you revoke a credential you think has leaked: disable, enable, rebuild the SDK.

Delivery is fire-and-forget. The SDK never waits on us, so monitoring cannot slow your application or take it down with ours. Ingest answers 202 Accepted once the event is durably stored. The eventId is an idempotency key, so a retried delivery is discarded rather than counted twice. Errors are never sampled.

Where it lives

Monitoring has its own analytics database, ClickHouse, on a server that does nothing else. We rent the box from OVH in its United Kingdom region and operate everything on it ourselves: the database, the ingest service, the reverse proxy. There is no managed database vendor, no third-party observability product, and no analytics pipeline between your SDK and the disk.

ClickHouse listens on loopback only. It is not reachable from the internet and not reachable from our other servers. The dashboard gets to it through the monitoring service over a private WireGuard link from our API server, with a separate credential that ingest does not hold. The firewall is default-deny inbound.

Tenant isolation inside the database is a WHERE environment = ... clause, not a separate database per customer. That is normal for this kind of system and it is also the kind of invariant that erodes one route at a time. So there is a test that registers the real router on a bare server, walks every route it declares, and asserts that each dashboard route requires an environment in its params, query or body. Forty-four routes today. Exactly one is allowed to be environment-agnostic (the scheduler's alert evaluation pass, because each rule carries its own environment), and it is listed in the test with the reason, so adding another is a reviewed decision rather than an accident.

Telemetry is never used to train a model, never joined across customers, and never read for any purpose beyond operating the service. Our operator console does not display it. Access to the server is by SSH key held by named staff, and support access to your organisation in the dashboard is written to your own audit log under the staff member's address.

When it dies

Raw events, spans and metrics expire after the retention window your plan includes:

Plan	Raw event retention
Free	7 days
Starter	30 days
Growth	90 days
Business	180 days
Enterprise	365 days, or as agreed in your order form

The window is resolved by our API and stamped on each record as it is admitted. Expiry is enforced by the storage engine on that stamp, not by a sweep job. This is a deliberate choice. A cron job that deletes old data is a promise that depends on the cron job running; a TTL is a property of the table. Aggregate rollups (counts, error rates, latency percentiles per day) outlive the raw events so long-range charts keep working, and they carry no payload and no user identifier.

Issues are the interesting case, and they were the gap we found. An issue keeps its title, its templated path, its counts, first and last seen, and the stack of its most recent occurrence for as long as the project exists. That is correct: an issue's count is not personal data and you want it to survive the raw events behind it. But the fact table that count is derived from also kept every user_id that had ever hit the issue, with no expiry, for as long as the issue lived. The count is not personal data. The identifier is. We were keeping identifiers longer than the retention window we had promised.

The fix is a column TTL on the identifier alone:

ALTER TABLE issue_occurrences
  ADD COLUMN IF NOT EXISTS expires_at Nullable(DateTime64(3, 'UTC'));

ALTER TABLE issue_occurrences
  UPDATE expires_at = timestamp + INTERVAL 365 DAY WHERE expires_at IS NULL;

ALTER TABLE issue_occurrences
  MODIFY COLUMN user_id Nullable(String)
  TTL toDateTime(
    ifNull(expires_at, toDateTime64('2299-12-31 00:00:00', 3, 'UTC'))
  );

Every new occurrence is stamped with the same expiry its raw event gets. When the stamp passes, the engine resets user_id to NULL on the next merge; the row, its timestamp and its count survive. The 2299-12-31 sentinel is there because a column TTL needs a date and NULL is not one; a row with no stamp, which after the backfill should not exist, would keep its identifier until the twenty-fourth century. Rows that predated the column were not left to live forever either: they were given the longest plan window from their own timestamp, so anything older lost its identifier on the first merge after the migration ran.

Two things back that up. A scheduled job runs the same reset as an explicit mutation every six hours, for parts that are never merged again, and rewrites each issue's affected-users projection to the identifiers still live. And every query that counts affected users carries a single predicate:

isNotNull(user_id)
  AND user_id != ''
  AND (expires_at IS NULL OR expires_at > now64(3))

So the "users affected" figure you see in the dashboard already excludes expired identifiers before the engine gets around to erasing them. An issue's users are the distinct users within your retention window. Its count is all-time.

When you ask us to forget someone

A data subject request reaches you, not us. You are the controller of what your application sends; we are the processor. For access requests, the log query in the dashboard filters events by the user id you attached, so you can answer one yourself without a ticket.

Erasure used to be an email. Now it is a button. An owner or admin opens Monitoring, then the connection settings, then "Erase a user's data", enters the id, and confirms. What happens next:

1. Our API calls the monitoring service, which answers 202 immediately and hands the work to a worker pool.
2. The worker deletes every event carrying that id, nulls the id on every occurrence record, and rewrites the affected-users list of every issue it touched. Counts are kept. The person is gone; the fact that a bug happened is not.
3. Either way, project.monitoring_user_erased lands in your organisation's audit log, with who asked and when.

The part we are most pleased with is what happens when step 1 fails. The monitoring service is on a different server over a private link, and links have bad days. Instead of surfacing an error and asking you to try again later, the API records the request in a pending-purges table, and an hourly job retries it until the service confirms. Project deletion works the same way: monitoring data goes with the project, and if the purge cannot be confirmed at that moment it is queued and retried rather than forgotten. A promise to delete that quietly depends on the network being up is not a promise.

If you would rather we did it, tell us the project and the identifier and we do the same within 30 days. Disabling monitoring stops ingestion at once but keeps what was stored until it expires, so you can turn reporting off during an incident without destroying the evidence.

On paper

Code is where privacy is decided, but the contract is what your DPO reads, so we wrote the contract to describe the code. There is now a Monitoring Data Processing Addendum: fifteen sections that walk the same path this post does, field by field, with the retention table, the deletion mechanics, the security measures and the transfer basis. It is a pre-signed Article 28 addendum to the main DPA. There is nothing to countersign.

The short version lives behind the "More about Telemetry" button on the landing and monitoring pages: one panel, the flow drawn left to right, a sample event with the redactions marked, and each GDPR obligation next to the mechanism that meets it. OVH joined the subprocessors page as the monitoring host, and Cloudflare's entry now says it carries ingest traffic.

For data subject to the EU GDPR, the United Kingdom is covered by an adequacy decision, so storage there needs no further transfer mechanism. Octri is operated from the United States, and access by our staff from there is covered by the Standard Contractual Clauses and the UK Addendum in the DPA.

What it still cannot do

Redaction is a safety net with a fixed list, and a fixed list has edges. It cannot know that a field you named notes holds a home address. The generated SDKs do not scan message text for patterns; only the backend packages and ingest do that. Tags and context you attach are stored as sent once their keys have been checked. And a consent gate only works if you open it for the people who said yes and leave it closed for everyone else.

Sending less is the only measure that removes a risk rather than reducing it. Prefer your own opaque user id to an email address; the id is what the affected-users count needs, and the address would be masked anyway. Read the generated source. It is your code now, and the scrubber is in it.

---

Every Octri plan includes monitoring, with the same redaction, the same expiry and the same erasure button. Generate a monitored SDK from your spec.