From 461b82d7372eac86e62cba6ebba60403f5d5a08b Mon Sep 17 00:00:00 2001 From: Karl Rankla Date: Thu, 10 Sep 2026 10:43:30 +0300 Subject: [PATCH] docs(integration-toolkit): document monitoring and alerting, fix stale facts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Monitoring was 998 of the section's ~34,600 words, and the single page it had documented the narrowest capability in the whole surface — how a middleware pushes its own spans in. The Hub's Monitoring and Notifications tabs, the code taxonomy, replay, retention and the entire alerting engine had no page at all. Corrections first, because these were actively costing people time: - Every curl example targeted erp-integration.sls.epilot.io, which does not resolve. 23 occurrences now use integration-toolkit.sls.epilot.io, matching the spec's own servers block. - configuration.md documented POST /v1/monitoring/events and /v1/monitoring/stats. Neither has ever existed in any version of the spec. Replaced with a short stub carrying one correct v2 example and a pointer to the new section. - The permissions table listed erp:read / erp:write / erp:events / erp:monitoring, none of which appear anywhere in the codebase. Replaced with the real actions — integration:view, integration:manage, integration:consume — plus a per-endpoint table, because the view-vs-manage split is what bites when minting a scoped token, and consume is poll-queue only. - The ACK example sent a `status` field that is not in the contract, and sourced ack_id from a webhook header. ack_id is the only field, and it arrives as the `_ack_id` payload field on every core event. - The component table called Monitoring "In progress". It is deployed, and the Notifications tab ships unflagged. Split into Monitoring, Alerting & Notifications and ACK Tracking, all Stable; Integration Hub and Pollable Outbound re-audited to match. New monitoring/ section, six pages, written UI-first with the API alongside because the people who live in this are operators rather than middleware devs: - overview — levels (including why info is excluded from the success rate), codes, the event_id vs correlation_id distinction, use case lanes, "General"/__unknown__, retention, and entity sync-status - codes — all 80 codes, GENERATED, see below - investigating — traces, what body capture stores and elides, replay, querying, and a "which number am I looking at" section for the monitoring-vs-incoming source split that silently under-reports past 14 days - alerting — the six rule types, auto thresholds, scoping, recipients and channels, digests, muting, and the notification history - acks — the lifecycle, the timeout, and per-use-case ack_tracking - external-events — the existing page, moved in. Its slug is unchanged, so the published URL does not break. codes.md is generated by scripts/update-monitoring-codes.js from a snapshot erp-integration-api emits and asserts in CI, following the same pattern this repo already uses for entity-api schemas and event-catalog-api events. The descriptions used to live only in the integration hub, where a backend taxonomy was owned by a frontend; the producer owns them now and this page is a consumer, so the published reference cannot drift from what the code actually emits. Retention figures (90 days monitoring, 14 days received payloads) are stated because the 14-day one is a hard ceiling on replay that operators otherwise discover by failing. Body-capture redaction is documented as a guarantee rather than a match list, with a pointer to the secure proxy for custom auth headers. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- .../integration-toolkit/configuration.md | 99 ++++---- .../integration-toolkit/inbound/examples.md | 4 +- .../inbound/getting-started.md | 6 +- .../monitoring/_category_.json | 4 + .../integration-toolkit/monitoring/acks.md | 135 +++++++++++ .../monitoring/alerting.md | 229 ++++++++++++++++++ .../integration-toolkit/monitoring/codes.md | 139 +++++++++++ .../external-events.md} | 4 +- .../monitoring/investigating.md | 181 ++++++++++++++ .../monitoring/overview.md | 168 +++++++++++++ .../outbound-file-delivery.md | 2 +- .../integration-toolkit/overview.md | 35 ++- .../integration-toolkit/pollable-outbound.md | 2 +- .../integration-toolkit/use-cases.md | 20 +- package.json | 1 + scripts/update-monitoring-codes.js | 128 ++++++++++ 16 files changed, 1076 insertions(+), 81 deletions(-) create mode 100644 docs/integrations/integration-toolkit/monitoring/_category_.json create mode 100644 docs/integrations/integration-toolkit/monitoring/acks.md create mode 100644 docs/integrations/integration-toolkit/monitoring/alerting.md create mode 100644 docs/integrations/integration-toolkit/monitoring/codes.md rename docs/integrations/integration-toolkit/{external-monitoring-events.md => monitoring/external-events.md} (99%) create mode 100644 docs/integrations/integration-toolkit/monitoring/investigating.md create mode 100644 docs/integrations/integration-toolkit/monitoring/overview.md create mode 100644 scripts/update-monitoring-codes.js diff --git a/docs/integrations/integration-toolkit/configuration.md b/docs/integrations/integration-toolkit/configuration.md index d85a97f4..7f623e60 100644 --- a/docs/integrations/integration-toolkit/configuration.md +++ b/docs/integrations/integration-toolkit/configuration.md @@ -22,7 +22,7 @@ Integrations support two types: ### Creating an Integration ```bash -curl -X POST 'https://erp-integration.sls.epilot.io/v2/integrations' \ +curl -X POST 'https://integration-toolkit.sls.epilot.io/v2/integrations' \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -d '{ @@ -34,7 +34,7 @@ curl -X POST 'https://erp-integration.sls.epilot.io/v2/integrations' \ **Creating a Connector Integration:** ```bash -curl -X POST 'https://erp-integration.sls.epilot.io/v2/integrations' \ +curl -X POST 'https://integration-toolkit.sls.epilot.io/v2/integrations' \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -d '{ @@ -68,14 +68,14 @@ curl -X POST 'https://erp-integration.sls.epilot.io/v2/integrations' \ ### Listing Integrations ```bash -curl -X GET 'https://erp-integration.sls.epilot.io/v2/integrations' \ +curl -X GET 'https://integration-toolkit.sls.epilot.io/v2/integrations' \ -H 'Authorization: Bearer ' ``` ### Updating an Integration ```bash -curl -X PUT 'https://erp-integration.sls.epilot.io/v2/integrations/{integrationId}' \ +curl -X PUT 'https://integration-toolkit.sls.epilot.io/v2/integrations/{integrationId}' \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -d '{ @@ -91,7 +91,7 @@ Deleting an integration also removes all associated use cases. ::: ```bash -curl -X DELETE 'https://erp-integration.sls.epilot.io/v2/integrations/{integrationId}' \ +curl -X DELETE 'https://integration-toolkit.sls.epilot.io/v2/integrations/{integrationId}' \ -H 'Authorization: Bearer ' ``` @@ -100,7 +100,7 @@ curl -X DELETE 'https://erp-integration.sls.epilot.io/v2/integrations/{integrati ### Creating a Use Case ```bash -curl -X POST 'https://erp-integration.sls.epilot.io/v1/integrations/{integrationId}/use-cases' \ +curl -X POST 'https://integration-toolkit.sls.epilot.io/v1/integrations/{integrationId}/use-cases' \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -d '{ @@ -132,7 +132,7 @@ curl -X POST 'https://erp-integration.sls.epilot.io/v1/integrations/{integration ### Enabling/Disabling a Use Case ```bash -curl -X PUT 'https://erp-integration.sls.epilot.io/v1/integrations/{integrationId}/use-cases/{useCaseId}' \ +curl -X PUT 'https://integration-toolkit.sls.epilot.io/v1/integrations/{integrationId}/use-cases/{useCaseId}' \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -d '{ @@ -151,7 +151,7 @@ incoming events. Plan cutovers accordingly. View the change history for a use case: ```bash -curl -X GET 'https://erp-integration.sls.epilot.io/v1/integrations/{integrationId}/use-cases/{useCaseId}/history' \ +curl -X GET 'https://integration-toolkit.sls.epilot.io/v1/integrations/{integrationId}/use-cases/{useCaseId}/history' \ -H 'Authorization: Bearer ' ``` @@ -160,7 +160,7 @@ curl -X GET 'https://erp-integration.sls.epilot.io/v1/integrations/{integrationI Outbound use cases deliver standardized epilot events (event-catalog events) to your external system. The configuration consists of an `event_catalog_event` and one or more `mappings`: ```bash -curl -X POST 'https://erp-integration.sls.epilot.io/v1/integrations/{integrationId}/use-cases' \ +curl -X POST 'https://integration-toolkit.sls.epilot.io/v1/integrations/{integrationId}/use-cases' \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -d '{ @@ -350,7 +350,7 @@ Managed call use cases define synchronous API operations against external partne ### Creating a Managed Call Use Case ```bash -curl -X POST 'https://erp-integration.sls.epilot.io/v1/integrations/{integrationId}/use-cases' \ +curl -X POST 'https://integration-toolkit.sls.epilot.io/v1/integrations/{integrationId}/use-cases' \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -d '{ @@ -377,7 +377,7 @@ curl -X POST 'https://erp-integration.sls.epilot.io/v1/integrations/{integration Managed calls are executed via the `/v1/managed-call/{slug}/execute` endpoint, where the slug acts as the RPC method name: ```bash -curl -X POST 'https://erp-integration.sls.epilot.io/v1/managed-call/get-customer/execute' \ +curl -X POST 'https://integration-toolkit.sls.epilot.io/v1/managed-call/get-customer/execute' \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -d '{ @@ -433,7 +433,7 @@ Secure proxy use cases route HTTP requests through epilot's dedicated proxy infr ### Creating a Secure Proxy Use Case ```bash -curl -X POST 'https://erp-integration.sls.epilot.io/v1/integrations/{integrationId}/use-cases' \ +curl -X POST 'https://integration-toolkit.sls.epilot.io/v1/integrations/{integrationId}/use-cases' \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -d '{ @@ -458,7 +458,7 @@ curl -X POST 'https://erp-integration.sls.epilot.io/v1/integrations/{integration ### Sending a Proxy Request ```bash -curl -X POST 'https://erp-integration.sls.epilot.io/v1/secure-proxy' \ +curl -X POST 'https://integration-toolkit.sls.epilot.io/v1/secure-proxy' \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -d '{ @@ -602,48 +602,22 @@ Include a correlation ID to track related events: - Filter monitoring data by batch - Trace event processing through the pipeline -## Monitoring Configuration +## Monitoring -### Query Events +Every inbound and outbound event the toolkit processes is recorded as a monitoring +event, queryable per integration and visible in the Integration Hub's **Monitoring** +tab. A one-off example — the last day's error events for one integration: ```bash -curl -X POST 'https://erp-integration.sls.epilot.io/v1/monitoring/events' \ +curl -X POST 'https://integration-toolkit.sls.epilot.io/v2/integrations/{integrationId}/monitoring/events' \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ - -d '{ - "integration_id": "", - "from": "2024-01-15T00:00:00Z", - "to": "2024-01-15T23:59:59Z", - "status": ["error"], - "limit": 100 - }' + -d '{ "level": "error", "from_date": "2026-01-15T00:00:00Z", "limit": 50 }' ``` -### Query Statistics +Querying, stats and time series, event traces, replay, the code reference, and +alerting are documented in the [Monitoring section](./monitoring/overview.md). -```bash -curl -X POST 'https://erp-integration.sls.epilot.io/v1/monitoring/stats' \ - -H 'Authorization: Bearer ' \ - -H 'Content-Type: application/json' \ - -d '{ - "integration_id": "", - "from": "2024-01-01T00:00:00Z", - "to": "2024-01-31T23:59:59Z" - }' -``` - -### Event Replay - -Reprocess failed or specific events: - -```bash -curl -X POST 'https://erp-integration.sls.epilot.io/v1/integrations/{integrationId}/events/replay' \ - -H 'Authorization: Bearer ' \ - -H 'Content-Type: application/json' \ - -d '{ - "event_ids": ["evt-001", "evt-002", "evt-003"] - }' -``` ## Rate Limits @@ -668,9 +642,28 @@ Each organization's data is fully isolated. Integration IDs are scoped to organi ### Permissions -| Action | Permission | -|--------|------------| -| Read integrations | `erp:read` | -| Create/Update integrations | `erp:write` | -| Send events | `erp:events` | -| View monitoring | `erp:monitoring` | +Three actions govern the Integration Toolkit. They are checked against the +organization the token belongs to, and — where the endpoint names one — against the +specific integration. + +| Permission | Grants | +|---|---| +| `integration:view` | Read integrations and use cases; read monitoring events, stats, time series, traces, access logs, notification history and status | +| `integration:manage` | Create, update and delete integrations and use cases; edit notification configuration; push external monitoring events; send test notifications; replay events | +| `integration:consume` | Poll and acknowledge the outbound queue. Scoped to [Pollable Outbound](./pollable-outbound.md) only — it grants nothing else | + +The `view` / `manage` split is the one that catches people out when minting a scoped +token for a middleware: + +| Endpoint | Needs | +|---|---| +| `POST /v2/integrations/{id}/monitoring/events`, `…/stats`, `…/time-series` | `integration:view` | +| `GET /v2/integrations/{id}/monitoring/traces/{correlationId}` | `integration:view` | +| `GET /v2/integrations/{id}/notifications/history`, `…/status` | `integration:view` | +| `POST /v2/integrations/{id}/monitoring/external-events` | `integration:manage` | +| `POST /v2/integrations/{id}/notifications/test` | `integration:manage` | +| `POST /v1/integrations/{id}/events/replay` | `integration:manage` | +| `POST /v1/integrations/{id}/outbound/messages/poll`, `…/ack` | `integration:consume` | + +Inbound event submission authenticates as the integration's own API token rather +than through these actions — see [Inbound Getting Started](./inbound/getting-started.md). diff --git a/docs/integrations/integration-toolkit/inbound/examples.md b/docs/integrations/integration-toolkit/inbound/examples.md index ee679c91..a7b63ec7 100644 --- a/docs/integrations/integration-toolkit/inbound/examples.md +++ b/docs/integrations/integration-toolkit/inbound/examples.md @@ -93,7 +93,7 @@ Synchronize customer data from an ERP system into epilot contacts. ### API Request ```bash -curl -X POST 'https://erp-integration.sls.epilot.io/v3/erp/updates/events' \ +curl -X POST 'https://integration-toolkit.sls.epilot.io/v3/erp/updates/events' \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -d '{ @@ -480,7 +480,7 @@ Handle different entity types based on payload conditions. Before deploying your mapping, test it using the simulation endpoint: ```bash -curl -X POST 'https://erp-integration.sls.epilot.io/v2/erp/updates/mapping_simulation' \ +curl -X POST 'https://integration-toolkit.sls.epilot.io/v2/erp/updates/mapping_simulation' \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -d '{ diff --git a/docs/integrations/integration-toolkit/inbound/getting-started.md b/docs/integrations/integration-toolkit/inbound/getting-started.md index f99d2caa..efa8bef3 100644 --- a/docs/integrations/integration-toolkit/inbound/getting-started.md +++ b/docs/integrations/integration-toolkit/inbound/getting-started.md @@ -19,7 +19,7 @@ This guide walks you through setting up an inbound integration to synchronize da Create a new integration to represent your ERP connection: ```bash title="Create an integration" -curl -X POST 'https://erp-integration.sls.epilot.io/v2/integrations' \ +curl -X POST 'https://integration-toolkit.sls.epilot.io/v2/integrations' \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -d '{ @@ -49,7 +49,7 @@ Save the `id` — you'll need it for subsequent API calls. A use case defines how specific data types are mapped and synchronized. Create an inbound use case for customer data: ```bash title="Create a use case" -curl -X POST 'https://erp-integration.sls.epilot.io/v1/integrations/{integrationId}/use-cases' \ +curl -X POST 'https://integration-toolkit.sls.epilot.io/v1/integrations/{integrationId}/use-cases' \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -d '{ @@ -78,7 +78,7 @@ curl -X POST 'https://erp-integration.sls.epilot.io/v1/integrations/{integration Push data from your ERP system using the events endpoint: ```bash title="Send an inbound event" -curl -X POST 'https://erp-integration.sls.epilot.io/v3/erp/updates/events' \ +curl -X POST 'https://integration-toolkit.sls.epilot.io/v3/erp/updates/events' \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -d '{ diff --git a/docs/integrations/integration-toolkit/monitoring/_category_.json b/docs/integrations/integration-toolkit/monitoring/_category_.json new file mode 100644 index 00000000..37aa3068 --- /dev/null +++ b/docs/integrations/integration-toolkit/monitoring/_category_.json @@ -0,0 +1,4 @@ +{ + "label": "Monitoring & Alerting", + "position": 9 +} diff --git a/docs/integrations/integration-toolkit/monitoring/acks.md b/docs/integrations/integration-toolkit/monitoring/acks.md new file mode 100644 index 00000000..470b8e56 --- /dev/null +++ b/docs/integrations/integration-toolkit/monitoring/acks.md @@ -0,0 +1,135 @@ +--- +sidebar_position: 5 +title: ACK Tracking +description: How ERPs acknowledge events they processed, the ACK lifecycle and timeout, and how to turn tracking off per use case +slug: /integrations/integration-toolkit/monitoring/acks +--- + +# ACK Tracking + +epilot knows it *delivered* an outbound event — the webhook returned 2xx, or the +message was polled. It does not know the ERP actually **processed** it. An +acknowledgement closes that gap: your middleware confirms the work is done, and the +event's status in the Integration Hub becomes end-to-end rather than delivery-only. + +## The flow + +```mermaid +sequenceDiagram + participant EP as epilot + participant MW as Your middleware + + EP->>MW: outbound event (payload carries _ack_id) + Note over EP: ACK_PENDING recorded + MW->>MW: process the event + MW->>EP: POST /v1/erp/tracking/acknowledgement { ack_id } + Note over EP: ACK_CONFIRMED — tracking record cleared +``` + +If the acknowledgement never arrives, the record goes stale and epilot records +`ACK_TIMEOUT` instead. + +## Where `ack_id` comes from + +It arrives **in the event payload**, as the `_ack_id` field, alongside the other +common metadata every core event carries (`_event_version`, `_event_source`, …). It is +not an HTTP header, and it is not something you construct. + +```jsonc +{ + "_event_source": "epilot", + "_ack_id": "ack_01HZY…", // ← acknowledge with this + "contract": { "...": "..." } +} +``` + +Each delivered event gets its own `_ack_id`. Acknowledge each one individually — an +id is consumed once, and acknowledging clears its tracking record. + +:::note +A JSONata outbound mapping controls what your ERP receives. If your mapping builds a +brand-new object rather than extending the event, carry `_ack_id` through explicitly, +or your middleware will never see the id it needs to acknowledge with. +::: + +## Sending the acknowledgement + +```bash title="Send ACK" +curl -X POST 'https://integration-toolkit.sls.epilot.io/v1/erp/tracking/acknowledgement' \ + -H 'Content-Type: application/json' \ + -d '{ "ack_id": "ack_01HZY…" }' +``` + +`ack_id` is the **only** field. There is no status to report: sending the +acknowledgement *is* the signal that processing succeeded. A failure is simply an ACK +that never arrives, which the timeout below turns into a visible warning. + +| Response | Meaning | +|---|---| +| `200` | Acknowledged; the tracking record is cleared | +| `400` | `ack_id` missing from the body | +| `404` | No tracking record — already acknowledged, already timed out, or an unknown id | + +A `404` after a successful `200` is normal if you retry: the record is gone because +the first call consumed it. Treat it as success. + +## The lifecycle and its codes + +Three [monitoring codes](./codes.md) tell the whole story, and all three are +filterable in the Monitoring tab: + +| Code | Level | When | +|---|---|---| +| `ACK_PENDING` | info | The event was delivered and epilot is waiting for the acknowledgement | +| `ACK_CONFIRMED` | info | Your acknowledgement arrived | +| `ACK_TIMEOUT` | warning | No acknowledgement within the timeout window | + +`ACK_PENDING` and `ACK_CONFIRMED` are **info**-level: they are lifecycle markers, not +outcomes, so they are counted in total events but deliberately excluded from the +success rate. `ACK_TIMEOUT` is a **warning** — the delivery itself worked, so it is +not an error on epilot's side, but something on yours needs attention. + +### The timeout window + +A checker runs every **10 minutes** and times out any record older than **15 +minutes**. In practice an unacknowledged event surfaces as `ACK_TIMEOUT` within about +25 minutes of delivery — so do not treat a missing ACK as final for roughly half an +hour. + +`ACK_TIMEOUT` is also promoted to its own figure in the stats response, +`ack_timeout_count`, so you can chart "how often is the ERP failing to confirm" without +filtering the event stream by code: + +```bash +curl -X POST 'https://integration-toolkit.sls.epilot.io/v2/integrations/{integrationId}/monitoring/stats' \ + -H 'Authorization: Bearer ' \ + -H 'Content-Type: application/json' \ + -d '{ "from_date": "2026-01-01T00:00:00Z", "use_case_type": "outbound" }' +``` + +## Turning tracking off + +ACK tracking is per outbound use case, via `ack_tracking`: + +| Value | Behaviour | +|---|---| +| `on` *(default)* | A tracking record is written, `ACK_PENDING` is recorded, and an unacknowledged event eventually raises `ACK_TIMEOUT` | +| `off` | No tracking record, no `ACK_PENDING`, and no `ACK_TIMEOUT` | + +Set it to `off` for consumers that will never acknowledge, and for deliveries that +already keep their own durable per-item record — [Pollable +Outbound](../pollable-outbound.md) queues, for instance, track consumption through the +queue's own `MSG_ACKED` lifecycle, so ACK tracking on top of it produces timeouts that +mean nothing. + +:::warning +Leaving `ack_tracking: on` for a consumer that never acknowledges generates a steady +stream of `ACK_TIMEOUT` warnings. That is noise on its own, and it will trip a +`warning_threshold` [alert rule](./alerting.md) if you enable one. +::: + +## Related + +- [Monitoring Codes](./codes.md) — the full code reference +- [Investigating events](./investigating.md) — traces and replay +- [Pollable Outbound](../pollable-outbound.md) — the queue's own delivery lifecycle diff --git a/docs/integrations/integration-toolkit/monitoring/alerting.md b/docs/integrations/integration-toolkit/monitoring/alerting.md new file mode 100644 index 00000000..bdafad3f --- /dev/null +++ b/docs/integrations/integration-toolkit/monitoring/alerting.md @@ -0,0 +1,229 @@ +--- +sidebar_position: 4 +title: Alerting & Notifications +description: Configure per-integration alert rules, anomaly thresholds, digests and recipients so integration problems find you instead of the other way round +slug: /integrations/integration-toolkit/monitoring/alerting +--- + +# Alerting & Notifications + +Monitoring tells you what happened when you go and look. Alerting tells you without +looking. Each integration has its own notification configuration: who gets told, about +what, and how often. + +Open the integration in epilot 360 and use the **Notifications** tab. Everything below +is configurable there, and through the API on the integration's +`settings.notifications`. + +## Getting started + +Notifications are **off** until you turn them on. Switching them on gives you a +sensible starting set rather than a blank page: + +| Rule | On by default? | +|---|---| +| Critical error | ✅ | +| Error rate | ✅ | +| Success-rate drop | ✅ | +| Recovery — all clear | ✅ | +| Warning rate | ❌ opt-in | +| Silence / heartbeat | ❌ opt-in | + +Plus a **weekly digest**, Monday 08:00 in your organization's timezone, that skips +itself when nothing happened. + +Add at least one recipient and save. The defaults are deliberately quiet — the two +noisiest rules start off. + +:::tip +Use **Send test** on the Notifications tab to see exactly what an alert or digest looks +like. It renders a real notification and sends it **only to you**, and it is not +written to the notification history. +::: + +## The rules + +Six trigger types, each answering a different question. + +### Critical error + +Notifies the moment a single error occurs. This is the real-time path — it does not +wait for a sweep. + +Because one bad deployment on your side can produce thousands of identical errors in +a minute, identical alerts are collapsed: once an alert has fired for a given code, the +same code stays quiet for the rule's window (2 hours by default) instead of paging you +repeatedly for one incident. Suppressed decisions still appear in the history, marked +`debounced`, so you can see what was folded in. + +### Error rate and warning rate + +Notify when errors — or warnings — in a window climb above normal. Use these instead +of *Critical error* when a trickle of failures is expected and only a surge matters. + +Warning rate is off by default and, when enabled, is in-app only: warnings are common +enough that mailing every spike is rarely what you want. + +### Success-rate drop + +Notifies when the proportion of successful events falls below normal. This is the best +single health signal, because it catches "everything is slow and half of it is failing" +without you having to guess an error count. + +It only evaluates once there are enough events to be meaningful — a handful of events +with one failure is not a 50% outage. That minimum sample counts **outcomes**: `info` +events (queued messages, ignored duplicates) are excluded, exactly as they are from the +[success rate itself](./overview.md#levels). + +### Recovery — all clear + +Notifies when an alerting integration returns to healthy. Worth leaving on: without it, +you learn things broke but never that they resolved. + +### Silence / heartbeat + +Notifies when an integration goes unusually **quiet**. This is the rule that catches +what counts and rates cannot: a middleware that has stopped calling produces no errors, +so every other rule stays green while nothing is being synced at all. + +Off by default because a legitimately intermittent integration would cry wolf. If yours +sends continuously, turn it on — it is the difference between noticing a dead feed in +minutes and noticing it in a month. + +## Fixed thresholds or automatic + +Rate rules take either a number you choose or `auto`. + +**A fixed threshold** is predictable and right when you know your own numbers — "more +than 20 errors an hour is a problem." + +**`auto`** learns what normal looks like for *this* integration at *this* time of week, +and alerts on departures from it. It exists because integration traffic is not flat: a +nightly batch at 03:00 and a quiet Sunday afternoon have completely different normal +error counts, and one fixed number cannot be right for both. `auto` compares like with +like — this Tuesday 09:00 against previous Tuesday 09:00s — and is deliberately +resistant to one-off spikes, so a single bad hour does not teach it that bad hours are +normal. + +Two knobs shape it: + +| Setting | Effect | +|---|---| +| **Sensitivity** — low / medium / high | How far from normal counts as abnormal. Higher sensitivity alerts on smaller departures. Start at medium. | +| **Fallback threshold** | A plain number used while there is not yet enough history to know what normal is. | + +**Cold start:** a new integration has no history, so `auto` behaves as the fallback +threshold until roughly two weeks of data exist, then switches over on its own. You do +not have to do anything, but it does mean a brand-new integration alerts on the +fallback number — set one you would actually be happy with. + +The Notifications tab draws the learned range as a band with your current value marked +against it. The band is what the rule currently considers normal for this hour of the +week; a value outside it is what triggers the alert. A rule still in cold start shows +no band. + +## Choosing what to watch {#choosing-what-to-watch} + +By default a rule watches **every error-level code** across every use case. You can +narrow both axes. + +**By use case** — restrict the integration to specific use cases, so a known-noisy +import does not drown a critical sync. + +**By code** — either pick individual [codes](./codes.md), or use a group: + +| Group | Matches | +|---|---| +| `_error_` | Every error-level code | +| `_warning_` | Every warning-level code | +| `_success_` | Every success-level code | +| `_info_` | Every info-level code | +| `_any_` | Everything, any level | +| `_parent_` | Whatever the integration-level scope is set to | + +Groups and individual codes combine, so `_error_` plus `ACK_TIMEOUT` means "all errors, +and also that one warning I care about". Because [a code's level is +fixed](./overview.md#levels), a group scope keeps meaning the same thing as the +taxonomy grows. + +A maximum of 20 rules per integration keeps evaluation bounded. + +## Recipients and channels + +Recipients are epilot users in the same organization. Channels are **email** and +**in-app**, set once as a default and overridable per rule — for instance error rate by +email and in-app, warning rate in-app only. + +:::note +Channel settings are a ceiling, not an override. Each recipient's own notification +preferences still apply, so turning a channel **on** here never forces a delivery +someone has opted out of; turning it **off** does suppress it. A notification skipped +this way appears in the history as `recipient_opt_out`. +::: + +## Digests + +A digest is the scheduled counterpart to alerts: a periodic summary rather than an +interruption. + +| Setting | Options | +|---|---| +| Frequency | Daily, or weekly on a chosen day | +| Time | Any time of day, in a timezone you pick | +| Channels | Email, in-app | +| Include healthy | List every integration, or only ones with issues | +| Skip if empty | Suppress the digest entirely when nothing happened | + +Leaving **skip if empty** on is what keeps a weekly digest worth reading. + +## Muting + +**Mute** silences all non-digest alerts until a time you choose — for a planned ERP +migration, or while you work through a known backlog. Muted decisions are still +recorded in the history, marked `muted`, so muting never hides what happened. It is +time-boxed by design: there is no permanent mute to forget about. + +## Seeing what fired, and why nothing did + +The **Activity** list on the Notifications tab is the notification history: every real +decision, including ones that did **not** result in a message. That second part is what +makes it useful — when someone says "we should have been alerted", the history tells +you whether the rule never fired, or fired and was suppressed: + +| `suppressed_reason` | Meaning | +|---|---| +| `muted` | The integration was muted at the time | +| `debounced` | An alert for that code had already fired inside its window | +| `recipient_opt_out` | Every recipient had that channel switched off | + +Alongside it, the live status shows each rule's current state — `ok`, `alerting` or +`recovered` — with when it last fired and last cleared. + +## From the API + +The configuration lives on the integration itself, at `settings.notifications`, so it +round-trips through the normal integration `GET` and `PUT`. Three endpoints cover the +rest: + +| Endpoint | Purpose | Permission | +|---|---|---| +| `GET …/notifications/status` | Live per-rule state, and the learned band for `auto` rules. Add `?include=baseline_series` for the full picture. | `integration:view` | +| `GET …/notifications/history` | Cursor-paginated decisions, newest first; `?type=` filters by rule type | `integration:view` | +| `POST …/notifications/test` | Render and send one notification to the calling user only | `integration:manage` | + +```bash +curl 'https://integration-toolkit.sls.epilot.io/v2/integrations/{integrationId}/notifications/status' \ + -H 'Authorization: Bearer ' +``` + +:::note +Rules are evaluated on a sweep every few minutes, so status and rate-based alerts are +near-real-time rather than instant. *Critical error* is the exception — it fires on the +event itself. +::: + +## Related + +- [Monitoring Codes](./codes.md) — what to scope your rules to +- [Monitoring Overview](./overview.md) — levels, and why `info` is excluded from the success rate +- [Investigating events](./investigating.md) — what to do once an alert arrives diff --git a/docs/integrations/integration-toolkit/monitoring/codes.md b/docs/integrations/integration-toolkit/monitoring/codes.md new file mode 100644 index 00000000..57ae52e6 --- /dev/null +++ b/docs/integrations/integration-toolkit/monitoring/codes.md @@ -0,0 +1,139 @@ +--- +sidebar_position: 2 +title: Monitoring Codes +description: Every monitoring code the Integration Toolkit emits, what it means, and what to do about it +slug: /integrations/integration-toolkit/monitoring/codes +--- + +{/* AUTOGENERATED — do not edit by hand. + Source: erp-integration-api packages/erp-utils/src/monitoring/__snapshots__/monitoring-codes.json + Regenerate: npm run update-monitoring-codes */} + +# Monitoring Codes + +Every monitoring event carries a **code** and a **level**. The code says what +happened; the level says how much you should care. Both are filterable in the +Integration Hub's [Monitoring tab](./overview.md) and through the events API. + +There are 80 codes. You are most likely here because you saw one in a failed +event — find it below. + +:::tip +A code's level is fixed. That is what makes alert rules scoped to `_error_` or +`_warning_` predictable — see [Alerting](./alerting.md#choosing-what-to-watch). +::: + +## Error + +Something failed and the event did not do what it was meant to do. These are what the default alert rules watch. + +| Code | What it means | +|---|---| +| `ATTACHMENT_NOT_FOUND` | The file no longer exists — it was removed between the event and the delivery | +| `ATTRIBUTE_TYPE_MISMATCH` | An attribute value did not match the type declared in the entity schema | +| `DEPRECATED_ENDPOINT` | This endpoint version is deprecated | +| `DIRECT_ENTITY_NOT_ALLOWED` | The entity is not permitted by the use case entity allowlist | +| `DIRECT_PAYLOAD_INVALID` | The direct mode payload failed validation against the versioned payload schema | +| `DIRECT_VERSION_UNSUPPORTED` | The direct mode payload version is not supported by the platform | +| `ENTITY_REFERENCE_NOT_FOUND` | A direct-by-id entity reference points at an entity that does not exist or has a different schema | +| `EVENT_NOT_CONFIGURED` | No mapping configuration found for this event type | +| `EXTERNAL_API_ERROR` | An external API returned an error | +| `EXTERNAL_ERROR` | An error span pushed by an external system (e.g. your integration middleware) via the external monitoring events endpoint. epilot assigns the code from the span's level; the middleware never sends one. | +| `FAN_OUT_INVALID_RESULT` | The split expression returned something other than a list, so no deliveries could be created — see split_expression_result_type | +| `FILE_EXTRACTION_FAILED` | Failed to extract file from the request | +| `FILE_FETCH_FAILED` | The file could not be fetched from epilot before it could be uploaded | +| `FILE_PROXY_UPLOAD_FAILED` | A file upload failed terminally, or every delivery attempt was exhausted | +| `FILE_TOO_LARGE` | The file exceeds the maximum size configured on the upload use case | +| `INTEGRATION_NOT_FOUND` | The inbound event referenced an integration_id that does not exist in the calling token's organization. Check the integration id and that the token belongs to the same org. | +| `INVALID_METER_READING_ATTRIBUTES` | Meter reading has invalid attributes | +| `MALFORMED_PAYLOAD` | The event payload could not be parsed (malformed JSON) | +| `MAPPING_EXPRESSION_FAILED` | A mapping expression failed to evaluate | +| `METERING_API_ERROR` | The metering API returned an error | +| `METER_READING_GROUP_FAILED` | A batch write of meter readings failed permanently after all retries — summary event alongside the per-reading errors | +| `METER_READING_GROUP_RETRYING` | A batch write of meter readings failed and will be retried automatically — one event per attempt covering the whole group (reading_count and external_ids in details) | +| `MISSING_REQUIRED_PARAM` | A required parameter is missing from the request | +| `MISSING_UNIQUE_IDENTIFIERS` | The event is missing the unique identifier field(s) required to match an entity | +| `OAUTH2_TOKEN_FAILURE` | Failed to obtain an OAuth2 access token | +| `PAYLOAD_TOO_LARGE` | The payload exceeded the maximum size accepted by the receiving system | +| `PRUNE_SCOPE_PARTIAL_FAILURE` | Scope pruning completed with some failures | +| `RECURSION_DEPTH_EXCEEDED` | Maximum recursion depth was exceeded during processing | +| `RELATION_REF_ITEM_NOT_FOUND` | The relation_ref target entity exists but the referenced item/value could not be matched — skipped as non-retryable. Check the mapping configuration and the entity data. | +| `RELATION_REF_VALUE_UNDEFINED` | A relation_ref mapping value resolved to undefined — check the mapping expression | +| `REQUIRED_PARAM_MISSING` | A param the use case marks as required resolved to nothing, so the delivery was stopped before anything was sent — see param_name | +| `SECURE_PROXY_DISABLED` | The secure proxy use case is disabled | +| `SECURE_PROXY_DOMAIN_BLOCKED` | The target domain is blocked | +| `SECURE_PROXY_DOMAIN_NOT_ALLOWED` | The target domain is not in the allowlist | +| `SECURE_PROXY_ERROR` | An error occurred in the secure proxy | +| `SECURE_PROXY_INVALID_CONFIG` | The secure proxy configuration is invalid | +| `SECURE_PROXY_INVALID_TYPE` | The secure proxy type is invalid | +| `SECURE_PROXY_INVALID_URL` | The target URL is invalid | +| `SECURE_PROXY_IP_BLOCKED` | The target IP address is blocked | +| `SECURE_PROXY_IP_NOT_ALLOWED` | The target IP address is not in the allowlist | +| `SECURE_PROXY_NOT_FOUND` | The secure proxy use case was not found | +| `SECURE_PROXY_UNAVAILABLE` | Secure proxy service is unavailable | +| `SIGNATURE_VERIFICATION_FAILED` | Request signature could not be verified | +| `SIGNATURE_VERIFICATION_UNAVAILABLE` | The file service could not be reached to verify the request signature | +| `STEP_DISABLED` | A request step's "run this step when" expression returned false, so this step and every step after it were skipped. This is the configuration working as written, not a fault. | +| `TIMEOUT` | The operation timed out | +| `UNIQUE_ID_MULTIPLE_MATCHES` | Multiple entities matched the unique ID | +| `UNIQUE_ID_NOT_IN_SCHEMA` | The unique ID attribute is not defined in the entity schema | +| `UNKNOWN_ERROR` | An unexpected error occurred during processing | +| `USE_CASE_DISABLED` | The use case is currently disabled | +| `USE_CASE_INVALID_TYPE` | The use case type is invalid or unsupported | +| `USE_CASE_MISSING_CONFIG` | The use case is missing required configuration | +| `USE_CASE_NOT_FOUND` | The use case could not be found | + +## Warning + +Processing continued, but something needs a human eye — often a retry in flight or a value nobody has mapped yet. + +| Code | What it means | +|---|---| +| `ACK_TIMEOUT` | Acknowledgement timed out waiting for the ERP system | +| `EXTERNAL_WARNING` | A warning span pushed by an external system via the external monitoring events endpoint. | +| `FILE_PROXY_UPLOAD_RETRYING` | A file upload failed with a retryable error and will be retried automatically — one event per attempt | +| `LOOKUP_UNMAPPED` | A value was not listed in a lookup table and its fallback was used — see lookup_name and lookup_key for the gap | +| `SOFT_DELETED_ENTITY_MATCHED` | A soft-deleted entity matched the unique ID — it will be resurrected on upsert, or referenced as-is by a relation. Investigate why the ERP source is sending events for a deleted entity. | + +## Success + +The event did what it was meant to do. Useful for confirming a sync actually landed. + +| Code | What it means | +|---|---| +| `ENTITY_CREATED` | A new entity was created in epilot | +| `ENTITY_DELETED` | An entity was deleted from epilot | +| `ENTITY_NO_OP` | No changes were needed for the entity | +| `ENTITY_UPDATED` | An existing entity was updated in epilot | +| `EXTERNAL_SUCCESS` | A success span pushed by an external system via the external monitoring events endpoint. | +| `FILE_PROXY_OK` | File proxy request completed successfully | +| `FILE_PROXY_UPLOADED` | The external system accepted the file | +| `METER_READING_DELETED` | One or more meter readings were deleted — emitted once per batch, not per reading (reading_count and external_ids in details) | +| `METER_READING_UPSERTED` | One or more meter readings were created or updated — emitted once per batch, not per reading (reading_count and external_ids in details) | +| `PRUNE_SCOPE_COMPLETED` | Scope pruning completed successfully | +| `WEBHOOK_DELIVERED` | Webhook was delivered successfully | + +## Info + +Lifecycle markers rather than outcomes: a message was queued, a duplicate was ignored, a step was skipped by configuration. Counted in total events, but excluded from the success rate. + +| Code | What it means | +|---|---| +| `ACK_CONFIRMED` | Acknowledgement was confirmed by the ERP system | +| `ACK_PENDING` | Acknowledgement is pending from the ERP system | +| `DUPLICATE_EVENT` | This event was already processed (duplicate) | +| `EXTERNAL_INFO` | An informational span pushed by an external system via the external monitoring events endpoint. | +| `FAN_OUT_EMPTY` | The split expression returned an empty list, so nothing was sent — expected for events that carry no relevant items | +| `FILE_PROXY_UPLOAD_ENQUEUED` | A per-file upload was accepted for delivery during fan-out | +| `MSG_ACKED` | Outbound message acknowledged by the polling consumer and removed from the queue | +| `MSG_DEAD_LETTERED` | Outbound message moved to the dead-letter queue after exhausting delivery attempts, or via an operator skip | +| `MSG_ENQUEUED` | Outbound message enqueued to the poll queue, awaiting consumption by the ERP | +| `MSG_EXPIRED_UNPOLLED` | Outbound message expired before being consumed — retention elapsed without a successful poll | +| `MSG_HEAD_BLOCKED` | Outbound stream halted by a poison head message (block policy) — requires operator unblock or consumer acknowledgement | +## Status-code families + +Some codes are generated from an upstream response rather than drawn from the fixed list above. + +| Pattern | Level | What it means | +|---|---|---| +| `HTTP_{status}` | warning | The upstream system answered the proxied request with this HTTP status. The proxy itself worked — it reached the target and returned its answer — so the refusal is recorded against the use case that owns the request. | diff --git a/docs/integrations/integration-toolkit/external-monitoring-events.md b/docs/integrations/integration-toolkit/monitoring/external-events.md similarity index 99% rename from docs/integrations/integration-toolkit/external-monitoring-events.md rename to docs/integrations/integration-toolkit/monitoring/external-events.md index 7214abf9..0ae27643 100644 --- a/docs/integrations/integration-toolkit/external-monitoring-events.md +++ b/docs/integrations/integration-toolkit/monitoring/external-events.md @@ -1,5 +1,5 @@ --- -sidebar_position: 8 +sidebar_position: 6 title: External Monitoring Events description: Push monitoring events from an external system (e.g. an integration middleware) into epilot so the Integration Hub is your central monitoring point, with a cross-system event trace slug: /integrations/integration-toolkit/external-monitoring-events @@ -12,7 +12,7 @@ When part of your integration pipeline runs **outside epilot** — typically an :::info Topology This is for the case where epilot still does the bulk of the work (mapping, entity/metering updates). The middleware does two independent things: -1. **forwards the inbound data event** to the standard [inbound endpoint](./inbound/getting-started.md), and +1. **forwards the inbound data event** to the standard [inbound endpoint](../inbound/getting-started.md), and 2. **separately pushes its own monitoring events** (its processing steps) to the endpoint below. The two halves are linked into one trace by a shared `correlation_id`. diff --git a/docs/integrations/integration-toolkit/monitoring/investigating.md b/docs/integrations/integration-toolkit/monitoring/investigating.md new file mode 100644 index 00000000..818167ab --- /dev/null +++ b/docs/integrations/integration-toolkit/monitoring/investigating.md @@ -0,0 +1,181 @@ +--- +sidebar_position: 3 +title: Investigating Events +description: Trace a failed event across systems, read the captured request and response, replay it, and query the monitoring stream from the API +slug: /integrations/integration-toolkit/monitoring/investigating +--- + +# Investigating Events + +Something went wrong and you need to find out what. This page is the path from "the +Hub shows red" to "I know why, and I have fixed it". + +## Start with the event + +In the integration's **Monitoring** tab, filter the event table to `error` and pick +the event. The detail panel gives you the [code](./codes.md), the message, and the +`detail` object with everything the producer knew at the time. + +For anything that made an HTTP call — file proxy, managed call, secure proxy, outbound +delivery — `detail` also carries the **captured request and response**, rendered as +panels. This is usually where the answer is: you can see exactly what epilot sent and +exactly what came back. + +### What is captured, and what is not + +Two deliberate limits shape those panels. + +**Credentials are never stored.** Headers and body fields that carry secrets are +replaced with `` before anything is written — in both directions, and +wherever they appear in the structure. That covers the obvious names and the +credential-shaped ones a mapping might introduce, and it covers bodies as well as +headers, because an OAuth2 exchange puts the secret in the body and gets a token back +in the response. + +:::tip +If you configure a custom authentication header, prefer routing the call through the +[Secure Proxy](../configuration.md#secure-proxy-use-cases). It injects credentials +server-side, so the secret never travels through the part of the request that gets +captured at all — a stronger guarantee than relying on name matching. +::: + +**Long values are elided, not truncated.** A string over 256 characters is replaced +with its first 32 characters, a `✂ truncated` marker, and its original length. The +result is still valid, copyable JSON with every field in place — you can see that a +base64 document was there and how big it was, without the analytics store having to +hold it. A whole captured body is capped at 4KB. + +So a payload that looks cut off is working as intended. If you need the full body, +reproduce the call against your own system with the field values shown around the +elision. + +## Tracing {#tracing} + +One failure is rarely one row. Two endpoints group the stream, and they answer +different questions — see +[event_id vs correlation_id](./overview.md#event_id-and-correlation_id-are-different-groupings) +for why. + +**Everything one epilot event produced** — the fan-out: child entities, post-actions, +relation resolutions: + +```bash +curl 'https://integration-toolkit.sls.epilot.io/v2/integrations/{integrationId}/monitoring/events/{eventId}/associated' \ + -H 'Authorization: Bearer ' +``` + +It returns the associated events in chronological order, plus the original inbound +payload as the head of the trace when it is still within its 14-day window. + +**The whole business operation, across systems** — including spans your middleware +pushed: + +```bash +curl 'https://integration-toolkit.sls.epilot.io/v2/integrations/{integrationId}/monitoring/traces/{correlationId}' \ + -H 'Authorization: Bearer ' +``` + +This one rolls the spans up into a single `status` (`error` beats `warning` beats +`success` beats `info`), reports `started_at` / `ended_at`, and flags `truncated` when +the trace is longer than the returned window. Spans that came from outside epilot are +identifiable by their `EXTERNAL_*` code prefix. + +## Replaying events {#replaying-events} + +Once you have fixed the cause — corrected a mapping, mapped a missing value, brought +the ERP back up — replay the events that failed. + +```bash +curl -X POST 'https://integration-toolkit.sls.epilot.io/v1/integrations/{integrationId}/events/replay' \ + -H 'Authorization: Bearer ' \ + -H 'Content-Type: application/json' \ + -d '{ "event_ids": ["evt_91af", "evt_91b0"] }' +``` + +Things worth knowing before you use it: + +- **Inbound only.** Replay reconstructs a received inbound event. It does not re-send + outbound deliveries. +- **Maximum 100 event ids per request.** +- **14-day horizon.** The event is rebuilt from the stored inbound payload, which is + kept for 14 days — see [retention](./overview.md#how-long-data-is-kept). Older + events return `not_found`. +- **Always HTTP 200.** Per-event outcomes are in `results`, one entry per requested id + in request order, with a `status` of `success` / `queued`, `not_found`, `skipped`, + `ignored` or `error`. Compare `replayed` against the number of ids you sent rather + than trusting the status code. +- **A replay is a new event.** It is assigned a fresh `event_id` and a new + `correlation_id` prefixed `replay_`, so the replayed run is distinguishable from the + original in monitoring rather than overwriting it. + +## Querying from the API + +Everything the Monitoring tab shows is available directly. All three endpoints take a +POST body and are scoped to one integration. + +**The event stream** — filter by `level`, `code`, `use_case_id`, `use_case_type`, +`event_id`, `correlation_id` and a time range, with cursor pagination: + +```bash +curl -X POST 'https://integration-toolkit.sls.epilot.io/v2/integrations/{integrationId}/monitoring/events' \ + -H 'Authorization: Bearer ' \ + -H 'Content-Type: application/json' \ + -d '{ + "level": "error", + "use_case_type": "inbound", + "from_date": "2026-01-15T00:00:00Z", + "limit": 50 + }' +``` + +**Aggregates** — `…/monitoring/stats` for totals, success rate and `ack_timeout_count`, +with an optional `group_by` of `use_case_id`, `use_case_type`, `level`, `code` or +`date`. **Time series** — `…/monitoring/time-series` for bucketed counts at `5m`, +`10m`, `30m`, `1h`, `3h` or `1d`. + +### Which number am I looking at? {#which-number} + +Stats have two sources, and they answer genuinely different questions. Reading one as +the other is the most common way to reach a wrong conclusion from this data. + +| `source` | Counts | Use it for | +|---|---|---| +| `monitoring` *(default)* | Every row in the processing tree — fan-out children, post-actions, relation resolutions | "How much work happened, and how much of it failed" | +| `incoming` | Distinct events actually received, before any fan-out | "How many events did the ERP actually send us" | + +One received event routinely produces many monitoring rows, so `monitoring` totals are +legitimately much larger than `incoming` ones. Neither is wrong; they are different +denominators. + +:::warning +`source: incoming` reads the received-payload store, which is kept for **14 days**. A +query with a longer window silently under-reports — a 30-day chart will show a +suspicious cliff two weeks back, and a 30-day total will simply be too low. For +anything beyond 14 days, use the default `monitoring` source. +::: + +`source: incoming` also supports only `group_by: use_case_id`, and returns zero for the +level breakdown, because the received-payload store has no notion of an outcome. + +## Who called the API + +The **Access Token** tab, and `…/monitoring/access-logs`, show how the integration's +API tokens are actually being used — filterable by `token_id`, `service`, `method`, +`path`, `status` and time range, with cursor pagination. + +```bash +curl -X POST 'https://integration-toolkit.sls.epilot.io/v1/integrations/{integrationId}/monitoring/access-logs' \ + -H 'Authorization: Bearer ' \ + -H 'Content-Type: application/json' \ + -d '{ "status": 403, "from_date": "2026-01-15T00:00:00Z" }' +``` + +This is the place to answer "is the middleware still calling us", "which token is +being used for what", and "why is it getting 403s" — questions the monitoring stream +cannot answer, because a rejected call never becomes a monitoring event. + +## Related + +- [Monitoring Codes](./codes.md) — what the code you found means +- [Alerting](./alerting.md) — find out without looking +- [External Monitoring Events](./external-events.md) — make traces span your middleware diff --git a/docs/integrations/integration-toolkit/monitoring/overview.md b/docs/integrations/integration-toolkit/monitoring/overview.md new file mode 100644 index 00000000..14edf657 --- /dev/null +++ b/docs/integrations/integration-toolkit/monitoring/overview.md @@ -0,0 +1,168 @@ +--- +sidebar_position: 1 +title: Monitoring Overview +description: How Integration Toolkit monitoring is structured — levels, codes, event and correlation ids, use case lanes, and how long data is kept +slug: /integrations/integration-toolkit/monitoring/overview +--- + +# Monitoring Overview + +Every event the Integration Toolkit processes leaves a trail. Inbound syncs, outbound +deliveries, file proxy fetches, managed calls and secure proxy requests all write to +one monitoring stream, visible in the Integration Hub's **Monitoring** tab and +queryable through the API. + +This page covers how that data is shaped, so the rest of the section makes sense. If +you are chasing a specific failure right now, go to +[Investigating events](./investigating.md) or look your code up in the +[code reference](./codes.md). + +## Where to look + +Open an integration in epilot 360 (**Integrations → your integration**) and use the +**Monitoring** tab. It gives you, in one place: + +- **Stat tiles** — total events, successes, errors, warnings, success rate, and + ACK timeouts for the selected period +- **Events over time** — a bucketed chart, optionally split by use case or lane +- **Use case breakdown** — which use cases produce the volume, and the errors +- **The event table** — filterable by level, code, use case and time, with a detail + panel per event +- **Access logs** — which API token called what + +The **Notifications** tab beside it is where you configure who gets told about all +this — see [Alerting](./alerting.md). + +## How an event is shaped + +Each row in the stream is one monitoring event: + +| Field | What it is | +|---|---| +| `level` | How much you should care: `success`, `error`, `warning` or `info` | +| `code` | What specifically happened — see the [code reference](./codes.md) | +| `message` | A human-readable line, usually the error text | +| `detail` | Free-form JSON with context for that code — including the captured request and response where there was one | +| `use_case_type` | Which lane produced it | +| `use_case_id` | Which configured use case, when one owns the event | +| `event_id` | The triggering event | +| `correlation_id` | The business operation | +| `created_at` | When it happened | + +### Levels + +There are four, and the distinction between the last two matters more than it looks: + +| Level | Meaning | +|---|---| +| `success` | The event did what it was meant to do | +| `error` | It failed | +| `warning` | It continued, but something needs a human eye — a retry in flight, an unmapped value | +| `info` | A lifecycle marker rather than an outcome: a message queued, a duplicate ignored, a step skipped by configuration | + +**`info` events are excluded from the success rate.** They are counted in +`total_events`, but a queued message or an ignored duplicate is not something that +could have succeeded or failed, so including them would drag the rate down for no +reason. An integration using ACK tracking or the poll queue emits a lot of them. + +A code's level is fixed — `ENTITY_CREATED` is always `success`, `ACK_TIMEOUT` is always +`warning`. That is what makes [alert rules](./alerting.md#choosing-what-to-watch) +scoped to a whole level predictable. + +### Use case lanes + +`use_case_type` separates the five kinds of work, so one noisy lane does not hide +another: + +| Lane | What it covers | +|---|---| +| `inbound` | ERP data arriving and being applied to entities and meter readings | +| `outbound` | epilot events delivered to your ERP by webhook or poll queue | +| `file_proxy` | Files fetched from, or delivered to, an external document system | +| `managed_call` | Synchronous calls to an external API | +| `secure_proxy` | Requests routed through the static-IP or VPN proxy | + +Two values in the use case column are not use cases: + +- **"General"** (an empty `use_case_id`) — system-level events that happened before + any use case was resolved, such as an event rejected at ingest. +- **`__unknown__`** — the event names a use case id that no longer exists on this + integration, usually because it was deleted or recreated. The history is real; the + configuration behind it is gone. + +### `event_id` and `correlation_id` are different groupings + +This is the single most useful thing to understand about the stream, because the two +answer different questions. + +One inbound event does not produce one monitoring row. It fans out — child entities, +post-actions, relation resolutions — and each step records its own event. All of those +share the **`event_id`** of the thing that triggered them. + +A **`correlation_id`** is wider: it identifies one *business operation*, and can span +several events and even several systems. If your middleware stamps the same +`correlation_id` on the spans it pushes and on the event it forwards, one trace covers +both halves — that is what [External Monitoring Events](./external-events.md) is for. + +``` +correlation_id: "bp-8f3a2c" ← one business operation +├── middleware span: received (external) +├── middleware span: mapped (external) +└── event_id: "evt_91af" ← one epilot event + ├── contact created + ├── billing account created + └── relation resolved +``` + +Two endpoints match the two groupings, and picking the wrong one is why a trace can +look incomplete: + +| To see | Use | +|---|---| +| Everything one epilot event produced | `GET …/monitoring/events/{eventId}/associated` | +| The whole business operation, across systems | `GET …/monitoring/traces/{correlationId}` | + +Both are covered in [Investigating events](./investigating.md#tracing). + +## How long data is kept + +| Data | Retained | What that limits | +|---|---|---| +| Monitoring events | **90 days** | How far back stats, charts and the event table can go | +| Received inbound payloads | **14 days** | How far back you can [replay](./investigating.md#replaying-events), and how long a trace can still show the original payload | + +The 14-day figure is the one that catches people out. Replay reconstructs the event +from the stored inbound payload, so **an event older than 14 days cannot be replayed** +— the monitoring record of it survives for the full 90 days, but the payload behind it +does not. If you are working through a backlog of failures, work through the oldest +first. + +## Is a specific entity up to date? + +Monitoring answers "what happened". A separate endpoint answers "is *this record* +current with the ERP": + +```bash +curl 'https://integration-toolkit.sls.epilot.io/v1/integrations/entities/{entityId}/sync-status' \ + -H 'Authorization: Bearer ' +``` + +It returns, per integration that has touched the entity, when it was last **synced** +and when it was last **changed**. + +The distinction matters: `last_synced_at` also advances on a no-op — an event that was +received and evaluated but changed nothing. Those deliberately leave no trace on the +entity itself: no activity feed entry, no `_updated_at` bump. So an entity that looks +untouched for weeks may be being checked constantly and simply not changing, and this +endpoint is the only way to tell those two states apart. An entity no inbound use case +has ever processed returns an empty list. + +Add `?integration_id=…` to narrow it to one integration. + +## Next + +- [Monitoring Codes](./codes.md) — look up what you saw +- [Investigating events](./investigating.md) — traces, captured payloads, replay +- [Alerting](./alerting.md) — get told without watching +- [ACK Tracking](./acks.md) — confirm the ERP processed what you sent +- [External Monitoring Events](./external-events.md) — bring your middleware's steps in diff --git a/docs/integrations/integration-toolkit/outbound-file-delivery.md b/docs/integrations/integration-toolkit/outbound-file-delivery.md index b0b59eee..2123ddf1 100644 --- a/docs/integrations/integration-toolkit/outbound-file-delivery.md +++ b/docs/integrations/integration-toolkit/outbound-file-delivery.md @@ -280,5 +280,5 @@ Open an entry to see the use case, delivery, attachment, attempt, and available - [Core Events](/docs/integrations/core-events) - [Configuration](./configuration.md) - [File Proxy downloads](./file-proxy.md) -- [External Monitoring Events](./external-monitoring-events.md) +- [External Monitoring Events](./monitoring/external-events.md) - [Pollable Outbound](./pollable-outbound.md) diff --git a/docs/integrations/integration-toolkit/overview.md b/docs/integrations/integration-toolkit/overview.md index 39ea430a..d03e1e5a 100644 --- a/docs/integrations/integration-toolkit/overview.md +++ b/docs/integrations/integration-toolkit/overview.md @@ -24,7 +24,7 @@ The Integration Toolkit is composed of the following components. Each plays a sp | Component | Description | Status | |-----------|-------------|--------| -| **[Integration Hub](#integration-hub)** | Admin UI in epilot 360 to configure and monitor integrations | In progress | +| **[Integration Hub](#integration-hub)** | Admin UI in epilot 360 to configure and monitor integrations | Stable | | **[ERP Integration API](#erp-integration-api)** | CRUD API to manage integrations, use cases, and mappings | Stable | | **[ERP Inbound API](#inbound-api)** | Dedicated API to receive and simulate inbound ERP events | Stable | | **[Use Cases](./use-cases.md)** | Documented integration flows with testing support | Stable | @@ -32,13 +32,15 @@ The Integration Toolkit is composed of the following components. Each plays a sp | **[Changesets](/docs/entities/changesets)** | Pending attribute updates that wait for ERP confirmation or human approval | Stable | | **[Core Events](/docs/integrations/core-events)** | Standardized event payloads for outbound notifications | Stable | | **[Webhooks](/docs/integrations/webhooks)** | Push events from epilot to ERPs via core events | Stable | -| **[Pollable Outbound](./pollable-outbound.md)** | Pull-based outbound delivery — ERPs poll a queue instead of receiving webhooks | In progress | +| **[Pollable Outbound](./pollable-outbound.md)** | Pull-based outbound delivery — ERPs poll a queue instead of receiving webhooks | Stable | | **[JSONata Mapping](#jsonata-mapping)** | Transformation language for inbound and outbound data | Stable | | **[File Proxy](./file-proxy.md)** | Serve files from external archives on demand without migrating them into epilot | Stable | | **[Outbound File Delivery](./outbound-file-delivery.md)** | Deliver files referenced by epilot events to external document APIs | Stable | | **[Managed Calls](#managed-calls)** | Synchronous external API calls with JSONata mapping via connector integrations | Stable | | **[Secure Proxy](#secure-proxy)** | Route HTTP requests through epilot's secure proxy for static IP egress or VPN access | Stable | -| **[Monitoring and ACKs](#monitoring-and-acks)** | Central logging, error tracking, and event replay | In progress | +| **[Monitoring](./monitoring/overview.md)** | Central event logging, stats, cross-system traces, replay, and the code reference | Stable | +| **[Alerting & Notifications](./monitoring/alerting.md)** | Per-integration alert rules, anomaly baselines, digests, and delivery to email/in-app | Stable | +| **[ACK Tracking](./monitoring/acks.md)** | ERPs acknowledge processed events, closing end-to-end delivery visibility | Stable | | **[Blueprints](https://marketplace.epilot.cloud/en/blueprints)** | Packaged, installable integration setups | Stable | | **[Apps](https://marketplace.epilot.cloud/en/apps)** | Custom automation actions and portal extensions for ERP logic | In progress | @@ -120,14 +122,25 @@ See the [Configuration Guide](./configuration.md#secure-proxy-use-cases) for set - Outbound webhook payloads (epilot event to ERP format) - The Map Data flow building block -### Monitoring and ACKs - -All inbound and outbound events are centrally logged and surfaced in the Integration Hub. Key capabilities: - -- **Event replay** -- reprocess failed events -- **ACK tracking** -- ERPs acknowledge processed events via `v1/erp/tracking/acknowledgement`, enabling end-to-end visibility -- **Error alerting** -- per-use-case status indicators with actionable error details -- **Partner log shipping** -- middleware partners can send logs to epilot for centralized monitoring +### Monitoring and Alerting + +Every event the toolkit processes — inbound, outbound, file proxy, managed call and +secure proxy alike — is recorded as a monitoring event and surfaced in the Integration +Hub's **Monitoring** tab, with a matching API for automation. + +- **[Event stream and stats](./monitoring/overview.md)** — filter by level, code, use + case or correlation id; success rates and time series per lane +- **[Code reference](./monitoring/codes.md)** — every monitoring code, what it means + and what to do about it +- **[Traces and replay](./monitoring/investigating.md)** — follow one business + operation across systems, inspect the captured request and response, and reprocess + events +- **[Alerting](./monitoring/alerting.md)** — per-integration rules with anomaly + baselines, plus scheduled digests, delivered to email and in-app +- **[ACK tracking](./monitoring/acks.md)** — ERPs confirm they processed an event, + closing end-to-end visibility +- **[External monitoring events](./monitoring/external-events.md)** — push your + middleware's own processing steps in, so one trace spans both systems ## Architecture diff --git a/docs/integrations/integration-toolkit/pollable-outbound.md b/docs/integrations/integration-toolkit/pollable-outbound.md index 4c063359..f1752d27 100644 --- a/docs/integrations/integration-toolkit/pollable-outbound.md +++ b/docs/integrations/integration-toolkit/pollable-outbound.md @@ -52,7 +52,7 @@ Key properties: A poll mapping is configured on a regular outbound use case — same endpoint, same envelope as webhook mappings, only the `delivery` object differs. See [Outbound Use Case Configuration](./configuration.md#outbound-use-case-configuration) for the full use-case contract. ```bash -curl -X POST 'https://erp-integration.sls.epilot.io/v1/integrations/{integrationId}/use-cases' \ +curl -X POST 'https://integration-toolkit.sls.epilot.io/v1/integrations/{integrationId}/use-cases' \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -d '{ diff --git a/docs/integrations/integration-toolkit/use-cases.md b/docs/integrations/integration-toolkit/use-cases.md index 432e9c2a..a2ed6dcc 100644 --- a/docs/integrations/integration-toolkit/use-cases.md +++ b/docs/integrations/integration-toolkit/use-cases.md @@ -364,7 +364,7 @@ flowchart LR **What happens in the ERP:** - Middle layer receives the webhook and extracts meter reading data - Middle layer calls the ERP API to submit the reading (e.g., meter reading endpoint) -- Middle layer sends an [ACK](/docs/integrations/integration-toolkit/overview#monitoring-and-acks) back to epilot to confirm processing +- Middle layer sends an [ACK](./monitoring/acks.md) back to epilot to confirm processing **Core Event:** [`MeterReadingAdded`](/docs/integrations/core-events#MeterReadingAdded) @@ -768,18 +768,22 @@ flowchart LR ## ACK Tracking -All outbound use cases support [ACK tracking](/docs/integrations/integration-toolkit/overview#monitoring-and-acks). After processing a webhook, your middle layer should send an acknowledgment back to epilot: +All outbound use cases support [ACK tracking](./monitoring/acks.md). After processing a webhook, your middle layer should send an acknowledgment back to epilot: ```bash title="Send ACK" -curl -X POST 'https://erp-integration.sls.epilot.io/v1/erp/tracking/acknowledgement' \ +curl -X POST 'https://integration-toolkit.sls.epilot.io/v1/erp/tracking/acknowledgement' \ -H 'Content-Type: application/json' \ - -d '{ - "ack_id": "", - "status": "processed" - }' + -d '{ "ack_id": "" }' ``` -This enables end-to-end monitoring in the Integration Hub: per-use-case status indicators show whether the ERP successfully processed each event. +`ack_id` is the only field. Acknowledging is the whole signal — there is no status to +report, because an ACK means "processed"; a failure is simply an ACK that never +arrives, and the event times out. + +This enables end-to-end monitoring in the Integration Hub: per-use-case status +indicators show whether the ERP processed each event. The full lifecycle, the timeout +behaviour and how to turn tracking off per use case are covered in +[ACK Tracking](./monitoring/acks.md). ## Next Steps diff --git a/package.json b/package.json index 90426084..042a9aaf 100644 --- a/package.json +++ b/package.json @@ -23,6 +23,7 @@ "update-events": "cp ../event-catalog-api/packages/core/src/builtin-events/__snapshots__/* static/events", "update-sdk": "./scripts/update-sdk-docs.sh", "update-cli": "./scripts/update-cli-docs.sh", + "update-monitoring-codes": "node scripts/update-monitoring-codes.js", "update-pricing-playground": "./scripts/update-pricing-playground.sh" }, "dependencies": { diff --git a/scripts/update-monitoring-codes.js b/scripts/update-monitoring-codes.js new file mode 100644 index 00000000..08ff9625 --- /dev/null +++ b/scripts/update-monitoring-codes.js @@ -0,0 +1,128 @@ +#!/usr/bin/env node +/** + * Generates the Integration Toolkit monitoring code reference from the snapshot + * emitted by erp-integration-api. + * + * The snapshot is produced and asserted by + * `packages/erp-utils/src/monitoring/code-catalog.test.ts` in that repo, so a code + * added to the taxonomy without a description fails CI there. This script turns + * that artifact into the published table — the docs never hand-maintain the list, + * which is what kept the old hub-only catalog drifting. + * + * Usage: npm run update-monitoring-codes + * Assumes erp-integration-api is checked out alongside this repo (../). + */ + +const fs = require('node:fs'); +const path = require('node:path'); + +const SNAPSHOT = path.resolve( + __dirname, + '../../erp-integration-api/packages/erp-utils/src/monitoring/__snapshots__/monitoring-codes.json', +); +const OUT = path.resolve( + __dirname, + '../docs/integrations/integration-toolkit/monitoring/codes.md', +); + +/** Rendering order and the operator-facing framing for each level. */ +const LEVELS = [ + { + key: 'error', + heading: 'Error', + blurb: + 'Something failed and the event did not do what it was meant to do. These are what the default alert rules watch.', + }, + { + key: 'warning', + heading: 'Warning', + blurb: + 'Processing continued, but something needs a human eye — often a retry in flight or a value nobody has mapped yet.', + }, + { + key: 'success', + heading: 'Success', + blurb: 'The event did what it was meant to do. Useful for confirming a sync actually landed.', + }, + { + key: 'info', + heading: 'Info', + blurb: + 'Lifecycle markers rather than outcomes: a message was queued, a duplicate was ignored, a step was skipped by configuration. Counted in total events, but excluded from the success rate.', + }, +]; + +function escapeCell(text) { + return text.replace(/\|/g, '\\|'); +} + +function render(snapshot) { + const { codes, families } = snapshot; + const byLevel = new Map(LEVELS.map((l) => [l.key, []])); + for (const entry of codes) { + if (!byLevel.has(entry.level)) { + throw new Error(`Unhandled level "${entry.level}" on code ${entry.code}`); + } + byLevel.get(entry.level).push(entry); + } + + const sections = LEVELS.map(({ key, heading, blurb }) => { + const rows = byLevel.get(key); + return [ + `## ${heading}`, + '', + blurb, + '', + '| Code | What it means |', + '|---|---|', + ...rows.map((r) => `| \`${r.code}\` | ${escapeCell(r.description)} |`), + '', + ].join('\n'); + }); + + const familySection = [ + '## Status-code families', + '', + 'Some codes are generated from an upstream response rather than drawn from the fixed list above.', + '', + '| Pattern | Level | What it means |', + '|---|---|---|', + ...families.map( + (f) => `| \`${f.pattern}\` | ${f.level} | ${escapeCell(f.description)} |`, + ), + '', + ].join('\n'); + + return `--- +sidebar_position: 2 +title: Monitoring Codes +description: Every monitoring code the Integration Toolkit emits, what it means, and what to do about it +slug: /integrations/integration-toolkit/monitoring/codes +--- + +{/* AUTOGENERATED — do not edit by hand. + Source: erp-integration-api packages/erp-utils/src/monitoring/__snapshots__/monitoring-codes.json + Regenerate: npm run update-monitoring-codes */} + +# Monitoring Codes + +Every monitoring event carries a **code** and a **level**. The code says what +happened; the level says how much you should care. Both are filterable in the +Integration Hub's [Monitoring tab](./overview.md) and through the events API. + +There are ${codes.length} codes. You are most likely here because you saw one in a failed +event — find it below. + +:::tip +A code's level is fixed. That is what makes alert rules scoped to \`_error_\` or +\`_warning_\` predictable — see [Alerting](./alerting.md#choosing-what-to-watch). +::: + +${sections.join('\n')}${familySection}`; +} + +const snapshot = JSON.parse(fs.readFileSync(SNAPSHOT, 'utf8')); +fs.writeFileSync(OUT, render(snapshot)); +console.log( + `Wrote ${path.relative(process.cwd(), OUT)} — ${snapshot.codes.length} codes, ${snapshot.families.length} famil${snapshot.families.length === 1 ? 'y' : 'ies'}`, +);