Skip to content

request-log.md

Latest commit

 

History

History
305 lines (211 loc) · 26.4 KB

request-log.md

File metadata and controls

305 lines (211 loc) · 26.4 KB
Raw

Stovepipe Request Log

Summary

Stovepipe retains an append-only request log for each validation request. Its internal RequestLog is the counterpart of SubmitQueue's RequestLog: both retain request status changes and explanatory lifecycle events, while Stovepipe calls its materializer directly instead of sending records through a cross-service log topic. The public API presents these records as request history. The log records every durable Request.State transition plus asynchronous milestones needed to explain those transitions, the public verdict, and abandoned stage work:

  • build_triggered;
  • build_finished;
  • validation_fact_recorded;
  • project_facts_recorded;
  • record_abandoned;
  • build_analysis_abandoned;
  • issue_tracking_abandoned;
  • build_analysis_completed;
  • issues_reconciled;
  • fix_submitted;
  • fix_submission_skipped;
  • fix_submission_abandoned.

The model deliberately follows SubmitQueue's distinction between statuses describing where a request is and events describing important activity that does not move it. It remains a bounded request-lifecycle log rather than a generic event bus or an audit of every correlated operation.

Request, Build, and ValidationFact remain the operational sources of truth. The request log is a required audit/read model derived from successful durable operations; it does not replace their optimistic-locking state machines.

The separate Stovepipe Request History API defines public lookup, projection, materialization, ordering, and retention semantics.

Related documents are SubmitQueue Gateway Request History APIs, Stovepipe Workflow, Process, Build, Buildsignal, and Record.

Problem and Scope

Stovepipe currently retains only the latest mutable state of a validation request and build. A ValidationFact retains the immutable verdict for a commit, but none of these records explains how a request reached that verdict.

State changes alone are insufficient. A request remains processing while a build is triggered and finishes, and its terminal state becomes durable before record creates the fact that establishes green or broken. These milestones must remain visible without inventing request states for work occurring underneath the request's position.

An occurrence belongs in the request log only when it is either:

  1. a durable Request.State transition; or
  2. an asynchronous durable milestone explaining validation, issue reconciliation, or a repair submission for the request.

The following remain with their owning entities, metrics, or structured logs:

  • Queue latest-request and last-green bookmarks;
  • source-control promotion;
  • project analysis and individual project facts;
  • hooks and downstream notifications;
  • build-slot claims, waits, and releases;
  • queue handoffs, delivery attempts, holds, nacks, and visibility timeouts;
  • transient dependency errors and DLQ mechanics.

Goals

  1. Retain every request-state transition with its successful request version.
  2. Retain the build and fact milestones needed to explain request progress and verdict creation.
  3. Make writes idempotent under redelivery, restart, and uncertain failure.
  4. Preserve enough source context to repair the request log after a source write succeeds.
  5. Keep storage queue-scoped and compatible with key/value, SQL, document, remote, and in-memory backends.
  6. Preserve persist-before-publish and controller-owned version arithmetic.

Non-goals

  1. Replacing the Request, Build, or ValidationFact stores with event-sourced projections.
  2. Recording every durable operation correlated with a request.
  3. Reconstructing transport behavior or unchanged build polls.
  4. Providing global ordering or cross-request search.
  5. Using the request log as the source of durable greenness; only ValidationFact establishes green or broken.
  6. Publishing internal request-log records directly as cross-domain hook events.

Existing SubmitQueue Pattern

SubmitQueue retains RequestLog separately from its mutable orchestrator Request. Each entry is either a customer-facing status or an event that happened while the request remained at a status. Status entries may carry the orchestrator request version for reconciliation; events carry no request version and never compete to become current status.

Its status vocabulary covers accepted, validating, batched, speculating, landing, landed, error, and cancellation. Its deliberately small event vocabulary covers building, built, waiting, and invalidated rather than every queue hop, retry, storage update, or side effect.

Producers use stable occurrence values in message IDs to deduplicate repeated publication of one logical occurrence. The gateway appends a flat RequestLog with type, status or event, timestamp, request version, error, and metadata. Consumer retries may still retain duplicate log rows because the stored record itself has no idempotency key.

Stovepipe adopts the status/event split, bounded vocabulary, queue scoping, and per-occurrence identity. It strengthens retained idempotency with a stable entry ID so retries converge on one stored occurrence.

Stovepipe writes the request log directly because every stage belongs to the same service and resolves the same queue-scoped storage aggregate. SubmitQueue's cross-service log topic is unnecessary unless log ownership later crosses a service boundary.

Request Log Entity

The retained unit is entity.RequestLog:

type RequestLog struct {
    // ID is the stable identity of one logical occurrence within the request. It is stable across redelivery and never derived from time or randomness.
    ID string
    // Queue is the logical queue containing the request and scopes RequestID.
    Queue string
    // RequestID identifies the request whose log contains this record.
    RequestID string
    // TimestampMs is when the occurrence was first retained, in Unix milliseconds.
    TimestampMs int64
    // State is the durable request state recorded by a state entry. It is unset on an event entry.
    State RequestState
    // Event identifies the occurrence recorded by an event entry. It is unset on a state entry.
    Event RequestEvent
    // RequestVersion is the successful request version recorded by a state entry. It is zero on an event entry.
    RequestVersion int32
    // OutcomeReason is the domain reason for a terminal request state and never names a transport mechanism.
    OutcomeReason RequestOutcomeReason
    // Metadata contains optional occurrence context. Nil and empty maps are equivalent.
    Metadata map[string]string
}

Enums are strings with unknown sentinels, and the entity has no storage or transport dependency. Constructors enforce exactly one of state and event and validate the required context for that kind. The populated field identifies the entry kind without a separate discriminator.

The core column shape follows SubmitQueue's RequestLog, except Stovepipe omits SubmitQueue's redundant Type column. Context that is meaningful only for one occurrence kind remains in the JSON metadata map instead of adding sparse columns.

Metadata is never used for occurrence identity or filtering. Nil and empty maps are equivalent. Storage treats the map as opaque JSON; writers and projections define the bounded base_uri, build_id, fact_degree, and project_fact_count keys. Producers must not store credentials, raw dependency errors, stack traces, or unbounded payloads. The public history API exposes supported keys through typed occurrence payloads, never the raw map; the API contract defines their presence semantics.

State entries retain the selected baseline URI when known, including an empty URI for a full build; this metadata feeds the summary projection and is not exposed in event payloads. When a build established the terminal state, that state entry records its ID as occurrence context. Other Request context and Build status/version remain on their operational entities; history does not load them to enrich old occurrences. The triggered and finished event kinds plus the terminal Request state describe the lifecycle without duplicating Build snapshots. Diagnostic error codes remain in structured logs until a concrete public vocabulary is required.

Vocabularies

State entries

State Required retained context
accepted Request version 1
processing Request version
superseded Superseding request ID metadata, outcome reason, and request version
succeeded Outcome reason, winning build ID metadata, and request version
failed Outcome reason, optional build ID metadata, and request version
cancelled Outcome reason, build ID metadata, and request version

Event entries

Event Meaning Required retained context
build_triggered A runner accepted a build and its Build row became durable. Build ID metadata and creation time
build_finished The Build first reached a write-once terminal status. Build ID metadata and status-change time
validation_fact_recorded The immutable whole-repository fact became durable. Degree metadata and fact creation time
project_facts_recorded All project validation facts returned for a request became durable. Project-fact count metadata and completion time
record_abandoned Record-stage work stopped after exhausting primary retries. Event retention time
build_analysis_abandoned Build analysis could not complete; issue observations may be incomplete. This is not a validation verdict. Build ID metadata and event retention time
issue_tracking_abandoned Issue tracking stopped before completing a pass; existing issue changes remain, but reconciliation may be incomplete. This is not a validation verdict. Build ID metadata and event retention time
build_analysis_completed Issue analysis for the winning terminal build completed and its detected issues were persisted. Build ID, first retained completion's issue count, and event retention time
issues_reconciled A final issue-tracking pass completed, including any deliberate suppression of absent-issue closure. Build ID, closure mode, and event retention time
fix_submitted An identified repair attempt was durably associated with a build issue. It does not imply a patch exists or the issue recovered. Build, task, unit, and fix IDs and event retention time
fix_submission_skipped No eligible open issue existed for the build. Build ID, bounded reason, and event retention time
fix_submission_abandoned Repair processing stopped without a durable fix ID; external acceptance remains unknown. Build ID, bounded reason, and event retention time

Build running and unchanged polls are not retained. Trigger and terminal result explain the request outcome without turning polling into an unbounded log. Individual project facts remain outside the vocabulary; one batch-completion event summarizes their durable recording. Its project_fact_count and broken_project_fact_count metadata contain the total and the number with degree above zero, counted from persisted facts after their indexes are durable. Project completion projection requires both counts and assigns them together in RequestSummary from the retained event.

build_analysis_completed and issues_reconciled use the build ID as their occurrence key. Per-item fix_submitted, fix_submission_skipped, and fix_submission_abandoned events use a digest of opening build, task, unit, and executor identity, retaining the verbatim fix ID or skip reason and executor ID in metadata. Existing issue-tracking abandonment uses a digest of the failed notification ID to fit the storage key limit. The materializer prefixes each key with its event kind, and request identity scopes all entries. Replays retain the first stored timestamp.

Analysis and repair events do not advance Request state or alter validation facts. Analysis completion is retained only when the Request records the analyzed terminal build as its winning outcome, immediately before the final issue-tracking handoff. Its issue count and timestamp are an immutable snapshot of the first retained completion, including zero for an empty result. Retries still persist detected issues and publish the handoff, but reuse that snapshot if later detector results differ; count discrepancies are logged rather than treated as history conflicts. Other semantic conflicts remain errors. issues_reconciled records closure mode reconciled, suppressed_cancelled, or suppressed_task_wide_issue; suppression does not mean reconciliation failed. Skipped repair uses reason no_eligible_issue; abandoned repair uses acceptance_unknown. Raw dependency errors remain in operational logs.

Evolution

State and event strings never change meaning or get reused. A new event must satisfy the scope rule and update required-field validation, storage mapping, public projection, and compatibility tests.

Schema evolution is additive: nullable or defaulted columns and tolerant readers land before writers populate a new field or kind. Incompatible reinterpretation requires a new field or event rather than changing stored meaning; metadata-key semantics are defined when their writers and readers are introduced.

Outcome reasons

Terminal entries retain domain reasons rather than transport mechanisms. Initial reasons include:

  • build_succeeded;
  • build_failed;
  • build_cancelled;
  • processing_failed;
  • build_polling_exhausted;
  • validation_timeout;
  • superseded_by_newer_head.

Stable IDs and Idempotency

stovepipe/core/requestlog composes readable IDs from durable identities with the same publish.IntentID convention used by SubmitQueue before passing each record to Materializer.PersistLog. The surrounding (queue, request_id) storage key scopes the ID to one request, so a request state entry uses state/<request-version> without repeating the request identity:

SubmitQueue applies that identity to the message carrying a log to its materializer, while its log store remains append-only and may retain a duplicate after a later materialization step fails. Stovepipe has no intermediate log topic, so it applies the identity to the retained row itself: retrying the direct call reloads the existing occurrence and succeeds only when its semantic content is compatible.

Entry Stable identity inputs
Request state transition Request ID and request version
Build triggered Request ID, event kind, and build ID
Build finished Request ID, event kind, and build ID
Validation fact recorded Request ID, event kind, and whole-repository fact identity
Project facts recorded Request ID, event kind, and the completed project-fact batch

The controller passes the materializer the same queue-scoped storage aggregate used for the source write. The materializer preserves a supplied occurrence time or assigns the current time immediately before the first insertion attempt, then calls RequestLogStore.Create. If the ID already exists, it loads the stored record and compares the explicitly designated stable semantic fields. The first successfully retained timestamp is authoritative and is not compared with a later retry's newly sampled time. Metadata keys emitted by both records must agree, while a key present on only one record remains compatible so an additive metadata rollout does not turn retries of older occurrences into conflicts. Compatible content is idempotent success; conflicting content is an internal consistency error, and the stored record is never overwritten or enriched.

Storage Contract

The queue-scoped store is append-only and key/value-shaped:

type RequestLogStore interface {
    Create(ctx context.Context, log entity.RequestLog) error
    Get(ctx context.Context, requestID, logID string) (entity.RequestLog, error)
    List(ctx context.Context, requestID string) ([]entity.RequestLog, error)
}

Create returns ErrAlreadyExists for an existing ID. List returns every retained record for one request ordered by (timestamp_ms ASC, log_id ASC). An empty result does not determine whether the request exists; the controller loads Request separately to distinguish an unknown request from an existing request with no retained records.

The contract exposes no update, delete, query-by-state, query-by-event, query-by-build, cross-request search, or cross-queue enumeration. A key/value backend can represent the request as a partition and entries as immutable child records. The bounded lifecycle vocabulary and reasonable per-request build limits keep this point read modest, matching SubmitQueue's RequestLogStore.List contract.

The initial MySQL table uses:

metadata JSON NOT NULL
PRIMARY KEY (queue, request_id, log_id)
KEY idx_request_log_order (queue, request_id, timestamp_ms, log_id)

Metadata is serialized as a JSON object and normalized to an empty map on write and read. The storage layer does not interpret its keys or values. The ordering index covers the request partition and stable order without introducing query-by-attribute capability.

Write and Repair Protocol

Request-log durability is part of completing a pipeline transition. The source write succeeds first, the required log record is retained second, and a dependent handoff is published only after log creation or identical-existing reconciliation succeeds.

For a Request transition, the controller:

  1. builds an immutable updated copy for the state transition;
  2. computes newVersion = oldVersion + 1;
  3. calls RequestStore.Update(updated, oldVersion, newVersion);
  4. assigns the in-memory version only after the store succeeds;
  5. constructs the log record from the durable Request and bounded context still owned by that stage, then calls Materializer.PersistLog;
  6. publishes the downstream handoff.

Request creation, Build changes, and fact creation use the same source-write, log-write, dependent-publish ordering. A request-log outage can leave a source update visible, but it cannot allow dependent processing to move past an unrecorded transition.

The project_facts_recorded and record_abandoned events project completion of project-result production into RequestSummary.ProjectResultsCompletedAtMs. Abandonment ends production without claiming that all project facts were established or changing recorded project counts. The first projected completion timestamp is immutable, including when both events are retained; successful recording still projects its project counts. Completion is independent of lifecycle state and request version; state updates preserve it, and optimistic locking retries concurrent projection updates. Completion requires an existing lifecycle summary rather than fabricating state from the event. A failed projection fails persistence, and redelivery reconciles the retained occurrence before retrying the summary update. Other non-state events do not update the summary.

Controller ownership

Owner Source operation and retained log Retry repair
Ingest Create accepted Request, then retain accepted. An existing Request ensures accepted before process publication.
Process CAS to superseded or processing, then retain that state. An existing state is reconstructed from Request context before ack or build publication.
Build Create Build after runner acceptance, then retain build_triggered. An identical existing Build ensures the event before buildsignal publication.
Buildsignal Persist terminal Build and retain build_finished; CAS the Request outcome and retain its terminal state. Existing terminal Build and Request outcome each ensure their own entry before record publication.
Record Create or verify the whole-repository fact, then retain validation_fact_recorded; after recording project facts, retain one project_facts_recorded event. An identical whole-repository fact and project-fact batch ensure their events before bookmark or promotion work.
Record DLQ Retain record_abandoned and project the end of project-result production; for failed Requests, create or verify the repository fact and retain validation_fact_recorded before publishing its hook. Existing facts remain authoritative; stable log and hook IDs make replay idempotent without resolving projects or replaying promotion.
Build analysis Persist terminal detection results for the recorded winning build, then retain build_analysis_completed before final issue-tracking publication. Repeat detection and issue creation, then reuse the first retained count and timestamp before publication. Polls and terminal passes waiting for the winning outcome add no event.
Build analysis DLQ Retain build_analysis_abandoned without rerunning detection or changing validation state. One stable event per build deduplicates recovery deliveries; only state reads and history retention retry.
Issue tracking DLQ Protect the current cursor from delayed writes, retain its issue history and issue_tracking_abandoned, then mark the pass abandoned. A stable hash of the notification ID deduplicates the event. Recovery retains accepted issue changes and never processes the remaining range.
Issue tracking Finalize the durable checkpoint, then retain issues_reconciled. Reconstruct history from the finalized checkpoint before it can be replaced by another pass.
Fix Retain each executor disposition, then retain its submitted or skipped event. Restore every durable disposition event before skipping that item for that executor.
Fix DLQ Repair submitted or skipped history from current durable state, or retain fix_submission_abandoned when acceptance is unknown. Only read sources and retain history; never call the executor or change issue state.
Reconciler CAS an unrecoverable non-terminal Request to failed, then retain failed. An existing terminal Request is repaired from its persisted outcome without relabeling it.

Build running and unchanged polls create no entry. A failed runner trigger that creates no Build creates no event. Cancelled and superseded requests create no validation fact.

No controller infers an outcome reason from RequestStateFailed alone or records DLQ as the domain reason. Record DLQ uses that persisted state to recover a missing broken repository fact; it does not infer project verdicts.

Repair trigger requirement

Every writer retains a retry trigger until the request-log write succeeds. A primary queue stage retries its delivery under its primary policy. A DLQ reconciler retains terminal/history outputs or explicit abandonment within its stage-specific limits; it does not resume the original workflow. The record DLQ may publish the failed Request's terminal repository-fact hook described above, but DLQ retry does not restart builds, detection, remaining issue-tracking ranges, or fix submission. A log or publish failure after a source transition never forces a different outcome.

Stages without a suitable DLQ reconciler add one or use a subscription policy that cannot discard the only repair trigger. Ingest relies on caller retry of the same (queue, URI) to complete an accepted Request whose log or process publication failed.

This is rollout work, not deferred cleanup: a mandatory request log without a durable repair trigger could turn a store outage into silently stranded pipeline work.

Rollout and Retention

Rollout therefore:

  1. deploys request-log storage, the materializer, and readers;
  2. enables writers and verifies every repair path stage by stage;
  3. enables the public API after every writer and repair path is active.

The service does not fabricate prior state transitions or emit a synthetic snapshot. Requests accepted before request-log writers are active are not exposed through the history API and are not backfilled.

Initial request-log retention matches Request and request-URI mapping retention. The API RFC owns the resulting selector guarantees.

Failure Semantics

  • Request-log store unavailability is retryable and prevents dependent publication.
  • Invalid entry construction is non-retryable.
  • ErrAlreadyExists followed by identical content is success.
  • ErrAlreadyExists followed by conflicting content is a non-overwriting consistency error and operator alert.
  • A log-write failure cannot roll back its source write; retry or reconciliation must converge the missing record.

Identifiers, outcome reasons, and reasonable per-request build counts have explicit limits. Raw dependency responses, credentials, tokens, stack traces, and unrestricted error messages are never retained.

Verification and Observability

Contract tests cover required-field validation, stable IDs, idempotent create/reload, conflict detection, queue binding, deterministic ordering, equal-timestamp tie breaking, and empty histories.

Writer tests cover source success followed by log failure, redelivery with the log record absent or present, CAS loss, conflicting terminal writers, downstream publish failure, first-insert timestamp reuse, and controller-owned version arithmetic. End-to-end tests cover successful, failed, cancelled, superseded, and fail-closed paths plus idempotent redelivery.

Tests reconstruct the latest Request state from state entries by request version and compare it with RequestStore.Get. They separately verify that only a durable fact produces green or broken.

The materializer reports create, identical-existing, conflict, validation failure, and storage failure counters tagged only by bounded state or event plus tags carried by the context. IDs, URIs, and errors remain structured log fields rather than metric tags. Alerts cover sustained repair gaps and content conflicts.

Alternatives Considered

Record request state changes only

Rejected because build activity occurs while the request remains processing, and terminal Request outcome precedes the fact establishing green or broken.

Record every correlated durable operation

Rejected because bookmarks, promotion, project facts, hooks, and operational bookkeeping have different owners and audiences. Correlation alone does not make an operation part of the request lifecycle log.

Use a generic versioned envelope

Rejected because explicit request states and event fields are simpler to inspect than a generic envelope. A future migration should be justified by actual vocabulary growth.

Copy SubmitQueue's log topic

Rejected while one Stovepipe service owns every writer and the queue-scoped storage aggregate. A topic becomes useful if ownership later crosses a service boundary.