feat: implement structured GCP Cloud Logging with explicit severity across all apps

[max-ostapenko]

Summary

Adds structured GCP Cloud Logging to all three apps in apps/ (report-api, dataform-service, bigquery-export). Every entry now has a severity, errors carry their stack trace in the field Error Reporting reads, and app logs are linked to the Cloud Run request log they belong to.

Problem

• Logs were plain text (console.error(Error ${operation}:, error) and similar), so Cloud Logging stored them as DEFAULT severity, not ERROR or WARNING.
• Cloud Run split multi-line stack traces into one entry per line.
• Some catch blocks logged no details, or nothing at all.
• App logs had no trace ID, so an error couldn't be matched to the request that caused it (URL, parameters, user agent, latency).

Changes

1. Shared logger (@httparchive/shared, packages/shared/src/logger.js)
   • Writes each entry as one line of JSON with severity and message, plus any metadata passed in. Metadata can't overwrite message, severity or the trace fields.
   • When an Error is passed, message becomes "<message>: <error.message>", so the reason shows on the Logs Explorer summary line. The error is also kept in error, code, details, reason, statusCode, errors[] (BigQuery/GCS sub-errors with reason and location) and cause (nested causes included).
   • Stacks go into stack_trace only on ERROR entries; lower severities put them in stack. Error Reporting reads stack_trace regardless of severity, so this keeps warnings out of it.
   • Trace correlation: withTraceContext(req, fn) uses AsyncLocalStorage to add logging.googleapis.com/trace, spanId and trace_sampled to every entry logged while handling a request, including after awaits. It reads the trace from traceparent, falling back to X-Cloud-Trace-Context, the same order @google-cloud/logging uses, without that dependency. The project ID comes from GOOGLE_CLOUD_PROJECT or PROJECT; without one, the trace fields are left out.
   • Never throws: BigInt values become strings, circular references become [Circular], and there's a minimal fallback entry if serialization still fails.
   • registerProcessLogging({ exitOnUnhandledRejection }) logs Node runtime warnings and unhandled rejections as structured entries.
   • The shared BigQueryExport helper, used by dataform-service and bigquery-export, logs through the logger too, with jobId, and rowCount on completion.
2. apps/report-api
   • All logging goes through the shared logger, and the request handler is wrapped in withTraceContext.
   • handleControllerError and the top-level catch log 4xx as WARNING and 5xx as ERROR. jsonPayload.statusCode is included when the error has one.
   • The hand-written warning and rejection handlers are replaced with registerProcessLogging().
3. apps/dataform-service
   • All console.* calls are replaced with the logger, the handler is wrapped in withTraceContext, and registerProcessLogging() is added.
   • Bad requests (400/404 trigger errors, payload and config parse failures, unknown destination) log as WARNING. Export failures log as ERROR. Response codes are unchanged.
   • handleExport now logs the error when an export fails, and returns error.message in errorMessage. Before, the Error was serialized as {}.
   • Fix: the invalid Pub/Sub message is logged as pubsubMessage. It used to be dropped.
4. apps/bigquery-export
   • All console.* calls are replaced with the logger. High memory usage and retried writes log as WARNING. Pagination, streaming, export failures and writes that fail after all retries log as ERROR.
   • Failed writes log documentPath, operationType, code and failedAttempts instead of a stack for every retry.
   • Fix: write failures now fail the job. BulkWriter writes that fail for good are counted, and the delete or write phase throws after flushing. Before, those write promises were never handled, so the job logged "Export finished successfully" and exited 0. A failed delete now stops the export before any new rows are written.
   • Exits with code 1 on an unhandled rejection (registerProcessLogging({ exitOnUnhandledRejection: true })).
5. Dockerfiles: NODE_NO_WARNINGS=1, so runtime warnings are logged once through the structured handler and not also as unstructured DEFAULT lines.
6. Terraform
   • Alert "Dataform Service Error": severity=ERROR → (severity>=ERROR OR (severity=WARNING AND log_id("run.googleapis.com/stderr"))). App warnings, such as bad requests to dataform-service, now page too. WARNING request logs are excluded: in the last 30 days prod had 565 of them, all 403s from scanners probing paths like /.env and /swagger/, which IAM blocks before they reach the app.
   • Alert "Report API Endpoint": adds -(jsonPayload.statusCode < 500), so 4xx app entries don't page. Entries without statusCode still match (runtime warnings, GCS errors, unhandled errors).
   • Alert "BigQuery Export Error": severity=ERROR → severity>=ERROR.
   • dataform-service: sets GOOGLE_CLOUD_PROJECT, which the trace resource name needs. report-api already has PROJECT.

Behavior changes to be aware of

• Saved searches for the old text (❌ Export, ⚠️ High memory, Error fetching dates from GCS:) won't match anymore. Messages are now in jsonPayload.message, without emojis or trailing colons.
• The "Dataform Service Error" alert now also fires on app warnings (bad requests, parse failures). In the last 30 days prod had no stderr output from dataform-service or bigquery-export, so current traffic won't add noise.

Verification

• npm test: shared 14/14, report-api 120/120, dataform-service 5/5. npm run lint shows no errors (3 warnings, all from before this PR). terraform fmt is clean.

• Deployed to report-api-dev and tested with real requests:

  Request	App log	Trace matches request log
  MCP tools/call (200)	INFO MCP Tool Call: list_categories	✅
  /v1/versions?technology=<40 values> (400)	WARNING Error fetching versions: Too many technologies specified…, statusCode: 400, stack in stack	✅
  /v1/technologies?technology=<40 values> (500)	ERROR Error fetching technologies: Cannot read properties of null…, with stack_trace	✅

  • Error Reporting shows 1 group, for the 500 only; the 400 warning did not create one.
  • The "Report API Endpoint" filter, run against these dev logs, matches only the 500 (the app entry and the request log).
  • The 500 on /technologies is the bug fixed in fix(report-api): return 400 instead of 500 for invalid technology and date input #156. That fix is in main and has since been merged into this branch; the dev test ran before the merge.