Skip to content

feat: add snapshot-consistent query service - #738

Draft
JingsongLi wants to merge 1 commit into
apache:mainfrom
JingsongLi:codex/query-service
Draft

feat: add snapshot-consistent query service#738
JingsongLi wants to merge 1 commit into
apache:mainfrom
JingsongLi:codex/query-service

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Aug 23, 2026

Copy link
Copy Markdown
Contributor

Purpose

Add a production-oriented single-node point-query service for Paimon tables. The first capability resolves snapshot-consistent Blob descriptors by database, table, and key fields, while the crate and process boundaries remain generic enough to add projected non-Blob fields later.

The required Paimon core primitives were reviewed and merged separately in #739. This branch is rebased onto that merge and contains only the query-service changes.

Linked issue: none yet.

Brief change log

  • Add paimon-query-service with typed key normalization, policy validation, bounded scan planning, strict snapshot selection, snapshot-pinned caching, and singleflight miss coalescing.
  • Add paimon-query-service-server with bearer-token authorization, explicit anonymous opt-in, request and connection resource limits, graceful H1/H2 draining, JSON errors, readiness checks, metrics, and non-blocking structured logs.
  • Keep the initial Blob HTTP API under /api/blob/v1; use generic query-service crate, binary, configuration, and metric naming for future point-query capabilities.

Tests

  • cargo test -p paimon-query-service --lib --test lookup (25 passed)
  • cargo test -p paimon-query-service-server --lib (25 passed)
  • cargo clippy -p paimon-query-service -p paimon-query-service-server --all-targets -- -D warnings
  • cargo fmt --all -- --check
  • PYTHONDONTWRITEBYTECODE=1 python3 scripts/dependencies.py verify
  • cargo deny --locked --all-features check --warn unmaintained advisories licenses (passes with the existing paste 1.0.15 unmaintained warning)

A local full-workspace run reached the existing DataFusion integration suite but seven fixture-dependent tests could not find default.partitioned_log_table or default.multi_partitioned_log_table; the Paimon core suite completed before those fixture failures.

API and Format

  • Adds the versioned /api/blob/v1/databases/{database}/tables/{table}/descriptors:batchGet HTTP endpoint and operational endpoints.
  • Adds public query-service Rust types.
  • No storage or serialization format change.

Documentation

Adds crates/query-service-server/README.md with configuration, authentication, resource limits, API examples, metrics, and operational behavior.

@JingsongLi
JingsongLi force-pushed the codex/query-service branch from eb153fc to c9a7209 Compare August 23, 2026 10:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant