Skip to content

fix(blockfrost): bound the query time of each Blockfrost request - #1303

Open
ducpm2303 wants to merge 3 commits into
mainfrom
fix/1207-blockfrost-query-timeout
Open

ducpm2303 wants to merge 3 commits into
mainfrom
fix/1207-blockfrost-query-timeout

Conversation

@ducpm2303

Copy link
Copy Markdown
Contributor

Fixes #1207

Problem

Apart from the address module's own 15 s queryTimeout, no Blockfrost read query had a time limit. Controllers are synchronous, so a client that times out or disconnects does not stop its query. PostgreSQL kept running it, with its parallel workers, holding a pool connection until it finished. Clients that retry multiply the work.

Measured on main, with the client giving up after 1–2 s:

  • preprod: /assets and /addresses/{addr}/utxos were still running 7.2 s after the request.
  • preview: 8 abandoned /assets/{unit}/addresses requests left 15 backends (8 + parallel workers) running 1.3 s later.

Change

Per-request query budget. Each request handled by a Blockfrost controller gets a budget: store.extensions.blockfrost.query-timeout, default 30s (0 disables it). Every statement gets the time left in the budget as its JDBC query timeout, so the request is bounded as a whole, not per statement.

  • jOOQ (all BF*StorageReaderImpl): an ExecuteListener registered through an ExecuteListenerProvider bean sets the timeout. A smaller explicit queryTimeout(..), such as address's 15 s, is kept. No reader changed.
  • Spring Data repositories used by the Blockfrost services (TransactionStorageReader, EpochStakeStorageReader, epoch services, …): each call runs in a read-only transaction whose timeout is the remaining budget, and Hibernate applies it to the statement. This is added as an interceptor on the repository proxy, just before its own TransactionInterceptor.
  • Scope: the budget is a ThreadLocal set by a HandlerInterceptor, only for handlers in com.bloxbean.cardano.yaci.store.blockfrost. Indexer threads never have one, so sync, rollback and the other APIs are unaffected.

Response.

Wiring. BFCommonConfiguration is now imported by BFAutoConfiguration. Before this, package blockfrost.common, which holds the error advices, was only scanned through BFPoolsConfiguration, so disabling the pools module silently removed them.

Docs. blockfrost/configuration: the new property, the 504, and an optional server-side statement_timeout for read-only instances. That setup needs a separate Flyway connection, because a SET in spring.flyway.init-sqls stays on the pooled connections.

Known limit. JDBC timeouts are whole seconds, so a request can overrun its budget by up to about 1 s.

Tests

Unit tests (blockfrost-common, JUnit 5 + AssertJ):

  • BFQueryTimeoutExecuteListenerTest (H2):
    • a ~10¹⁰-row query is cancelled at 1.0 s on a 1 s budget;
    • statements outside a budget keep timeout 0;
    • a smaller explicit timeout is kept;
    • an exhausted budget fails before running the statement.
  • BFRepositoryTimeoutInterceptorTest: read-only transaction with the remaining time; pass-through without a budget; placed before the TransactionInterceptor.
  • BFQueryTimeoutWebTest: budget only for Blockfrost handlers, cleared after the request; a swallowed timeout becomes 504 instead of an empty 200 or a 404.
  • BFExceptionHandlerTest: QueryTimeoutException → 504; PostgreSQL 57014 translated as a resource failure → 504; other DB errors → 500.
  • BFQueryTimeoutsTest, BFQueryTimeoutContextTest.

End to end on preprod and preview. Each DB was served by main @ a65a828a8 (before) and this branch (after), with a small budget so the local data is slow enough.

preprod (2.75M tx) preview (889k tx)
Heavy routes, after (/assets, /assets/{u}/addresses, /assets/{u}/txs, /addresses/{a}/utxos, /addresses/{a}/total) 4 × 504 in 2.1–2.7 s on a 2 s budget 32 × 504 in 1.0–1.9 s on a 1 s budget (8 in parallel per route)
Left running after the response none none
PostgreSQL canceling statement due to user request 4 32
JPA path: /txs/{h}/cbor behind an 8 s table lock before 200 after 7.1 s; after 504 at 3.05 s (Hibernate SQL cancelled) before 7.3 s; after 504 at 3.16 s
Sync during the test +16,517 blocks, to epoch 168, no gaps genesis → epoch 60 synced by this branch, then +8,493 blocks, to epoch 62, no gaps
Light routes, default 30 s (/, /blocks/{n}, /blocks/{n}/txs, /txs/{h}, /txs/{h}/utxos, /txs/{h}/metadata, /assets/{u}, /scripts/{h}, /metadata/txs/labels/{l}) 200, same body (except confirmations) 200, same body (except confirmations)

Also checked on preprod:

  • With query-timeout=0, the address module's own 15 s timeout on /addresses/{a}/utxos now answers 504 (500 on main).
  • The documented read-only setup (statement_timeout=2s in the URL, separate Flyway connection) starts normally and answers 504, with PostgreSQL logging canceling statement due to statement timeout.

Blockfrost read queries had no time limit except in the address module, so a
query kept running on the database after the client gave up.

Each request handled by a Blockfrost controller now gets a query time budget
(store.extensions.blockfrost.query-timeout, default 30s, 0 disables). A jOOQ
ExecuteListener sets every statement's JDBC query timeout to the time left,
keeping a smaller explicit queryTimeout. Spring Data repository calls made by
the request run in a read-only transaction with the same remaining time.
Threads without a budget, including all indexer threads, are not affected.

A cancelled query answers 504 instead of 500, recognised by SQLState 57014 as
Spring maps the PostgreSQL cancel to a resource failure. If a caught timeout
would still produce a 200 or 404, the response is replaced with the 504.

BFCommonConfiguration is imported by the auto-configuration, so the common
advices no longer depend on the pools module scanning their package.

Fixes #1207
@sonarqubecloud

Copy link
Copy Markdown

Copy link
Copy Markdown

Initial automated review · Aster

Hi @ducpm2303, thanks for the PR! 🙏 This is an automatic first check by an AI reviewer. It does not replace maintainer review and may be wrong.

Reviewed commit: 148c9b6a5478b97e603743d562c6eb61966e096a

No issues found in this first check.

Not tested: Tests and database scenarios were not run locally; build CI was still running when checked.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Blockfrost API: most read queries have no statement timeout, so abandoned requests keep running on the database

2 participants