feat: faucet address-cap race, retention, logging, tests and entity tidy-ups (follow-ups to #829)

README.md

@@ -114,10 +114,21 @@ some tokens on the testnet for making the process simpler.
 7. `FAUCET_SUBSCRIPTION_CACHE_SECONDS`: How long `POST /account/faucet` caches a customer's Stripe subscription status and plan, so API-key use does not call Stripe on every request. The stored subscription status is still checked on every request, so a cancellation takes effect immediately; a plan change can take this long to be seen. `0` disables the cache. (Default: 60)
 8. `FAUCET_MIN_INTERVAL_SECONDS`: Minimum gap between two `POST /account/faucet` requests from the same customer, so a script cannot hammer the endpoint in a tight loop. A request made too soon returns `429` with a `Retry-After` header and does not reach Stripe, the RPC node or the faucet. `0` disables it. Does not apply to the automatic top-up at account creation. (Default: 10)
 9. `FAUCET_PENDING_TIMEOUT_SECONDS`: How long a faucet quota reservation that has not been confirmed counts towards the quota before it is treated as abandoned (for example because the process died before calling the faucet). Keep it longer than the longest faucet call. A reservation whose faucet call threw (outcome unknown, so tokens may have been sent) always keeps counting. (Default: 600)
-10. `FAUCET_REQUEST_TIMEOUT_SECONDS`: How long to wait for the faucet service to answer before giving up. A `POST /account/faucet` that times out returns `504`; the faucet may still have credited the account, so the quota used is kept. (Default: 30)
-11. `FAUCET_API_KEY`: API key for faucet service authentication
-12. `FAUCET_ACCESS_CLIENT_ID`: Cloudflare Access client ID for accessing the Testnet Faucet API
-13. `FAUCET_ACCESS_CLIENT_SECRET`: Cloudflare Access client secret for accessing the Testnet Faucet API
+10. `FAUCET_BALANCE_SETTLE_SECONDS`: A faucet transfer takes a few blocks to show up in an address's on-chain balance. For this many seconds, requests already made for the same address are subtracted from the room left under `FAUCET_ADDRESS_CAP_CHEQ`, so simultaneous requests cannot each be allowed the full room. (Default: 20)
+11. `FAUCET_REQUEST_RETENTION_MONTHS`: How many calendar months of faucet request history to keep in the `faucetRequest` table. Only the current month is needed to enforce the quota; older rows are kept for support and deleted after this. The cleanup runs inside normal faucet requests, at most once an hour per process (Studio has no scheduler). `0` keeps everything. To clean up by hand: `DELETE FROM "faucetRequest" WHERE "createdAt" < date_trunc('month', now() AT TIME ZONE 'UTC') - interval '13 months';` (Default: 13)
+12. `FAUCET_REQUEST_TIMEOUT_SECONDS`: How long to wait for the faucet service to answer before giving up. A `POST /account/faucet` that times out returns `504`; the faucet may still have credited the account, so the quota used is kept. (Default: 30)
+13. `FAUCET_API_KEY`: API key for faucet service authentication
+14. `FAUCET_ACCESS_CLIENT_ID`: Cloudflare Access client ID for accessing the Testnet Faucet API
+15. `FAUCET_ACCESS_CLIENT_SECRET`: Cloudflare Access client secret for accessing the Testnet Faucet API
+
+`POST /account/faucet` accepts three kinds of credential: a Studio user session (bearer token), an API key (`x-api-key` header) and a machine-to-machine token (bearer token plus a `customer-id` header selecting the customer). All three are checked against the same rule: the scopes they carry must include `request:faucet:testnet`. A user session and an API key get their scopes from the user's LogTo roles, so the scope comes with the Basic and Build plan roles; a machine-to-machine token carries its scopes in the token, so `request:faucet:testnet` must be granted to the M2M application's role in LogTo. The plan check, address ownership, quota and limits apply identically to all three.
+
+The faucet quota, limits and handlers have two Postgres-backed test suites that are skipped unless `FAUCET_TEST_DATABASE_URL` points at a disposable database, so a plain `npm run test:unit` and CI are unaffected. Start a database with `docker run -d --rm --name faucet-test-db -e POSTGRES_PASSWORD=pw -e POSTGRES_DB=studio -p 55432:5432 postgres:16`, then:
+
+- the ledger tests (jest): `FAUCET_TEST_DATABASE_URL=postgres://postgres:pw@localhost:55432/studio npm run test:unit -- faucet-request.service`
+- the handler tests (Node's test runner, which needs Node 20 because the controller's dependencies do not load on newer versions): `FAUCET_TEST_DATABASE_URL=postgres://postgres:pw@localhost:55432/studio npx -y -p node@20 node --import tsx --import ./tests/integration/hooks/register.mjs --test tests/integration/faucet-handler.test.ts`
+
+Faucet decisions are logged as one JSON line each, with an `event` name, so a blocked or refunded request can be traced: `faucet.reserved`, `faucet.completed`, `faucet.released` (the faucet reported a failure, quota given back), `faucet.unknown` (the faucet call threw, quota kept), `faucet.quota_exceeded`, `faucet.too_frequent`, `faucet.address_cap_exceeded`, `faucet.reservations_abandoned`, `faucet.bootstrap_topup_skipped`, `faucet.retention_deleted` and the `*_failed` errors. Rejections are warnings and carry the customer, address, amount and the relevant figures (`usedNcheq`, `limitNcheq`, `roomNcheq`, `retryAfterSeconds`); the early checks that run before Stripe are marked `"stage":"precheck"`. A customer's history is also in the `faucetRequest` table: `SELECT "createdAt", address, "amountNcheq", status FROM "faucetRequest" WHERE "customerId" = '<id>' ORDER BY "createdAt" DESC;`.
 
 Studio exposes `POST /account/faucet` for authenticated Studio users on Basic-or-higher subscriptions. The endpoint accepts optional `address` and `amount` fields; `amount` is expressed in CHEQ. If `address` is omitted, Studio uses the customer's testnet payment account. If `amount` is omitted, Studio tops the address up to `FAUCET_ADDRESS_CAP_CHEQ`, or by whatever monthly quota is left if that is lower. Requests that exceed the address cap return a `requestMore.mailtoHref` for contacting `product@cheqd.io`.
 

example.env

@@ -48,6 +48,10 @@ TESTNET_INITIAL_TOPUP_CHEQ="10000"
 # FAUCET_MIN_INTERVAL_SECONDS="10"
 # Seconds an unconfirmed faucet quota reservation keeps counting before it is treated as abandoned
 # FAUCET_PENDING_TIMEOUT_SECONDS="600"
+# Seconds recent requests for an address are subtracted from its room under the address cap
+# FAUCET_BALANCE_SETTLE_SECONDS="20"
+# Months of faucet request history to keep (0 keeps everything)
+# FAUCET_REQUEST_RETENTION_MONTHS="13"
 # Seconds to wait for the faucet service to answer before giving up
 # FAUCET_REQUEST_TIMEOUT_SECONDS="30"
 # FAUCET_URI="https://faucet-api.cheqd.network/credit"

src/controllers/api/account.ts

@@ -15,6 +15,7 @@ import type { SafeAPIResponse } from '../../types/common.js';
 import { CheqdNetwork, checkBalance } from '@cheqd/sdk';
 import {
 	FAUCET_ADDRESS_CAP_CHEQ,
+	FAUCET_BALANCE_SETTLE_SECONDS,
 	FAUCET_MIN_INTERVAL_SECONDS,
 	FAUCET_MONTHLY_LIMIT_CHEQ,
 	FAUCET_SUBSCRIPTION_CACHE_SECONDS,
@@ -60,7 +61,7 @@ import { FaucetRequestService } from '../../services/api/faucet-request.js';
 import {
 	buildFaucetQuotaSummary,
 	creditWithinQuota,
-	getFaucetQuotaWindow,
+	logFaucetEvent,
 	maxRequestableNcheq,
 	remainingQuota,
 	secondsUntilReset,
@@ -587,9 +588,10 @@ export class AccountController {
 
 					if (resp.quotaExceeded) {
 						// Account creation must not fail because the monthly faucet quota is used up
-						console.warn(
-							`Initial testnet top-up skipped, monthly faucet quota exceeded: ${customerEntity.customerId}`
-						);
+						logFaucetEvent('warn', 'faucet.bootstrap_topup_skipped', {
+							customerId: customerEntity.customerId,
+							reason: 'quota_exceeded',
+						});
 					} else if (resp.status !== StatusCodes.OK) {
 						return response.status(StatusCodes.BAD_GATEWAY).json({
 							error: resp.error,
@@ -623,7 +625,7 @@ export class AccountController {
 	 *   post:
 	 *     tags: [Account]
 	 *     summary: Request cheqd testnet CHEQ tokens for a Studio payment account.
-	 *     description: Funds an authenticated Studio user's owned testnet payment account. An address can hold at most FAUCET_ADDRESS_CAP_CHEQ, and a customer can request at most FAUCET_MONTHLY_LIMIT_CHEQ per calendar month (UTC) across all their addresses; both default to 100,000 CHEQ. Without an amount the address is topped up to the cap, or by the quota left if that is lower. Works with a Studio user session or an API key; the response includes the remaining quota.
+	 *     description: Funds an authenticated Studio user's owned testnet payment account. An address can hold at most FAUCET_ADDRESS_CAP_CHEQ, and a customer can request at most FAUCET_MONTHLY_LIMIT_CHEQ per calendar month (UTC) across all their addresses; both default to 100,000 CHEQ. Without an amount the address is topped up to the cap, or by the quota left if that is lower. Authentication: a Studio user session (bearer token), an API key (`x-api-key`) or a machine-to-machine token (bearer token plus a `customer-id` header); whichever is used, the credential must carry the `request:faucet:testnet` scope. The response includes the remaining quota.
 	 *     requestBody:
 	 *       content:
 	 *         application/json:
@@ -684,6 +686,11 @@ export class AccountController {
 				FAUCET_MIN_INTERVAL_SECONDS
 			);
 			if (waitSeconds > 0) {
+				logFaucetEvent('warn', 'faucet.too_frequent', {
+					customerId: customer.customerId,
+					retryAfterSeconds: waitSeconds,
+					stage: 'precheck',
+				});
 				return tooManyFaucetRequests(response, waitSeconds);
 			}
 
@@ -720,10 +727,16 @@ export class AccountController {
 			const quotaLimitNcheq = cheqToNcheq(FAUCET_MONTHLY_LIMIT_CHEQ);
 
 			// Nothing can be requested once the quota is used up, so answer before the (slow) balance query
-			const usedNcheq = await FaucetRequestService.instance.getUsedNcheq(customer);
+			const { usedNcheq, window } = await FaucetRequestService.instance.getQuotaStatus(customer);
 			const quotaRemainingNcheq = remainingQuota(usedNcheq, quotaLimitNcheq);
 			if (quotaRemainingNcheq <= 0n) {
-				const window = getFaucetQuotaWindow();
+				logFaucetEvent('warn', 'faucet.quota_exceeded', {
+					customerId: customer.customerId,
+					address: testnetAccount.address,
+					usedNcheq: usedNcheq.toString(),
+					limitNcheq: quotaLimitNcheq.toString(),
+					stage: 'precheck',
+				});
 				return response
 					.status(StatusCodes.TOO_MANY_REQUESTS)
 					.set('Retry-After', String(secondsUntilReset(window)))
@@ -765,6 +778,13 @@ export class AccountController {
 			}
 
 			if (amountToRequestNcheq > roomUnderCapNcheq) {
+				logFaucetEvent('warn', 'faucet.address_cap_exceeded', {
+					customerId: customer.customerId,
+					address: testnetAccount.address,
+					amountNcheq: amountToRequestNcheq.toString(),
+					roomNcheq: roomUnderCapNcheq.toString(),
+					stage: 'precheck',
+				});
 				return response.status(StatusCodes.BAD_REQUEST).json({
 					error: 'Requested amount exceeds the maximum available top-up for this address.',
 					address: testnetAccount.address,
@@ -790,11 +810,25 @@ export class AccountController {
 					amountNcheq: amountToRequestNcheq,
 					limitNcheq: quotaLimitNcheq,
 					minIntervalSeconds: FAUCET_MIN_INTERVAL_SECONDS,
+					addressCap: {
+						capNcheq: addressCapNcheq,
+						currentBalanceNcheq,
+						settleSeconds: FAUCET_BALANCE_SETTLE_SECONDS,
+					},
 				}
 			);
 			if (credit.outcome === 'too_frequent') {
 				return tooManyFaucetRequests(response, credit.retryAfterSeconds);
 			}
+			if (credit.outcome === 'address_cap_exceeded') {
+				// Requests made moments ago for this address have not reached its balance yet
+				return response.status(StatusCodes.BAD_REQUEST).json({
+					error: 'Requested amount exceeds the maximum available top-up for this address.',
+					address: testnetAccount.address,
+					balance,
+					requestMore: buildRequestMore(customer, testnetAccount.address, balance, requestedAmountCheq),
+				});
+			}
 			if (credit.outcome === 'quota_exceeded') {
 				return response
 					.status(StatusCodes.TOO_MANY_REQUESTS)
@@ -998,14 +1032,12 @@ export class AccountController {
 			if (testnetAddress) {
 				try {
 					const addressCapNcheq = cheqToNcheq(FAUCET_ADDRESS_CAP_CHEQ);
-					const usedNcheq = await FaucetRequestService.instance.getUsedNcheq(response.locals.customer);
+					const { usedNcheq, window } = await FaucetRequestService.instance.getQuotaStatus(
+						response.locals.customer
+					);
 					faucet = {
 						cap: { cheq: ncheqToCheq(addressCapNcheq), ncheq: addressCapNcheq.toString() },
-						quota: buildFaucetQuotaSummary(
-							usedNcheq,
-							cheqToNcheq(FAUCET_MONTHLY_LIMIT_CHEQ),
-							getFaucetQuotaWindow()
-						),
+						quota: buildFaucetQuotaSummary(usedNcheq, cheqToNcheq(FAUCET_MONTHLY_LIMIT_CHEQ), window),
 					};
 				} catch (error) {
 					console.error('getBalances: faucet quota lookup failed:', (error as Error)?.message || error);
@@ -1263,7 +1295,8 @@ async function delegateWithinQuota(
 				quotaExceeded: true,
 			};
 		case 'too_frequent':
-			// not requested here: the bootstrap top-up does not apply a minimum interval
+		case 'address_cap_exceeded':
+			// not requested here: the bootstrap top-up applies neither a minimum interval nor the address cap
 			return { status: StatusCodes.TOO_MANY_REQUESTS, error: 'Too many faucet requests.' };
 		case 'faucet_failed':
 			return { status: credit.status, error: credit.error };

src/database/entities/faucet-request.entity.ts

@@ -1,4 +1,4 @@
-import { BeforeInsert, Check, Column, Entity, Index, JoinColumn, ManyToOne, PrimaryGeneratedColumn } from 'typeorm';
+import { Check, Column, CreateDateColumn, Entity, Index, JoinColumn, ManyToOne, PrimaryGeneratedColumn } from 'typeorm';
 
 import * as dotenv from 'dotenv';
 import { CustomerEntity } from './customer.entity.js';
@@ -13,6 +13,12 @@ dotenv.config();
 export const FAUCET_REQUEST_STATUSES = ['pending', 'completed', 'unknown', 'abandoned'] as const;
 export type FaucetRequestStatus = (typeof FAUCET_REQUEST_STATUSES)[number];
 
+// `bigint` columns come back from the driver as strings; expose them as bigint instead
+const bigintTransformer = {
+	to: (value?: bigint | null) => value?.toString(),
+	from: (value?: string | null) => (value === null || value === undefined ? value : BigInt(value)),
+};
+
 /**
  * One row per testnet faucet request made through `POST /account/faucet` or the account bootstrap top-up.
  * Summed per customer to enforce the monthly faucet quota.
@@ -34,12 +40,13 @@ export class FaucetRequestEntity {
 	})
 	address!: string;
 
-	// ncheq amount, stored as bigint and surfaced as a string by the driver
+	// ncheq amount
 	@Column({
 		type: 'bigint',
 		nullable: false,
+		transformer: bigintTransformer,
 	})
-	amountNcheq!: string;
+	amountNcheq!: bigint;
 
 	@Column({
 		type: 'text',
@@ -48,9 +55,9 @@ export class FaucetRequestEntity {
 	})
 	status!: FaucetRequestStatus;
 
-	@Column({
+	// Set by the database (DEFAULT now()), so every instance agrees on when a request was made
+	@CreateDateColumn({
 		type: 'timestamptz',
-		nullable: false,
 	})
 	createdAt!: Date;
 
@@ -60,16 +67,10 @@ export class FaucetRequestEntity {
 	})
 	completedAt?: Date;
 
-	@BeforeInsert()
-	setCreatedAt() {
-		this.createdAt = new Date();
-	}
-
 	constructor(customer: CustomerEntity, address: string, amountNcheq: bigint) {
 		this.customer = customer;
 		this.address = address;
-		// TypeORM instantiates entities without arguments when building metadata
-		this.amountNcheq = amountNcheq?.toString();
+		this.amountNcheq = amountNcheq;
 		this.status = 'pending';
 	}
 }

src/database/migrations/1791500000002-studio-migrations.ts

@@ -0,0 +1,13 @@
+import { MigrationInterface, QueryRunner } from 'typeorm';
+
+export class StudioMigrations1791500000002 implements MigrationInterface {
+	name = 'StudioMigrations1791500000002';
+
+	public async up(queryRunner: QueryRunner): Promise<void> {
+		await queryRunner.query(`ALTER TABLE "faucetRequest" ALTER COLUMN "createdAt" SET DEFAULT now()`);
+	}
+
+	public async down(queryRunner: QueryRunner): Promise<void> {
+		await queryRunner.query(`ALTER TABLE "faucetRequest" ALTER COLUMN "createdAt" DROP DEFAULT`);
+	}
+}

src/database/types/types.ts

@@ -59,6 +59,7 @@ import { UpdateWriteCursors1762775500000 } from '../migrations/1762775500000-Upd
 import { StudioMigrations1766408271347 } from '../migrations/1766408271347-studio-migrations.js';
 import { StudioMigrations1791500000000 } from '../migrations/1791500000000-studio-migrations.js';
 import { StudioMigrations1791500000001 } from '../migrations/1791500000001-studio-migrations.js';
+import { StudioMigrations1791500000002 } from '../migrations/1791500000002-studio-migrations.js';
 dotenv.config();
 
 const { EXTERNAL_DB_CONNECTION_URL, EXTERNAL_DB_CERT } = process.env;
@@ -161,6 +162,8 @@ export class Postgres implements AbstractDatabase {
 				StudioMigrations1791500000000,
 				// Track whether each faucet request completed, so abandoned reservations can stop counting
 				StudioMigrations1791500000001,
+				// Let the database set faucetRequest.createdAt
+				StudioMigrations1791500000002,
 			],
 			entities: [
 				...Entities,