cachedFetcher loads all tokens into memory causing excessive memory allocation

The cachedFetcher (used by Mixed strategy) loads ALL tokens from the database into memory, regardless of which user's tokens are actually needed. This causes significant memory allocation visible in pprof.

pprof Output

56.50MB   100% |   github.com/hyperledger-labs/fabric-token-sdk/token/services/selector/sherdlock.(*cachedFetcher).update
30MB 13.90% 13.90%    56.50MB 26.19%                | github.com/hyperledger-labs/fabric-smart-client/platform/view/services/storage/driver/sql/common.(*rowIterator[go.shape.struct { Id github.com/hyperledger-labs/fabric-token-sdk/token/token.ID; WalletID string; Type github.com/hyperledger-labs/fabric-token-sdk/token/token.Type; Quantity string }]).Next
26.50MB 46.90% |   database/sql.(*Rows).Next

Root Cause

token/services/selector/sherdlock/fetcher.go:184

it, err := f.tokenDB.SpendableTokensIteratorBy(ctx, "", "")

Empty string parameters result in SQL query without wallet/type filters:

SELECT * FROM tokens WHERE spendable = true
-- Returns ALL tokens from ALL users

Impact

100K tokens: 56.50MB allocated per refresh
1M tokens: 565MB allocated per refresh
Refreshes every 1 second OR every 5 queries
Memory grows unbounded with database size
High GC pressure and system slowdown

Configuration

Set in token/sdk/dig/sdk.go:147:
sherdlock.NewFetcherProvider(..., sherdlock.Mixed)

Mixed Strategy: Combines cachedFetcher (eager) + lazyFetcher (fallback)

Tries cachedFetcher first (loads ALL tokens)
Falls back to lazyFetcher if cache empty

The current strategy of fetching spendable tokens using cachedFetcher cache is fast however it is not scalable since the cache has no size limits or eviction policy, and loads all tokens even though only one user's tokens are needed per transfer request.

One solution is to replace the current cache unlimited implementation (map) to LRU cache.

Memory Analysis Summary

Token Memory: 289 bytes/token (worst case)

Breakdown

Token Data (156 bytes)

// UnspentTokenInWallet models an unspent token owner solely by a given wallet
type UnspentTokenInWallet struct {
// Id is used to uniquely identify the token in the ledger
Id ID
// WalletID is the ID of the wallet owning this token
WalletID string
// Type is the type of the token
Type Type
// Quantity represents the number of units of Type that this unspent token holds.
Quantity string
}

Field          | Size
---------------|-------
Id.TxId        | 80 bytes (16 header + 64 data)
Id.Index       | 8 bytes
WalletID       | 29 bytes (16 header + 13 data)
Type           | 19 bytes (16 header + 3 data)
Quantity       | 20 bytes (16 header + 4 data)
---------------|-------
TOTAL          | 156 bytes

Looking in actual data in database:

bankaDB=# SELECT 
    round(sum(pg_column_size(t)) / 1024.0, 2) AS total_kb,
    round(sum(pg_column_size(t)) / (1024.0*1024.0), 2) AS total_mb
FROM (
    SELECT token_type, quantity, tx_id, owner_wallet_id
    FROM fsc_mytopos__arma__tokenchaincode_tokens
    WHERE spendable = true
) t;
 total_kb | total_mb 
----------+----------
 76956.75 |    75.15
(1 row)

bankaDB=# SELECT COUNT(DISTINCT owner_wallet_id) AS distinct_wallets
FROM fsc_mytopos__arma__tokenchaincode_tokens;
 distinct_wallets 
------------------
                2
(1 row)
bankaDB=# SELECT 
    COUNT(*) FILTER (WHERE spendable = true) AS spendable_count,
    COUNT(*) AS total_count
FROM fsc_mytopos__arma__tokenchaincode_tokens;
 spendable_count | total_count 
-----------------+-------------
          723682 |      723682
(1 row)

The size of each row is:
size per row=723682/76956.75 ≈ 0.1063 KB per row
0.1063 KB×1024≈108.9 bytes per row in storage which is 156 bytes in go.

Go Runtime Overhead (133 bytes)

Component              | Size
-----------------------|-------
Pointer storage        | 8 bytes
Slice growth overhead  | 53 bytes
Map load factor        | 20 bytes
Memory allocator       | 18 bytes
GC metadata            | 16 bytes
Memory alignment       | 18 bytes
-----------------------|-------
TOTAL OVERHEAD         | 133 bytes

Total: 156 + 133 = 289 bytes per token

Scale Impact

Tokens	Memory
200K	~58 MB
500K	~145 MB
1M	289 MB
1B	~289 GB

Refresh Performance (Measured)

Pure Cache Update Time (No Database)

Simulation Code

This test measures ONLY the cache update operations, excluding database I/O:

// Pre-create tokens in memory (simulating database results)
tokens := make([]*token2.UnspentTokenInWallet, count)
for i := 0; i < count; i++ {
    tokens[i] = &token2.UnspentTokenInWallet{
        Id: token2.ID{
            TxId:  fmt.Sprintf("tx_%064d", i),
            Index: uint64(i % 10),
        },
        WalletID: fmt.Sprintf("wallet_%d", i%100),
        Type:     "USD",
        Quantity: fmt.Sprintf("0x%x", 100+i%1000),
    }
}

// Measure ONLY cache operations (identical to fetcher.go:191-200)
runtime.GC()
startTime := time.Now()

// Build map (same as fetcher.go:191-196)
m := map[string][]*token2.UnspentTokenInWallet{}
for _, t := range tokens {
    key := tokenKey(t.WalletID, t.Type)
    m[key] = append(m[key], t)
}

// Build iterators (same as fetcher.go:197-200)
its := map[string]permutatableIterator[*token2.UnspentTokenInWallet]{}
for key, toks := range m {
    its[key] = iterators.Slice(toks)
}

updateDuration := time.Since(startTime)

Pure Cache Update Results without Database Overhead

Token Count	Update Time	Tokens/sec	Projected 1B
10K	2.67ms	3.75M	4m 27s
100K	24.91ms	4.01M	4m 9s
1M	271.75ms	3.68M	4m 32s

The question now how much token per node we need to support?

Estimate Database Overhead

bankaDB=# EXPLAIN ANALYZE
SELECT token_type, quantity, tx_id, owner_wallet_id
FROM fsc_mytopos__arma__tokenchaincode_tokens
WHERE spendable = true;
                                                                     QUERY PLAN                                                                     
----------------------------------------------------------------------------------------------------------------------------------------------------
 Seq Scan on fsc_mytopos__arma__tokenchaincode_tokens  (cost=0.00..145026.95 rows=700595 width=84) (actual time=4.604..614.442 rows=723682 loops=1)
   Filter: spendable
 Planning Time: 0.040 ms
 JIT:
   Functions: 4
   Options: Inlining false, Optimization false, Expressions true, Deforming true
   Timing: Generation 0.406 ms, Inlining 0.000 ms, Optimization 0.345 ms, Emission 4.247 ms, Total 4.997 ms
 Execution Time: 635.819 ms
(8 rows)

Number of rows:
(actual time=4.604..614.442 rows=723682 loops=1)

Total execution time
Execution Time: 635.819 ms

Average time per row
635.819 ms / 723682 ≈ 0.000878ms = 0.879μs per row

Complete Operation Time: Database SELECT + Cache Update

Performance Metrics

Token Count	DB SELECT	Cache Update	Total (ms)	Total (sec)	Tokens/sec	Projected 1B
10K	8.78 ms	2.67 ms	11.45 ms	0.011 s	873K	19m 5s
100K	87.8 ms	24.91 ms	112.71 ms	0.113 s	887K	18m 47s
1M	878 ms	271.75 ms	1,149.75 ms	1.150 s	870K	19m 9s
1B	878,000 ms	271,750 ms	1,149,750 ms	1,149.75 s	870K	~19m 10s

The current cachedFetcher implementation uses an unbounded map that loads ALL tokens from the database into memory, causing excessive memory allocation visible in pprof. This leads to:

Unbounded memory growth as token count increases
Memory pressure on nodes with many tokens
Potential OOM (Out of Memory) issues in production

The current proposed solution is to use Ristretto Cache with LRU Eviction.
Replaced the unbounded map with Ristretto cache providing:

LRU (Least Recently Used) eviction - automatically removes least-used entries when limit reached
Configurable memory limits - prevents unbounded growth
Default: 100M entries - reasonable limit for most deployments
Configuration
token:
selector:
fetcherCacheSize: 100000000 # Max cache entries (0 = use default 100M)
fetcherCacheRefresh: 1s # How often to refresh cache
fetcherCacheMaxQueries: 5 # Queries before triggering refresh

Current Update Strategy: Eager Loading

How It Works
The cache uses eager loading with two update triggers:

Soft Update (Background, Non-blocking)
Trigger: After fetcherCacheMaxQueries queries (default: 5)

Starts background goroutine to refresh cache, fetches ALL tokens from DB: SpendableTokensIteratorBy(ctx, "", "")
User request returns immediately with current cached data
Update happens asynchronously

Hard Update (Blocking)
Trigger: When cache age exceeds fetcherCacheRefresh (default: 1 second)

Blocks the requesting thread
Fetches ALL tokens from DB: SpendableTokensIteratorBy(ctx, "", "")
Clears cache and repopulates with fresh data

Cache Miss Behavior
When a requested key is not in cache:

Returns empty iterator (no tokens found)
Falls back to lazy fetcher which queries DB directly for that specific wallet/currency

A better strategy is to keep following variables per cache key (type/owner combination) and lock/fetch only the 'expired' entry
This can be addressed in the future under: #633

Navigation Menu

Repository navigation

cachedFetcher loads all tokens into memory causing excessive memory allocation #1310

Memory Analysis Summary

Token Memory: 289 bytes/token (worst case)

Breakdown

Token Data (156 bytes)

Go Runtime Overhead (133 bytes)

Scale Impact

Refresh Performance (Measured)

Pure Cache Update Time (No Database)

Simulation Code

Pure Cache Update Results without Database Overhead

Estimate Database Overhead

Complete Operation Time: Database SELECT + Cache Update

Performance Metrics

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

cachedFetcher loads all tokens into memory causing excessive memory allocation #1310

Description

Activity

AkramBitar commented on Feb 4, 2026

Memory Analysis Summary

Token Memory: 289 bytes/token (worst case)

Breakdown

Token Data (156 bytes)

Go Runtime Overhead (133 bytes)

Scale Impact

Refresh Performance (Measured)

Pure Cache Update Time (No Database)

Simulation Code

Pure Cache Update Results without Database Overhead

Estimate Database Overhead

Complete Operation Time: Database SELECT + Cache Update

Performance Metrics

AkramBitar commented on Feb 14, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions