Synchronous (Iter[T]) and asynchronous (AsyncIter[T]) iterators with mandatory disposal.
Both iterator types share four rules:
- Single-consumer: do not call
next()concurrently from multiple threads or tasks. - Disposal required: always call
dispose()when done (usedeferortry/finally). - No concurrent dispose: do not call
dispose()whilenext()is in flight. - After dispose: calling
next()raises.
dispose() is idempotent and sets finished = true. There is deliberately no =destroy auto-dispose: the closures captured in disposeImpl may reference objects the GC collects before the Iter itself.
Construction:
Iter[T].new(genNext, isFinished, dispose, isDisposed, finishOnErr = true)- generic constructor from supplier callbacks (dispose/isDisposedare required and assert when nil)Iter[T].new(a, b, step = 1)/Iter[T].new(slice)- ordinal rangesIter[T].new(items: seq[T])- from a sequenceIter[T].new(iter: iterator)- from a native iteratorIter[T].empty- empty iterator
Access: next(), finished, disposed, finish(), items (iterates to completion), pairs (yields (index, item)).
Combinators (each chains dispose() to the underlying iterator):
map(iter, fn)-Iter[T]toIter[U]mapFilter(iter, mapPredicate)-fn: T -> Option[U], skipsnonefilter(iter, predicate)- implemented viamapFilter
let iter = Iter[int].new(0 ..< 10)
defer: iter.dispose()
let doubled = iter.map((i: int) => i * 2)
defer: doubled.dispose() # chains to iter.dispose()
for n in doubled:
echo nAsyncIter[T] extends Iter[Future[T]]: it inherits the finished-state machinery and the items/pairs iterators, and adds an async dispose contract (await iter.dispose(), wrapped in noCancel so cleanup completes even if the caller is cancelled).
Construction:
AsyncIter[T].new(genNext, isFinished, dispose = default, isDisposed = default, finishOnErr = true)AsyncIter[T].new(a, b, step = 1)/AsyncIter[T].new(slice)- viamapAsyncAsyncIter[T].empty
Access: next() (returns Future[T]), items/pairs (yield futures), finished, disposed.
Combinators:
flatMap(fut, fn)- on single futuresmap(iter: AsyncIter[T], fn)-fn: T -> Future[U], chains disposemapAsync(iter: Iter[T], fn)- lifts a syncIterintoAsyncIter[U]mapFilter(iter, mapPredicate)/filter(iter, predicate)- async predicates; returnFuture[AsyncIter[U]](construction fetches the first matching element)delayBy(iter, duration)- delays each itemcollectAsync(iter)-Future[seq[T]], the async analog of synctoSeq(iter); the first failing item aborts the collection and its error propagates (raised, never returned as a value)
collectAsync is named collectAsync (not collect) so it never collides with the std/sugar collect template when both are in scope.
let it = AsyncIter[int].new(0 ..< 10)
defer: await it.dispose()
let delayed = it.delayBy(50.milliseconds)
let results = await collectAsync(delayed)Streaming a paginated remote record set: lift a sync page-number range into an async fetch, flatten the pages into one record stream, filter out the small records and collect the ids of the rest. dispose() is deferred so cleanup runs even when the pipeline is cancelled.
import std/sugar
import pkg/chronos
import pkg/iter
type
Record = object
id: int
bytes: int
# Simulated paged remote store: each page arrives after a network-like delay.
proc fetchPage(page: int): Future[seq[Record]] {.async.} =
await sleepAsync(1.milliseconds)
@[
Record(id: page * 10 + 0, bytes: page + 1),
Record(id: page * 10 + 1, bytes: page + 2),
]
proc records(): AsyncIter[Record] =
# Fetch pages 0..2 and flatten each page into a single record stream.
flatMap[seq[Record], Record](
mapAsync[int, seq[Record]](Iter[int].new(0 ..< 3), fetchPage),
proc(page: seq[Record]): Future[AsyncIter[Record]] {.async.} =
var i = 0
AsyncIter[Record].new(
proc(): Future[Record] {.async.} =
let item = page[i]
inc i
item,
proc(): bool {.gcsafe, raises: [].} = i >= page.len,
)
)
proc main() {.async.} =
let iter = records()
defer: await iter.dispose() # mandatory; runs even on cancellation
let filtered = await filter[Record](
iter,
proc(r: Record): Future[bool] {.async.} =
r.bytes > 1,
)
let kept = await collectAsync(
map[Record, int](
filtered,
proc(r: Record): Future[int] {.async.} =
r.id,
)
)
doAssert kept == @[1, 10, 11, 20, 21]
when isMainModule:
waitFor main()Notes:
-
Explicit type arguments: combinators take them (
mapAsync[int, seq[Record]]) - theFunction[T, Future[U]]callback type does not reliably inferU. -
filter/mapFilterare async constructors: they returnFuture[AsyncIter[T]](construction fetches the first matching element), so await them before chaining. -
Error model: the first failing item aborts
collectAsyncwith anIteratorErrorwhoseparentcarries the wrapped cause.try: let kept = await collectAsync(iter) except IteratorError as err: echo err.parent.msg
-
flatMapneeds non-empty inner iterators: an empty inner iterator exhausts the source withIteratorError.
nimble install https://github.com/durability-labs/iter@#mainRequires Nim >= 2.0.14, chronos 4.0.x, questionable 0.10.x. Tests: nimble test.