Repository navigation
feat: add offline mode with local SQLite storage and sync - #79
Draft
tejapulagam wants to merge 1 commit into
Draft
tejapulagam wants to merge 1 commit into
tejapulagam wants to merge 1 commit into
Conversation
Adds a local-only mode that writes all experiment data (metrics,
metadata, artifacts) to a SQLite database instead of making network
calls. Users can later upload the offline data to the cloud using
the new sync() function.
API:
exp = litlogger.init(name='my-run', mode='offline')
exp['loss'].append(0.5)
exp.finalize()
litlogger.sync('./lightning_logs/my-run')
Review fixes applied:
1. Batched sync — sync() now chunks metrics via _send_metrics_batched()
mirroring _BackgroundThread._send_metrics, respecting max_batch_size
and rate_limiting_interval to avoid OOM and API limits on large
experiments.
2. Cached shim API properties — metrics_api, media_api, artifacts_api,
and teamspace are now cached in __init__ instead of allocating new
objects on every property access.
3. End-to-end sync test — TestSyncEndToEnd.test_sync_replays_all_data
writes offline data through the full Experiment stack, then syncs
with a mocked ExperimentSession and verifies metadata, metrics, and
finalization. test_sync_batches_large_experiments verifies chunking.
4. Artifact restoration on resume — _rebuild_state_offline() now
reconstructs _static_files and file series from the artifacts table,
matching the online mode's File._restore_all() behavior.
5. Defensive _offline_session access — all accesses to self._offline_session
now check for None and raise RuntimeError with a clear message.
6. Type annotations on shim classes — all shim API methods now have
explicit parameter types matching the real API interfaces instead
of **kwargs: Any. Added __slots__ to shim classes.
Minor fixes:
- _OfflineTeamspace.owner is now a proper _OfflineOwner instance with
name and id attributes (not a bare class).
- Removed unused json import.
- Schema uses REAL for timestamps (Unix epoch) instead of TEXT.
- Added _db_lock for thread-safe SQLite access.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add offline mode with local SQLite storage and sync
Adds mode="offline" to litlogger.init() and Experiment, enabling fully local experiment tracking with zero network calls. All data is stored in a SQLite database (offline.db) inside the experiment's log directory. When connectivity is available, litlogger.sync() replays the stored data to the cloud.
Usage:
OfflineSession, drop-in replacement for ExperimentSession backed by SQLite, with shim API objects so existing primitives (Metric, Metadata, File) work unchanged
sync() reads the offline DB and replays metrics (batched to respect API limits), metadata, and artifacts through a normal online session
Full support for experiment resume from the offline DB (metrics, metadata, and artifacts)