Home

YaraDB Documentation

📦 Welcome to YaraDB

YaraDB is an intelligent, in-memory-first document database with crash-safe Write-Ahead Logging (WAL) persistence. Built with Python and FastAPI, it delivers lightning-fast performance while ensuring data durability and consistency.

🎯 What is YaraDB?

YaraDB is a lightweight, high-performance document database designed for modern applications that need:

Speed: In-memory operations with O(1) document retrieval
Safety: WAL-based persistence ensures no data loss on crashes
Simplicity: RESTful API with automatic OpenAPI documentation
Flexibility: Schema-free or schema-enforced tables
Reliability: Optimistic concurrency control prevents data races

Perfect for rapid prototyping, microservices, real-time applications, and testing environments.

✨ Key Features

Core Capabilities

Feature	Description
⚡ In-Memory Performance	Lightning-fast O(1) document retrieval by ID with hash-based indexing
💾 WAL Persistence	Crash-safe Write-Ahead Logging with automatic recovery and checkpointing
🔐 Optimistic Concurrency	Version-based conflict detection prevents lost updates
🗑️ Soft Deletes	Archive documents without permanent data loss - easily restore when needed
🔒 Data Integrity	SHA-256 body hashing validates document consistency on every operation
📊 Smart Tables	Create tables with strict schemas, unique constraints, or keep them schema-free
🌐 RESTful API	Clean HTTP interface with automatic OpenAPI/Swagger documentation
🐳 Docker Ready	Official Docker images with health checks and volume persistence

Full API documentation with examples: http://localhost:8000/docs

🚦 Use Cases

1. Rapid Prototyping

Start building immediately without database setup:

# No schema, no migrations, just code
client.create(name="prototype", body={"any": "data"})

2. Real-Time Applications

In-memory speed for live dashboards and analytics:

# Sub-millisecond reads
doc = client.get(user_id)

3. Microservices

Lightweight database for service-local data:

services:
  api:
    image: my-api
  database:
    image: ashfromsky/yaradb:latest

4. Testing

Clean database state for each test:

@pytest.fixture
def db():
    # Fresh database per test
    client.self_destruct(...)
    yield client

5. Session Storage

Fast, persistent session management:

client.create(
    name="session",
    body={"user_id": 123, "expires": "2025-01-20"},
    table_name="sessions"
)

⚡ Performance Characteristics

Read Performance

By ID: O(1) - direct hash lookup
Find/Filter: O(n) - full table scan
Combined Documents: O(1) per source document

Write Performance

Create: ~1-2ms (in-memory + WAL write)
Update: ~1-2ms (in-memory + WAL write)
Archive: ~1ms (in-memory + WAL write)

Storage

Memory: ~500 bytes per document (overhead)
Disk: JSON format, ~1KB per document
WAL: Append-only, grows until checkpoint

Scalability

Documents: Tested with 100K+ documents
Concurrent Users: Single-worker design (use reverse proxy for scaling)
Rate Limits: 10 req/min per endpoint (configurable)

🔄 Backup & Recovery

Manual Backup

# Stop server
docker stop yaradb_server
Backup data files
cp yaradb_data/yaradb_storage.json backup/
cp yaradb_data/yaradb_wal backup/
Restart server
docker start yaradb_server

Automatic Checkpoints

YaraDB automatically checkpoints on shutdown:

Saves all data to yaradb_storage.json
Clears WAL file
Next startup loads snapshot directly

Disaster Recovery

If both files are corrupted:

Delete yaradb_storage.json and yaradb_wal
Restart server (creates empty database)
Restore from your external backups

🐛 Troubleshooting

Server won't start

Check Docker logs:

docker logs yaradb_server

Common issues:

Port 8000 already in use: Change port mapping
Permission denied: Check volume mount permissions
Missing files: Ensure DATA_DIR is writable

Documents not persisting

Verify WAL writes:

# Check WAL file exists and grows
ls -lh yaradb_data/yaradb_wal

Force checkpoint:

# Restart server to trigger checkpoint
docker restart yaradb_server

Version conflicts (409 errors)

Cause: Multiple clients updating same document

Solution: Re-read document to get latest version:

doc = client.get(doc_id)
client.update(doc_id, version=doc["version"], body=new_data)

High memory usage

Monitor document count:

curl http://localhost:8000/table/list

Solution: Archive old documents or implement data lifecycle policies

🤝 Contributing

We welcome contributions! Please see:

Contributing Guide
Code of Conduct
Security Policy

Quick Start:

Fork the repository
Create feature branch: git checkout -b feature/amazing-feature
Make changes and add tests
Run tests: pytest tests/ -v
Commit: git commit -m 'Add amazing feature'
Push: git push origin feature/amazing-feature
Open Pull Request

📄 License

This project is licensed under the Server Side Public License (SSPL).

See LICENSE for full terms.

🔗 Resources

GitHub: github.com/illusiOxd/yaradb
Docker Hub: hub.docker.com/r/ashfromsky/yaradb
Python Client: github.com/illusiOxd/yaradb-client-py
Issue Tracker: github.com/illusiOxd/yaradb/issues
Discussions: github.com/illusiOxd/yaradb/discussions
Notion Docs: YaraDB Documentation

⭐ Support

If you find YaraDB useful:

Star the repository on GitHub
Share it with others
Report bugs and request features
Contribute code or documentation

Made with ❤️ by illusiOxd

# YaraDB Documentation

📦 Welcome to YaraDB

YaraDB is an intelligent, in-memory-first document database with crash-safe Write-Ahead Logging (WAL) persistence. Built with Python and FastAPI, it delivers lightning-fast performance while ensuring data durability and consistency.

🎯 What is YaraDB?

YaraDB is a lightweight, high-performance document database designed for modern applications that need:

Speed: In-memory operations with O(1) document retrieval
Safety: WAL-based persistence ensures no data loss on crashes
Simplicity: RESTful API with automatic OpenAPI documentation
Flexibility: Schema-free or schema-enforced tables
Reliability: Optimistic concurrency control prevents data races

Perfect for rapid prototyping, microservices, real-time applications, and testing environments.

✨ Key Features

Core Capabilities

Feature	Description
⚡ In-Memory Performance	Lightning-fast O(1) document retrieval by ID with hash-based indexing
💾 WAL Persistence	Crash-safe Write-Ahead Logging with automatic recovery and checkpointing
🔐 Optimistic Concurrency	Version-based conflict detection prevents lost updates
🗑️ Soft Deletes	Archive documents without permanent data loss - easily restore when needed
🔒 Data Integrity	SHA-256 body hashing validates document consistency on every operation
📊 Smart Tables	Create tables with strict schemas, unique constraints, or keep them schema-free
🌐 RESTful API	Clean HTTP interface with automatic OpenAPI/Swagger documentation
🐳 Docker Ready	Official Docker images with health checks and volume persistence

Advanced Features

Document Combining: Merge multiple documents with configurable strategies (overwrite, append, namespace)
Read-Only Tables: Create immutable historical data collections
Unique Constraints: Enforce field uniqueness at the table level
Schema Validation: Optional JSON Schema validation for strict data models
Batch Operations: Create multiple documents in a single request
Rate Limiting: Built-in request throttling with SlowAPI
Monitoring: Prometheus metrics integration out of the box
CORS Support: Cross-origin requests enabled by default

🚀 Quick Start

Prerequisites

Docker (recommended) OR
Python 3.8+

Option 1: Docker (Recommended)

Using Docker Run:

# Linux / macOS
docker run -d -p 8000:8000 \
  -v $(pwd)/yaradb_data:/data \
  -e DATA_DIR=/data \
  --name yaradb_server \
  ashfromsky/yaradb:latest

# Windows (PowerShell)
docker run -d -p 8000:8000 `
  -v ${PWD}/yaradb_data:/data `
  -e DATA_DIR=/data `
  --name yaradb_server `
  ashfromsky/yaradb:latest

Using Docker Compose:

Create docker-compose.yml:

version: '3.8'

services:
  yaradb:
    image: ashfromsky/yaradb:latest
    container_name: yaradb_server
    ports:
      - "8000:8000"
    volumes:
      - ./yaradb_data:/data
    environment:
      - DATA_DIR=/data
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/ping"]
      interval: 30s
      timeout: 10s
      retries: 3

Start the service:

docker-compose up -d

Option 2: Local Development

# Clone repository
git clone https://github.com/illusiOxd/yaradb.git
cd yaradb

# Create virtual environment
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Run server
python main.py

Verify Installation

# Test the server
curl http://localhost:8000/ping
# Expected: {"status":"alive"}

Access interactive API documentation at: http://localhost:8000/docs

💡 Basic Usage

Creating Documents

curl -X POST "http://localhost:8000/document/create" \
  -H "Content-Type: application/json" \
  -d '{
    "table_name": "users",
    "name": "user_alice",
    "body": {
      "username": "alice",
      "email": "alice@example.com",
      "role": "admin"
    }
  }'

Response:

{
  "_id": "550e8400-e29b-41d4-a716-446655440000",
  "name": "user_alice",
  "table_data": {
    "id": "123e4567-e89b-12d3-a456-426614174000",
    "name": "users"
  },
  "body": {
    "username": "alice",
    "email": "alice@example.com",
    "role": "admin"
  },
  "body_hash": "a7f8d...",
  "created_at": "2025-01-15T10:30:00Z",
  "version": 1
}

Reading Documents

curl "http://localhost:8000/document/get/550e8400-e29b-41d4-a716-446655440000"

Updating Documents (with Optimistic Locking)

curl -X PUT "http://localhost:8000/document/update/550e8400-e29b-41d4-a716-446655440000" \
  -H "Content-Type: application/json" \
  -d '{
    "version": 1,
    "body": {
      "username": "alice",
      "email": "alice@example.com",
      "role": "superadmin"
    }
  }'

Searching Documents

curl -X POST "http://localhost:8000/document/find" \
  -H "Content-Type: application/json" \
  -d '{"role": "admin"}'

Archiving Documents (Soft Delete)

curl -X PUT "http://localhost:8000/document/archive/550e8400-e29b-41d4-a716-446655440000"

📊 Working with Tables

Creating a Schema-Free Table

curl -X POST "http://localhost:8000/table/create" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "products",
    "mode": "free"
  }'

Creating a Strict Table with Schema

curl -X POST "http://localhost:8000/table/create" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "customers",
    "mode": "strict",
    "schema_definition": {
      "type": "object",
      "required": ["email", "name"],
      "properties": {
        "email": {"type": "string"},
        "name": {"type": "string"},
        "age": {"type": "integer"}
      }
    }
  }'

Creating a Read-Only Table

curl -X POST "http://localhost:8000/table/create" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "archive_2024",
    "mode": "free",
    "read_only": true
  }'

Adding Unique Constraints

curl -X POST "http://localhost:8000/table/create" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "accounts",
    "mode": "free",
    "unique_fields": ["email", "username"]
  }'

Listing All Tables

curl "http://localhost:8000/table/list"

Getting Table Details

curl "http://localhost:8000/table/users"

Getting Documents in a Table

curl "http://localhost:8000/table/users/documents"

Deleting a Table

curl -X DELETE "http://localhost:8000/table/products"

🔗 Advanced Features

Combining Documents

Merge multiple documents into one with different strategies:

Overwrite Strategy (last value wins):

curl -X POST "http://localhost:8000/document/combine" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "combined_profile",
    "document_ids": [
      "550e8400-e29b-41d4-a716-446655440000",
      "660e8400-e29b-41d4-a716-446655440001"
    ],
    "merge_strategy": "overwrite"
  }'

Append Strategy (merge lists):

curl -X POST "http://localhost:8000/document/combine" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "combined_logs",
    "document_ids": ["doc1-id", "doc2-id"],
    "merge_strategy": "append"
  }'

Namespace Strategy (keep documents separate):

curl -X POST "http://localhost:8000/document/combine" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "combined_configs",
    "document_ids": ["doc1-id", "doc2-id"],
    "merge_strategy": "namespace"
  }'

Batch Document Creation

curl -X POST "http://localhost:8000/document/batch/create" \
  -H "Content-Type: application/json" \
  -d '[
    {
      "table_name": "users",
      "name": "user1",
      "body": {"username": "bob"}
    },
    {
      "table_name": "users",
      "name": "user2",
      "body": {"username": "charlie"}
    }
  ]'

🏗️ Architecture Overview

YaraDB uses a simple but powerful three-layer architecture:

┌──────────────────────────────────────┐
│         REST API Layer               │
│  (FastAPI with OpenAPI docs)         │
└──────────────┬───────────────────────┘
               │
┌──────────────▼───────────────────────┐
│     In-Memory Storage Layer          │
│  • db_storage (list of documents)    │
│  • db_index_by_id (UUID → Document)  │
│  • db_tables_by_name (Table config)  │
└──────────────┬───────────────────────┘
               │
┌──────────────▼───────────────────────┐
│      Persistence Layer (WAL)         │
│  • yaradb_wal (operation log)        │
│  • yaradb_storage.json (snapshot)    │
└──────────────────────────────────────┘

Key Components

Document Store (core/state.py): In-memory collections with O(1) ID lookups
Repository (core/repository.py): Business logic and CRUD operations
WAL Engine (core/wal.py): Write-Ahead Logging with automatic recovery
API Layer (main.py): FastAPI endpoints with rate limiting and metrics

Data Flow

Write Path:
- Request → Repository → Log to WAL → Update memory → Return response
Read Path:
- Request → Repository → Memory lookup → Return response
Crash Recovery:
- Load snapshot from yaradb_storage.json
- Replay all operations from yaradb_wal
- Rebuild in-memory state
Checkpoint (on shutdown):
- Save all tables and documents to snapshot file
- Truncate WAL file

🔐 Data Safety

Optimistic Concurrency Control (OCC)

Every document has a version field that increments on each update. To modify a document, you must provide the current version:

# Read document (version = 1)
doc = client.get(doc_id)

# Update with version check
client.update(doc_id, version=1, body={"new": "data"})
# Success! Version is now 2

# Another client tries to update with old version
client.update(doc_id, version=1, body={"other": "data"})
# Error 409: Conflict - version mismatch

Write-Ahead Logging (WAL)

All write operations are logged to disk before updating memory:

Create: Logs entire document
Update: Logs doc_id, new version, new body
Archive: Logs doc_id, archive timestamp
Table Operations: Logs table creation/deletion

Operations are flushed to disk with fsync() to ensure durability.

Data Integrity

Every document has a SHA-256 hash of its body:

{
  "body": {"username": "alice"},
  "body_hash": "a7f8d9c2b1e4..." 
}

Hash is automatically recalculated on every update to detect data corruption.

📈 Monitoring & Observability

Prometheus Metrics

YaraDB exposes metrics at /metrics:

curl http://localhost:8000/metrics

Available metrics:

Request count by endpoint
Request duration histograms
Response status codes
Active requests

Logging

Logs are written to both console and yaradb.log:

2025-01-15 10:30:45 - yaradb - INFO - Document created: 550e8400...
2025-01-15 10:31:12 - yaradb - INFO - Document updated: 550e8400...
2025-01-15 10:32:00 - yaradb - WARNING - ☢️ DATABASE WIPED VIA SELF-DESTRUCT

Health Checks

# Liveness probe
curl http://localhost:8000/ping

# Docker health check (built-in)
docker inspect --format='{{.State.Health.Status}}' yaradb_server

🛡️ Security Features

Rate Limiting

API endpoints are rate-limited to prevent abuse:

@limiter.limit("10/minute")
async def create_document_endpoint(request: Request, ...):
    ...

Self-Destruct Protection

The database wipe endpoint requires two-factor verification:

curl -X DELETE "http://localhost:8000/system/self-destruct" \
  -H "Content-Type: application/json" \
  -d '{
    "verification_phrase": "BDaray",
    "safety_pin": 2026,
    "confirm": true
  }'

verification_phrase: "YaraDB" spelled backwards
safety_pin: Current year + 1

🔧 Configuration

Environment Variables

Variable	Default	Description
`DATA_DIR`	`.`	Directory for data files
`PYTHONUNBUFFERED`	`1`	Disable Python output buffering
`PYTHONDONTWRITEBYTECODE`	`1`	Don't create .pyc files

Data Files

yaradb_storage.json: Snapshot of all tables and documents
yaradb_wal: Write-Ahead Log (operation journal)

🐍 Python Client Library

For production use, we recommend the official Python client:

pip install yaradb-client

Example:

from yaradb_client import YaraClient

client = YaraClient("http://localhost:8000")

# Create document
doc = client.create(
    name="user",
    body={"username": "alice"},
    table_name="users"
)

# Read document
doc = client.get(doc["_id"])

# Update document
updated = client.update(
    doc_id=doc["_id"],
    version=doc["version"],
    body={"username": "alice_updated"}
)

# Search documents
results = client.find({"role": "admin"})

# Archive document
client.archive(doc["_id"])

See the [client repository](https://github.com/illusiOxd/yaradb-client-py) for full documentation.

📚 API Reference

Document Endpoints

Method	Endpoint	Description
POST	`/document/create`	Create a new document
GET	`/document/get/{doc_id}`	Get document by ID
POST	`/document/find`	Search documents by filter
PUT	`/document/update/{doc_id}`	Update document with version check
PUT	`/document/archive/{doc_id}`	Archive (soft delete) document
POST	`/document/batch/create`	Create multiple documents
POST	`/document/combine`	Combine multiple documents

Table Endpoints

Method	Endpoint	Description
POST	`/table/create`	Create a new table
GET	`/table/list`	List all tables
GET	`/table/{name}`	Get table details
GET	`/table/{name}/documents`	Get all documents in table
DELETE	`/table/{name}`	Delete table

System Endpoints

Method	Endpoint	Description
GET	`/ping`	Health check
GET	`/docs`	OpenAPI documentation
GET	`/metrics`	Prometheus metrics
DELETE	`/system/self-destruct`	Wipe all data (protected)