Skip to content

Latest commit

 

History

History
677 lines (563 loc) · 17.6 KB

STRUCTURE_IMPROVEMENTS.md

File metadata and controls

677 lines (563 loc) · 17.6 KB

Repository Structure Improvement Suggestions

Date: November 2025 Repository: python_GPT Purpose: Recommendations for improving code organization, maintainability, and scalability

🎯 Executive Summary

The python_GPT repository contains excellent, functional code with GPU-optimized implementations. However, the current structure has grown organically, resulting in:

  • Date-based versioning that makes it hard to find the "current" implementation
  • Multiple versions (v1-v9) scattered across directories
  • Difficulty in understanding which code is production-ready vs experimental
  • No clear separation between library code and scripts

This document proposes actionable improvements to enhance maintainability while preserving existing work.

📊 Current State Analysis

Strengths

  1. Rich functionality: 45+ projects covering diverse ML/AI domains
  2. GPU optimization: Hardware-specific tuning for production workloads
  3. Comprehensive documentation: Individual README files per project
  4. Active development: Monthly iterations show ongoing improvement

Challenges

  1. Date-based organization: nov14/, nov15/, nov19/ directories obscure latest versions
  2. Version proliferation: Multiple v1, v2, v3 implementations without clear deprecation
  3. Flat hierarchy: 45+ top-level directories make navigation difficult
  4. Mixed concerns: Scripts, libraries, and data intermixed
  5. No clear entry points: Unclear which files to run for each workflow

🏗️ Proposed Structure

Option A: Functional Grouping (Recommended)

python_GPT/
├── docs/                          # All documentation
│   ├── architecture/
│   ├── guides/
│   └── api/
│
├── src/                           # Library code (importable)
│   ├── document_processing/
│   │   ├── ocr/                   # copali_OCR refactored
│   │   │   ├── __init__.py
│   │   │   ├── colqwen_processor.py
│   │   │   ├── tesseract_processor.py
│   │   │   └── batch_processor.py
│   │   ├── pdf/                   # PDF utilities
│   │   │   ├── validator.py
│   │   │   ├── downloader.py
│   │   │   └── extractor.py
│   │   └── reference_extraction/
│   │
│   ├── vector_db/
│   │   ├── faiss_index.py
│   │   ├── rag_pipeline.py
│   │   └── enroller.py
│   │
│   ├── nlp/
│   │   ├── novelty/
│   │   ├── topic_modeling/
│   │   └── vectorization/
│   │
│   ├── llm_clients/
│   │   ├── deepseek.py
│   │   ├── gemini.py
│   │   └── ollama.py
│   │
│   ├── gpu_computing/
│   │   ├── cupy_ops.py
│   │   └── parallel_processing.py
│   │
│   └── neural_networks/
│       ├── ltc/
│       ├── gru/
│       └── training/
│
├── scripts/                       # Executable scripts
│   ├── ocr_pipeline.py
│   ├── build_rag_index.py
│   ├── query_documents.py
│   ├── train_ltc.py
│   └── batch_process_pdfs.py
│
├── examples/                      # Example usage
│   ├── ocr_example.py
│   ├── rag_example.py
│   └── notebooks/
│
├── tests/                         # Unit and integration tests
│   ├── test_ocr/
│   ├── test_rag/
│   └── test_nlp/
│
├── configs/                       # Configuration files
│   ├── gpu_config.yaml
│   ├── models.yaml
│   └── paths.yaml
│
├── data/                          # Data directory (gitignored except examples)
│   ├── samples/
│   └── .gitignore
│
├── experimental/                  # Research and prototypes
│   ├── archive/                   # Old versions
│   │   ├── nov14/
│   │   ├── nov15/
│   │   └── v1_v2_v3_implementations/
│   └── current/                   # Active experiments
│
├── infrastructure/                # DevOps and tooling
│   ├── proof_of_workforce/
│   ├── fail2ban/
│   └── monitoring/
│
├── utilities/                     # Standalone utilities
│   ├── ballistics/
│   ├── ishihara/
│   └── visualization/
│
├── requirements/                  # Dependency management
│   ├── base.txt
│   ├── gpu.txt
│   ├── dev.txt
│   └── production.txt
│
├── .github/                       # GitHub workflows
│   └── workflows/
│
├── setup.py                       # Package installation
├── pyproject.toml                 # Modern Python packaging
├── README.md                      # Main documentation
└── LICENSE

Option B: Minimal Reorganization (Low Risk)

Keep existing structure but add:

python_GPT/
├── [existing 45 directories]
├── _archive/                      # Move old versions here
│   ├── copali_OCR_nov14/
│   ├── copali_OCR_nov15/
│   └── deprecated_implementations/
│
├── _current/                      # Symlinks to current versions
│   ├── ocr -> ../copali_OCR/nov28/
│   ├── rag -> ../RAG/
│   └── [other current versions]
│
└── PROJECT_INDEX.md              # Map of what's where

📋 Specific Recommendations

1. Version Management

Current Problem:

copali_OCR/
├── working_nov9/
├── Nov14_GPUselect/
├── Nov15_GPUopt/
├── Nov19/
└── Nov28/          # Which one is current?

Recommended Approach:

src/document_processing/ocr/
├── core/           # Current stable implementation
│   ├── __init__.py
│   ├── processor.py
│   └── config.py
├── experimental/   # Active development
│   └── gpu_optimization_dec2025/
└── archive/        # Historical versions
    ├── nov14_gpu_select/
    ├── nov15_gpu_opt/
    └── CHANGELOG.md

Implementation:

  • Maintain one canonical version in core/
  • Use git branches for experiments, not directories
  • Archive old versions with clear CHANGELOG
  • Use semantic versioning in code: __version__ = "2.3.0"

2. Dependency Management

Current Problem:

  • Multiple requirements.txt scattered across directories
  • Unclear which dependencies are for what purpose
  • No distinction between dev/prod requirements

Recommended Approach:

requirements/
├── base.txt              # Core dependencies
├── gpu.txt               # CUDA, CuPy, GPU-specific
├── ocr.txt               # Tesseract, PyMuPDF, ColQwen
├── rag.txt               # FAISS, transformers, embeddings
├── llm.txt               # API clients
├── neural_networks.txt   # PyTorch, training deps
├── dev.txt               # Testing, linting, formatting
└── production.txt        # Consolidated production deps

Installation commands:

# Base installation
pip install -r requirements/base.txt

# With GPU support
pip install -r requirements/base.txt -r requirements/gpu.txt

# Full development environment
pip install -r requirements/dev.txt

3. Configuration Management

Current Problem:

  • Hardcoded paths and configurations
  • GPU device selection scattered across files
  • No centralized configuration

Recommended Approach:

Create configs/ directory:

# configs/gpu_config.yaml
gpu:
  default_device: 0
  memory_fraction: 0.8
  optimization_level: "high"
  supported_models:
    - RTX_3060
    - RTX_3080
    - RTX_4080

# configs/models.yaml
models:
  ocr:
    tesseract_lang: "eng"
    colqwen_model: "vidore/colqwen2-v0.1"
  embedding:
    default_model: "sentence-transformers/all-MiniLM-L6-v2"
    dimensions: 384
  llm:
    deepseek_endpoint: "https://api.deepseek.com"

# configs/paths.yaml
paths:
  data_dir: "./data"
  models_dir: "./models"
  output_dir: "./output"
  temp_dir: "/tmp/python_gpt"

Usage:

from pathlib import Path
import yaml

def load_config(config_name):
    config_path = Path(__file__).parent.parent / "configs" / f"{config_name}.yaml"
    return yaml.safe_load(config_path.read_text())

gpu_config = load_config("gpu_config")

4. Entry Points and Scripts

Current Problem:

  • Unclear which Python files are meant to be executed
  • No clear workflow documentation
  • Scripts and libraries intermixed

Recommended Approach:

Create scripts/ directory with clear entry points:

# scripts/ocr_pipeline.py
"""
Entry point for OCR processing pipeline.

Usage:
    python scripts/ocr_pipeline.py --input docs/ --output results/
"""
import argparse
from src.document_processing.ocr import BatchProcessor

def main():
    parser = argparse.ArgumentParser(description="Run OCR pipeline")
    parser.add_argument("--input", required=True)
    parser.add_argument("--output", required=True)
    parser.add_argument("--gpu-device", type=int, default=0)
    args = parser.parse_args()

    processor = BatchProcessor(gpu_device=args.gpu_device)
    processor.process_directory(args.input, args.output)

if __name__ == "__main__":
    main()

Add to pyproject.toml:

[project.scripts]
ocr-process = "scripts.ocr_pipeline:main"
rag-build = "scripts.build_rag_index:main"
rag-query = "scripts.query_documents:main"

5. Testing Structure

Current Problem:

  • No visible test infrastructure
  • No CI/CD pipeline
  • Difficult to verify changes don't break existing code

Recommended Approach:

tests/
├── conftest.py              # Pytest configuration
├── fixtures/                # Test data
│   ├── sample.pdf
│   └── test_corpus.txt
├── unit/
│   ├── test_ocr/
│   │   ├── test_colqwen.py
│   │   └── test_batch_processor.py
│   ├── test_rag/
│   │   ├── test_faiss_index.py
│   │   └── test_embeddings.py
│   └── test_nlp/
│       └── test_novelty.py
├── integration/
│   ├── test_ocr_to_rag_pipeline.py
│   └── test_end_to_end.py
└── performance/
    └── test_gpu_benchmarks.py

Example test:

# tests/unit/test_ocr/test_batch_processor.py
import pytest
from src.document_processing.ocr import BatchProcessor

@pytest.fixture
def sample_pdf(tmp_path):
    # Create or copy sample PDF
    return tmp_path / "sample.pdf"

def test_batch_processor_initialization():
    processor = BatchProcessor(gpu_device=0)
    assert processor.device == 0

def test_process_single_pdf(sample_pdf):
    processor = BatchProcessor()
    result = processor.process_file(sample_pdf)
    assert result.success
    assert len(result.text) > 0

6. Documentation Structure

Current Problem:

  • Documentation scattered across many README files
  • No API documentation
  • No architecture documentation

Recommended Approach:

docs/
├── index.md                          # Main documentation hub
├── getting_started/
│   ├── installation.md
│   ├── quickstart.md
│   └── common_workflows.md
├── architecture/
│   ├── overview.md
│   ├── data_flow.md
│   ├── gpu_optimization.md
│   └── component_interaction.md
├── guides/
│   ├── ocr_pipeline.md
│   ├── rag_system.md
│   ├── training_neural_networks.md
│   └── gpu_tuning.md
├── api/
│   ├── document_processing.md
│   ├── vector_db.md
│   ├── nlp.md
│   └── llm_clients.md
├── development/
│   ├── contributing.md
│   ├── testing.md
│   └── release_process.md
└── troubleshooting/
    ├── common_issues.md
    ├── gpu_problems.md
    └── performance_tuning.md

7. Package Installation

Current Problem:

  • Not installable as a package
  • No namespace management
  • Relative imports problematic

Recommended Solution:

Create setup.py or pyproject.toml:

# pyproject.toml
[build-system]
requires = ["setuptools>=61.0", "wheel"]
build-backend = "setuptools.build_meta"

[project]
name = "python-gpt"
version = "0.1.0"
description = "AI/ML experimentation platform for document understanding and RAG"
authors = [{name = "Your Name"}]
readme = "README.md"
requires-python = ">=3.10"
license = {text = "MIT"}

dependencies = [
    "numpy>=1.24.0",
    "pandas>=2.0.0",
    "torch>=2.0.0",
    "transformers>=4.30.0",
]

[project.optional-dependencies]
gpu = [
    "cupy-cuda11x>=12.0.0",
]
ocr = [
    "pytesseract>=0.3.10",
    "pymupdf>=1.22.0",
]
rag = [
    "faiss-cpu>=1.7.4",
    "sentence-transformers>=2.2.0",
]
dev = [
    "pytest>=7.3.0",
    "black>=23.0.0",
    "ruff>=0.0.270",
    "mypy>=1.3.0",
]

[project.scripts]
ocr-process = "python_gpt.scripts.ocr_pipeline:main"
rag-build = "python_gpt.scripts.build_rag_index:main"
rag-query = "python_gpt.scripts.query_documents:main"

[tool.setuptools.packages.find]
where = ["src"]

[tool.black]
line-length = 100

[tool.ruff]
line-length = 100

Installation:

# Editable install for development
pip install -e .

# With GPU support
pip install -e ".[gpu]"

# Full install with all features
pip install -e ".[gpu,ocr,rag,dev]"

🔄 Migration Strategy

Phase 1: Non-Breaking Additions (Week 1)

  1. Add PROJECT_INDEX.md mapping current vs deprecated code
  2. Create configs/ directory with YAML configs
  3. Add scripts/ directory with entry points
  4. Create docs/ structure with existing READMEs reorganized
  5. Add requirements/ directory with split dependencies

Impact: Zero breaking changes, additive only

Phase 2: New Structure (Week 2-3)

  1. Create src/ directory with package structure
  2. Copy (don't move) current implementations to new locations
  3. Add __init__.py files for proper package structure
  4. Create setup.py/pyproject.toml
  5. Add basic tests in tests/

Impact: Old structure still works, new structure available

Phase 3: Migration (Week 4)

  1. Update import paths to use new structure
  2. Create symlinks from old locations to new
  3. Add deprecation warnings in old locations
  4. Update documentation to reference new structure

Impact: Old code still works but warns about deprecation

Phase 4: Cleanup (Week 5+)

  1. Move old versions to experimental/archive/
  2. Remove deprecated symlinks
  3. Final documentation update
  4. Celebrate cleaner codebase!

Impact: Old structure removed, only new structure remains

🎯 Quick Wins (Implement First)

1. Add PROJECT_INDEX.md

Create a single file mapping all 45+ directories:

# Project Index

## Current/Production Code
- **OCR**: `copali_OCR/nov28/` - Latest GPU-optimized implementation
- **RAG**: `RAG/` - Production RAG pipeline
- **FAISS**: `faiss/` - Vector indexing
...

## Experimental/Deprecated
- `copali_OCR/nov14/` - Deprecated, use nov28
- `copali_OCR/nov15/` - Deprecated, use nov28
...

2. Create Unified Entry Points

# scripts/workflows.sh
#!/bin/bash

case "$1" in
    ocr)
        python copali_OCR/nov28/pdf-ocr-ds.py "${@:2}"
        ;;
    rag-build)
        python RAG/setup_faiss_index.py "${@:2}"
        ;;
    rag-query)
        python RAG/query_engine.py "${@:2}"
        ;;
    *)
        echo "Usage: $0 {ocr|rag-build|rag-query} [args]"
        exit 1
        ;;
esac

3. Add .gitignore Improvements

# Virtual environments
venv/
venv_activate/
python3-12-venv-production/
*.pyc
__pycache__/

# Data files
*.pdf
*.png
*.jpg
data/
output/
models/
*.db

# IDE
.vscode/
.idea/
*.swp

# Temporary
tmp/
temp/
*.log

4. Create CONTRIBUTING.md

Guide for adding new code to prevent future sprawl:

# Contributing Guide

## Adding New Features

1. **Create a feature branch**: `git checkout -b feature/your-feature`
2. **Add code to appropriate directory**:
   - Libraries: `src/category/`
   - Scripts: `scripts/`
   - Experiments: `experimental/current/`
3. **Add tests**: `tests/unit/test_your_feature.py`
4. **Update docs**: `docs/guides/your_feature.md`
5. **Submit PR** with description

## Version Management

- **Don't create date-based directories**
- **Use git branches** for experiments
- **Use semantic versioning** in code
- **Archive old code** to `experimental/archive/`

## Naming Conventions

- Files: `snake_case.py`
- Classes: `PascalCase`
- Functions: `snake_case()`
- Constants: `UPPER_SNAKE_CASE`

📊 Success Metrics

Track these to measure improvement success:

  1. Time to find current implementation: Should be <30 seconds
  2. Onboarding time: New developers productive in <1 day
  3. Test coverage: Target >70% for core modules
  4. Build time: pip install completes in <5 minutes
  5. Documentation coverage: All public APIs documented

🚀 Immediate Action Items

Priority 1 (This week):

  • Create PROJECT_INDEX.md
  • Add unified scripts/workflows.sh
  • Improve .gitignore
  • Create CONTRIBUTING.md

Priority 2 (Next 2 weeks):

  • Create configs/ directory
  • Split requirements.txt into modular files
  • Create docs/ structure
  • Add basic tests

Priority 3 (Month 2):

  • Create src/ package structure
  • Add setup.py/pyproject.toml
  • Migrate imports
  • Archive old versions

📚 References


Remember: The goal is incremental improvement, not a complete rewrite. Each change should add value while maintaining backward compatibility during the transition.