Date: November 2025 Repository: python_GPT Purpose: Recommendations for improving code organization, maintainability, and scalability
The python_GPT repository contains excellent, functional code with GPU-optimized implementations. However, the current structure has grown organically, resulting in:
- Date-based versioning that makes it hard to find the "current" implementation
- Multiple versions (v1-v9) scattered across directories
- Difficulty in understanding which code is production-ready vs experimental
- No clear separation between library code and scripts
This document proposes actionable improvements to enhance maintainability while preserving existing work.
- Rich functionality: 45+ projects covering diverse ML/AI domains
- GPU optimization: Hardware-specific tuning for production workloads
- Comprehensive documentation: Individual README files per project
- Active development: Monthly iterations show ongoing improvement
- Date-based organization:
nov14/,nov15/,nov19/directories obscure latest versions - Version proliferation: Multiple
v1,v2,v3implementations without clear deprecation - Flat hierarchy: 45+ top-level directories make navigation difficult
- Mixed concerns: Scripts, libraries, and data intermixed
- No clear entry points: Unclear which files to run for each workflow
python_GPT/
├── docs/ # All documentation
│ ├── architecture/
│ ├── guides/
│ └── api/
│
├── src/ # Library code (importable)
│ ├── document_processing/
│ │ ├── ocr/ # copali_OCR refactored
│ │ │ ├── __init__.py
│ │ │ ├── colqwen_processor.py
│ │ │ ├── tesseract_processor.py
│ │ │ └── batch_processor.py
│ │ ├── pdf/ # PDF utilities
│ │ │ ├── validator.py
│ │ │ ├── downloader.py
│ │ │ └── extractor.py
│ │ └── reference_extraction/
│ │
│ ├── vector_db/
│ │ ├── faiss_index.py
│ │ ├── rag_pipeline.py
│ │ └── enroller.py
│ │
│ ├── nlp/
│ │ ├── novelty/
│ │ ├── topic_modeling/
│ │ └── vectorization/
│ │
│ ├── llm_clients/
│ │ ├── deepseek.py
│ │ ├── gemini.py
│ │ └── ollama.py
│ │
│ ├── gpu_computing/
│ │ ├── cupy_ops.py
│ │ └── parallel_processing.py
│ │
│ └── neural_networks/
│ ├── ltc/
│ ├── gru/
│ └── training/
│
├── scripts/ # Executable scripts
│ ├── ocr_pipeline.py
│ ├── build_rag_index.py
│ ├── query_documents.py
│ ├── train_ltc.py
│ └── batch_process_pdfs.py
│
├── examples/ # Example usage
│ ├── ocr_example.py
│ ├── rag_example.py
│ └── notebooks/
│
├── tests/ # Unit and integration tests
│ ├── test_ocr/
│ ├── test_rag/
│ └── test_nlp/
│
├── configs/ # Configuration files
│ ├── gpu_config.yaml
│ ├── models.yaml
│ └── paths.yaml
│
├── data/ # Data directory (gitignored except examples)
│ ├── samples/
│ └── .gitignore
│
├── experimental/ # Research and prototypes
│ ├── archive/ # Old versions
│ │ ├── nov14/
│ │ ├── nov15/
│ │ └── v1_v2_v3_implementations/
│ └── current/ # Active experiments
│
├── infrastructure/ # DevOps and tooling
│ ├── proof_of_workforce/
│ ├── fail2ban/
│ └── monitoring/
│
├── utilities/ # Standalone utilities
│ ├── ballistics/
│ ├── ishihara/
│ └── visualization/
│
├── requirements/ # Dependency management
│ ├── base.txt
│ ├── gpu.txt
│ ├── dev.txt
│ └── production.txt
│
├── .github/ # GitHub workflows
│ └── workflows/
│
├── setup.py # Package installation
├── pyproject.toml # Modern Python packaging
├── README.md # Main documentation
└── LICENSE
Keep existing structure but add:
python_GPT/
├── [existing 45 directories]
├── _archive/ # Move old versions here
│ ├── copali_OCR_nov14/
│ ├── copali_OCR_nov15/
│ └── deprecated_implementations/
│
├── _current/ # Symlinks to current versions
│ ├── ocr -> ../copali_OCR/nov28/
│ ├── rag -> ../RAG/
│ └── [other current versions]
│
└── PROJECT_INDEX.md # Map of what's where
Current Problem:
copali_OCR/
├── working_nov9/
├── Nov14_GPUselect/
├── Nov15_GPUopt/
├── Nov19/
└── Nov28/ # Which one is current?
Recommended Approach:
src/document_processing/ocr/
├── core/ # Current stable implementation
│ ├── __init__.py
│ ├── processor.py
│ └── config.py
├── experimental/ # Active development
│ └── gpu_optimization_dec2025/
└── archive/ # Historical versions
├── nov14_gpu_select/
├── nov15_gpu_opt/
└── CHANGELOG.md
Implementation:
- Maintain one canonical version in
core/ - Use git branches for experiments, not directories
- Archive old versions with clear CHANGELOG
- Use semantic versioning in code:
__version__ = "2.3.0"
Current Problem:
- Multiple
requirements.txtscattered across directories - Unclear which dependencies are for what purpose
- No distinction between dev/prod requirements
Recommended Approach:
requirements/
├── base.txt # Core dependencies
├── gpu.txt # CUDA, CuPy, GPU-specific
├── ocr.txt # Tesseract, PyMuPDF, ColQwen
├── rag.txt # FAISS, transformers, embeddings
├── llm.txt # API clients
├── neural_networks.txt # PyTorch, training deps
├── dev.txt # Testing, linting, formatting
└── production.txt # Consolidated production deps
Installation commands:
# Base installation
pip install -r requirements/base.txt
# With GPU support
pip install -r requirements/base.txt -r requirements/gpu.txt
# Full development environment
pip install -r requirements/dev.txtCurrent Problem:
- Hardcoded paths and configurations
- GPU device selection scattered across files
- No centralized configuration
Recommended Approach:
Create configs/ directory:
# configs/gpu_config.yaml
gpu:
default_device: 0
memory_fraction: 0.8
optimization_level: "high"
supported_models:
- RTX_3060
- RTX_3080
- RTX_4080
# configs/models.yaml
models:
ocr:
tesseract_lang: "eng"
colqwen_model: "vidore/colqwen2-v0.1"
embedding:
default_model: "sentence-transformers/all-MiniLM-L6-v2"
dimensions: 384
llm:
deepseek_endpoint: "https://api.deepseek.com"
# configs/paths.yaml
paths:
data_dir: "./data"
models_dir: "./models"
output_dir: "./output"
temp_dir: "/tmp/python_gpt"Usage:
from pathlib import Path
import yaml
def load_config(config_name):
config_path = Path(__file__).parent.parent / "configs" / f"{config_name}.yaml"
return yaml.safe_load(config_path.read_text())
gpu_config = load_config("gpu_config")Current Problem:
- Unclear which Python files are meant to be executed
- No clear workflow documentation
- Scripts and libraries intermixed
Recommended Approach:
Create scripts/ directory with clear entry points:
# scripts/ocr_pipeline.py
"""
Entry point for OCR processing pipeline.
Usage:
python scripts/ocr_pipeline.py --input docs/ --output results/
"""
import argparse
from src.document_processing.ocr import BatchProcessor
def main():
parser = argparse.ArgumentParser(description="Run OCR pipeline")
parser.add_argument("--input", required=True)
parser.add_argument("--output", required=True)
parser.add_argument("--gpu-device", type=int, default=0)
args = parser.parse_args()
processor = BatchProcessor(gpu_device=args.gpu_device)
processor.process_directory(args.input, args.output)
if __name__ == "__main__":
main()Add to pyproject.toml:
[project.scripts]
ocr-process = "scripts.ocr_pipeline:main"
rag-build = "scripts.build_rag_index:main"
rag-query = "scripts.query_documents:main"Current Problem:
- No visible test infrastructure
- No CI/CD pipeline
- Difficult to verify changes don't break existing code
Recommended Approach:
tests/
├── conftest.py # Pytest configuration
├── fixtures/ # Test data
│ ├── sample.pdf
│ └── test_corpus.txt
├── unit/
│ ├── test_ocr/
│ │ ├── test_colqwen.py
│ │ └── test_batch_processor.py
│ ├── test_rag/
│ │ ├── test_faiss_index.py
│ │ └── test_embeddings.py
│ └── test_nlp/
│ └── test_novelty.py
├── integration/
│ ├── test_ocr_to_rag_pipeline.py
│ └── test_end_to_end.py
└── performance/
└── test_gpu_benchmarks.py
Example test:
# tests/unit/test_ocr/test_batch_processor.py
import pytest
from src.document_processing.ocr import BatchProcessor
@pytest.fixture
def sample_pdf(tmp_path):
# Create or copy sample PDF
return tmp_path / "sample.pdf"
def test_batch_processor_initialization():
processor = BatchProcessor(gpu_device=0)
assert processor.device == 0
def test_process_single_pdf(sample_pdf):
processor = BatchProcessor()
result = processor.process_file(sample_pdf)
assert result.success
assert len(result.text) > 0Current Problem:
- Documentation scattered across many README files
- No API documentation
- No architecture documentation
Recommended Approach:
docs/
├── index.md # Main documentation hub
├── getting_started/
│ ├── installation.md
│ ├── quickstart.md
│ └── common_workflows.md
├── architecture/
│ ├── overview.md
│ ├── data_flow.md
│ ├── gpu_optimization.md
│ └── component_interaction.md
├── guides/
│ ├── ocr_pipeline.md
│ ├── rag_system.md
│ ├── training_neural_networks.md
│ └── gpu_tuning.md
├── api/
│ ├── document_processing.md
│ ├── vector_db.md
│ ├── nlp.md
│ └── llm_clients.md
├── development/
│ ├── contributing.md
│ ├── testing.md
│ └── release_process.md
└── troubleshooting/
├── common_issues.md
├── gpu_problems.md
└── performance_tuning.md
Current Problem:
- Not installable as a package
- No namespace management
- Relative imports problematic
Recommended Solution:
Create setup.py or pyproject.toml:
# pyproject.toml
[build-system]
requires = ["setuptools>=61.0", "wheel"]
build-backend = "setuptools.build_meta"
[project]
name = "python-gpt"
version = "0.1.0"
description = "AI/ML experimentation platform for document understanding and RAG"
authors = [{name = "Your Name"}]
readme = "README.md"
requires-python = ">=3.10"
license = {text = "MIT"}
dependencies = [
"numpy>=1.24.0",
"pandas>=2.0.0",
"torch>=2.0.0",
"transformers>=4.30.0",
]
[project.optional-dependencies]
gpu = [
"cupy-cuda11x>=12.0.0",
]
ocr = [
"pytesseract>=0.3.10",
"pymupdf>=1.22.0",
]
rag = [
"faiss-cpu>=1.7.4",
"sentence-transformers>=2.2.0",
]
dev = [
"pytest>=7.3.0",
"black>=23.0.0",
"ruff>=0.0.270",
"mypy>=1.3.0",
]
[project.scripts]
ocr-process = "python_gpt.scripts.ocr_pipeline:main"
rag-build = "python_gpt.scripts.build_rag_index:main"
rag-query = "python_gpt.scripts.query_documents:main"
[tool.setuptools.packages.find]
where = ["src"]
[tool.black]
line-length = 100
[tool.ruff]
line-length = 100Installation:
# Editable install for development
pip install -e .
# With GPU support
pip install -e ".[gpu]"
# Full install with all features
pip install -e ".[gpu,ocr,rag,dev]"- Add
PROJECT_INDEX.mdmapping current vs deprecated code - Create
configs/directory with YAML configs - Add
scripts/directory with entry points - Create
docs/structure with existing READMEs reorganized - Add
requirements/directory with split dependencies
Impact: Zero breaking changes, additive only
- Create
src/directory with package structure - Copy (don't move) current implementations to new locations
- Add
__init__.pyfiles for proper package structure - Create
setup.py/pyproject.toml - Add basic tests in
tests/
Impact: Old structure still works, new structure available
- Update import paths to use new structure
- Create symlinks from old locations to new
- Add deprecation warnings in old locations
- Update documentation to reference new structure
Impact: Old code still works but warns about deprecation
- Move old versions to
experimental/archive/ - Remove deprecated symlinks
- Final documentation update
- Celebrate cleaner codebase!
Impact: Old structure removed, only new structure remains
Create a single file mapping all 45+ directories:
# Project Index
## Current/Production Code
- **OCR**: `copali_OCR/nov28/` - Latest GPU-optimized implementation
- **RAG**: `RAG/` - Production RAG pipeline
- **FAISS**: `faiss/` - Vector indexing
...
## Experimental/Deprecated
- `copali_OCR/nov14/` - Deprecated, use nov28
- `copali_OCR/nov15/` - Deprecated, use nov28
...# scripts/workflows.sh
#!/bin/bash
case "$1" in
ocr)
python copali_OCR/nov28/pdf-ocr-ds.py "${@:2}"
;;
rag-build)
python RAG/setup_faiss_index.py "${@:2}"
;;
rag-query)
python RAG/query_engine.py "${@:2}"
;;
*)
echo "Usage: $0 {ocr|rag-build|rag-query} [args]"
exit 1
;;
esac# Virtual environments
venv/
venv_activate/
python3-12-venv-production/
*.pyc
__pycache__/
# Data files
*.pdf
*.png
*.jpg
data/
output/
models/
*.db
# IDE
.vscode/
.idea/
*.swp
# Temporary
tmp/
temp/
*.logGuide for adding new code to prevent future sprawl:
# Contributing Guide
## Adding New Features
1. **Create a feature branch**: `git checkout -b feature/your-feature`
2. **Add code to appropriate directory**:
- Libraries: `src/category/`
- Scripts: `scripts/`
- Experiments: `experimental/current/`
3. **Add tests**: `tests/unit/test_your_feature.py`
4. **Update docs**: `docs/guides/your_feature.md`
5. **Submit PR** with description
## Version Management
- **Don't create date-based directories**
- **Use git branches** for experiments
- **Use semantic versioning** in code
- **Archive old code** to `experimental/archive/`
## Naming Conventions
- Files: `snake_case.py`
- Classes: `PascalCase`
- Functions: `snake_case()`
- Constants: `UPPER_SNAKE_CASE`Track these to measure improvement success:
- Time to find current implementation: Should be <30 seconds
- Onboarding time: New developers productive in <1 day
- Test coverage: Target >70% for core modules
- Build time:
pip installcompletes in <5 minutes - Documentation coverage: All public APIs documented
Priority 1 (This week):
- Create
PROJECT_INDEX.md - Add unified
scripts/workflows.sh - Improve
.gitignore - Create
CONTRIBUTING.md
Priority 2 (Next 2 weeks):
- Create
configs/directory - Split
requirements.txtinto modular files - Create
docs/structure - Add basic tests
Priority 3 (Month 2):
- Create
src/package structure - Add
setup.py/pyproject.toml - Migrate imports
- Archive old versions
Remember: The goal is incremental improvement, not a complete rewrite. Each change should add value while maintaining backward compatibility during the transition.