Data processing pipeline for the BTO instrument, transforming raw spacecraft telemetry into calibrated science-ready data products across four processing levels.
- Getting Started
- Installation
- Usage
- Repository Structure
- Pipeline Overview
- Data Levels
- Contributing
- Open Decisions
Placeholder — detailed quickstart instructions will be added once the environment is stable.
You will need:
- Python ≥ 3.10
astropy,numpy,fitsio(full list inrequirements.txt)- Access to the HEASARC CALDB for calibration files
# Clone the repository
git clone https://github.com/cositools/bto-pipeline.git
cd bto-pipeline
# Install dependencies
pip install -r requirements.txtPlaceholder — command-line interface and example runs will be documented here.
# Example: Process a day of raw telemetry through L0 → L1a
python run_pipeline.py --input /data/raw/2024/01/15 --level l1a
# Example: Apply calibration to produce L1b products
python run_pipeline.py --input /data/l1a/2024/01/15 --level l1bPlaceholder — update once the directory layout is finalised.
bto-pipeline/
├── l0/ # L0 binary processing
├── l1a/ # L1a FITS conversion
├── l1b/ # L1b calibration
├── l2/ # L2 science products
├── caldb/ # Local calibration file handling
├── utils/ # Shared utilities (time conversion, checksum, etc.)
├── tests/ # Unit and integration tests
├── docs/ # Additional documentation
└── README.md
Raw telemetry received from the spacecraft is ingested, verified, and progressively calibrated through the following stages:
Raw Telemetry → L0 (Binary) → L1a (Raw FITS) → L1b (Calibrated FITS) → L2 (Science FITS)
↓
influxDB + Grafana
All pipeline code lives under version control on this server and is publicly available. Calibration files are hosted on HEASARC via CALDB.
Raw binary packages received from the spacecraft, sorted by date (yyyy/mm/dd).
| Package Type | Description |
|---|---|
| D6 | Spectrum / Histograms |
| D7 | Event packages |
| D8 | Housekeeping |
| D9 | Registers |
| F6 | Bootloader HK |
Processing steps:
- Package verification (checksum)
- Sort files by date
Parsed and split FITS files, converted from binary with GPS→UTC time conversion and mission elapsed time computed.
Output files:
| File Pattern | Contents | Cadence |
|---|---|---|
csYYMMDDbto.hk |
Housekeeping + Bootloader | Daily |
csYYMMDDbto.lc |
Histograms | Daily |
| (light curves) | Light curves | Daily |
csYYMMDDXXXt_bto.evt |
Events / photon list (TTE) | Per event (XXX = fraction of day, t = trigger ID) |
Calibrated FITS files with ADC values converted to physical units. Hosted on HEASARC; calibration files served from CALDB.
Calibrations applied:
| Data Type | Calibration |
|---|---|
| Housekeeping | ADC → physical units |
| Histograms | ADC → energy + deadtime |
| Light curves | Energy bands |
| Photon list | ADC → energy + deadtime |
Auxiliary inputs required:
- BTO HK calibration (link TBD)
- BTO Energy calibration (link TBD)
- BTO Deadtime calibration
- COSI position information + orientation (TBD)
Reconstructed good photon data. Data products must conform to Gamma-ray Data Tools standards.
Auxiliary inputs:
- Response matrix from lab setup
Time-stamped raw and calibrated data (housekeeping + histograms) are ingested into an influxDB database. A Grafana dashboard provides real-time monitoring (public availability TBD).
This repository uses a feature branch workflow:
mainis the stable, reviewed branch. Direct pushes are not permitted.- All new work is developed on a dedicated branch and merged via a Pull Request (PR).
- Only the repository maintainer may merge PRs into
main, following code review.
Use the following prefixes to keep the branch list readable:
| Prefix | Use for |
|---|---|
feature/ |
New functionality (e.g. feature/l1b-energy-calibration) |
fix/ |
Bug fixes (e.g. fix/gps-utc-conversion) |
docs/ |
Documentation only (e.g. docs/update-readme) |
refactor/ |
Code cleanup with no functional change |
test/ |
Adding or improving tests |
-
Create a branch from the latest
main:git checkout main && git pull git checkout -b feature/your-feature-name -
Develop and commit your changes with clear, descriptive commit messages.
-
Push your branch and open a Pull Request on GitHub:
git push origin feature/your-feature-name
-
Fill in the PR template — describe what changed, reference any relevant issues, and confirm local testing.
-
Request review from the maintainer. Address any feedback with follow-up commits on the same branch.
-
Merge — once approved, the maintainer merges the PR into
main. Prefer Squash and merge for a clean history on small features, or Merge commit to preserve detailed history on larger efforts. -
Delete your branch after merging to keep the repo tidy.
Before requesting review, confirm:
- Code runs without errors locally
- New functionality includes tests (or tests are noted as a follow-up)
- Docstrings/comments updated where relevant
- No hardcoded paths or credentials
- FITS output validated against expected format
- Update the version number of the pipeline
Reviewers should check for:
- Correctness of calibration logic
- FITS header compliance
- Time system handling (GPS vs UTC vs MET)
- Consistent file naming conventions (see Data Levels above)
The following items are currently unresolved and tracked here until closed:
| Level | Question |
|---|---|
| L0 | Do we append packages? |
| L0 | Do we sort by time? (BTO data may not arrive time-sorted) |
| L1a | FITS Header format — port Carolyn's email |
| L1a | Mirror BTO ADC housekeeping into both data streams? |
| L1a | Should there be a filler for time periods with no data? |
| L1a | Do we archive register readbacks? |
| L1b | Config file format — FITS extensions, FITS header, or standalone file? |
| L1b | Visualization of data approach |
| L1b | BTO HK calibration link (missing) |
| L1b | BTO Energy calibration link (missing) |
| L1b | COSI position + orientation as auxiliary input? |
| L2 | Confirm conformance requirements with Gamma-ray Data Tools |
| influxDB | Document table schema and ingestion details |
| Grafana | Should the dashboard be publicly available? |
This project is licensed under the Apache License, Version 2.0.
You are free to use, modify, and distribute this software for any purpose, including commercial use, provided that you include the required attribution notice. Contributors are also granted an explicit patent license under Apache 2.0.
See the LICENSE file for the full license text, or visit apache.org/licenses/LICENSE-2.0.