Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 6 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -306,14 +306,12 @@ Our work is built upon [LW-DETR](https://arxiv.org/pdf/2406.03459), [DINOv2](htt
If you find our work helpful for your research, please consider citing the following BibTeX entry.

```bibtex
@misc{rf-detr,
title={RF-DETR: Neural Architecture Search for Real-Time Detection Transformers},
author={Isaac Robinson and Peter Robicheaux and Matvei Popov and Deva Ramanan and Neehar Peri},
year={2025},
eprint={2511.09554},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2511.09554},
@inproceedings{robinson2026rfdetr,
title = {RF-DETR: Real-Time Detection Transformer},
author = {Robinson, Isaac and Robicheaux, Peter and Popov, Fedor and Ramanan, Deva and Peri, Neehar},
booktitle = {International Conference on Learning Representations (ICLR)},
year = {2026},
url = {https://arxiv.org/abs/2511.09554}
}
```

Expand Down
17 changes: 17 additions & 0 deletions docs/getting-started/install.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,23 @@ If you plan to contribute to RF-DETR or modify the codebase locally, set up a lo
uv pip install -e . --all-extras
```

## Optional Extras

RF-DETR provides several optional extras for additional functionality:

| Extra | Install command | Purpose |
| --------- | ----------------------------------- | --------------------------------------------------------------- |
| `train` | `pip install "rfdetr[train]"` | Training dependencies (PyTorch Lightning, albumentations, etc.) |
| `loggers` | `pip install "rfdetr[loggers]"` | Experiment tracking (TensorBoard, W&B, MLflow, ClearML) |
| `onnx` | `pip install "rfdetr[onnx]"` | ONNX export |
| `tflite` | `pip install "rfdetr[onnx,tflite]"` | TFLite export (Python 3.12 only) |
| `trt` | `pip install "rfdetr[trt]"` | TensorRT inference (pycuda, onnxruntime-gpu, tensorrt) |
| `kornia` | `pip install "rfdetr[kornia]"` | GPU-accelerated augmentations via Kornia |
| `lora` | `pip install "rfdetr[lora]"` | LoRA fine-tuning with PEFT |
| `visual` | `pip install "rfdetr[visual]"` | Visualization utilities (matplotlib, pandas, seaborn) |
| `cli` | `pip install "rfdetr[cli]"` | CLI with typed argument parsing (jsonargparse) |
| `plus` | `pip install "rfdetr[plus]"` | XLarge and 2XLarge detection models (PML 1.0 license) |

## Additional Notes

- Ensure you have Python 3.10 or higher installed.
Expand Down
90 changes: 76 additions & 14 deletions docs/getting-started/migration.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,12 +51,76 @@ See the [Changelog](../changelog.md) for the full list of changes in each releas
does **not** register the override and RF-DETR may still auto-align the schema from the
checkpoint. Always pass it to the constructor.

### Planned for Removal in v1.9

The following APIs were deprecated in earlier releases and will be removed in v1.9. They still work in the current release (v1.8.x) but emit `DeprecationWarning`. Update your code before upgrading.

!!! warning "Planned for removal: `rfdetr.util.*` and `rfdetr.deploy.*` import paths"

Deprecated since v1.6. Use the canonical replacements listed in the [Upgrade 1.5 → 1.6](#upgrade-15--16) section.

```python
# These imports still work in v1.8 but emit DeprecationWarning; update before v1.9
from rfdetr.util.coco_classes import COCO_CLASSES # → rfdetr.assets.coco_classes
from rfdetr.util.misc import get_rank # → rfdetr.utilities
from rfdetr.deploy import export_onnx # → rfdetr.export.main
```

!!! warning "Planned for removal: `build_namespace(model_config, train_config)`"

Deprecated since v1.7. Use `build_model_from_config` and `build_criterion_from_config` instead.

!!! warning "Planned for removal: `load_pretrain_weights(nn_model, model_config, train_config)` with `train_config`"

Deprecated since v1.7. Drop the `train_config` positional argument.

!!! warning "Planned for removal: `start_epoch` kwarg in `train()`"

Deprecated since v1.7. PyTorch Lightning resumes automatically via `resume=`.

!!! warning "Planned for removal: `do_benchmark` kwarg in `train()`"

Deprecated since v1.7. Use the `rfdetr.export.benchmark` module instead.

!!! warning "Planned for removal: `callbacks` dict kwarg in `train()`"

Deprecated since v1.7. Pass PTL `Callback` objects directly via the Lightning API instead.

!!! warning "Planned for removal: misplaced config fields"

The following `TrainConfig` and `ModelConfig` fields moved to their correct config class in v1.7 and the deprecated compatibility shims will be removed in v1.9. Update any direct references:

| Field | Removed from | Use in |
| ------------------- | ------------- | ------------- |
| `group_detr` | `TrainConfig` | `ModelConfig` |
| `ia_bce_loss` | `TrainConfig` | `ModelConfig` |
| `segmentation_head` | `TrainConfig` | `ModelConfig` |
| `num_select` | `TrainConfig` | `ModelConfig` |
| `cls_loss_coef` | `ModelConfig` | `TrainConfig` |

---

## Upgrade 1.7 → 1.8

### Breaking changes

!!! note "Breaking in v1.8.2: default keypoint schema changed to active-first `[17]`"

New checkpoints created from v1.8.2 onwards use `class_id=0` for person. Legacy `[0, 17]` checkpoints
are still supported — RF-DETR auto-detects the schema from the checkpoint at load time.

If your post-processing code offsets class IDs by 1 (common for background-first models), update it:

```python
# Before (background-first [0, 17]: person was at class_id=1)
class_name = "person" if detection.class_id == 1 else "other"

# After (active-first [17]: person is at class_id=0)
class_name = "person" if detection.class_id == 0 else "other"
```

Use `detection.data["class_name"]` for schema-agnostic name resolution.

!!! warning "Breaking: `rfdetr.datasets.aug_config` renamed to `rfdetr.datasets.aug_configs`"

The augmentation config module was renamed (singular → plural). If you import from it directly:
Expand Down Expand Up @@ -203,23 +267,21 @@ See the [Changelog](../changelog.md) for the full list of changes in each releas

!!! note "Deprecated: `RFDETRSegPreview` replaced by size-specific segmentation classes"

````
**`RFDETRSegPreview`** defaulted to the small variant and is replaced by size-specific
segmentation classes. If you used `RFDETRSegPreview()` without arguments, switch to
`RFDETRSegSmall()`.
**`RFDETRSegPreview`** defaulted to the small variant and is replaced by size-specific
segmentation classes. If you used `RFDETRSegPreview()` without arguments, switch to
`RFDETRSegSmall()`.

```python
# Before (deprecated)
from rfdetr import RFDETRSegPreview
```python
# Before (deprecated)
from rfdetr import RFDETRSegPreview

model = RFDETRSegPreview()
model = RFDETRSegPreview()

# After — pick one
from rfdetr import RFDETRSegNano, RFDETRSegSmall, RFDETRSegMedium, RFDETRSegLarge
# After — pick one
from rfdetr import RFDETRSegNano, RFDETRSegSmall, RFDETRSegMedium, RFDETRSegLarge

model = RFDETRSegSmall()
```
````
model = RFDETRSegSmall()
```

---

Expand Down Expand Up @@ -369,5 +431,5 @@ model = RFDETRSegSmall()
from rfdetr import OPEN_SOURCE_MODELS

# After
from rfdetr import ModelWeights
from rfdetr.assets.model_weights import ModelWeights
```
2 changes: 1 addition & 1 deletion docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,7 @@ RF-DETR achieves the best accuracy–latency trade-off among real-time object de
RF-DETR (Roboflow Detection Transformer) is a real-time object detection and instance segmentation model from Roboflow, accepted at ICLR 2026. It uses a DINOv2 vision transformer backbone and achieves state-of-the-art accuracy–latency trade-offs on COCO (60.1 AP50:95 for RF-DETR-2XL) and RF100-VL.

**How does RF-DETR compare to YOLOv11?**
RF-DETR-L achieves 56.5 AP50:95 on COCO at 6.8 ms latency on an NVIDIA T4, outperforming YOLOv11x (54.7 AP) at lower latency. The DINOv2 backbone gives RF-DETR stronger performance on domain-shift benchmarks such as RF100-VL.
RF-DETR-L achieves 56.5 AP50:95 on COCO at 6.8 ms latency on an NVIDIA T4, outperforming YOLOv11x (50.9 AP) at lower latency. The DINOv2 backbone gives RF-DETR stronger performance on domain-shift benchmarks such as RF100-VL.

**What GPU is required to train RF-DETR?**
A CUDA-capable GPU with at least 8 GB VRAM (e.g., NVIDIA RTX 3060, T4, A10) is recommended for fine-tuning. Smaller models (RF-DETR-N and RF-DETR-S) can fit in 6 GB VRAM with reduced batch size. CPU inference is supported for evaluation.
Expand Down
2 changes: 1 addition & 1 deletion docs/learn/benchmarks.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ description: RF-DETR benchmark results on COCO and RF100-VL for detection, segme
!!! tip "Key Takeaways"

- RF-DETR-2XL achieves 60.1 AP50:95 on COCO detection at 17.2 ms latency (T4, TensorRT FP16)
- RF-DETR-L outperforms YOLOv11x (56.5 vs 54.7 AP50:95) at lower latency (6.8 vs 10.5 ms)
- RF-DETR-L outperforms YOLOv11x (56.5 vs 50.9 AP50:95) at lower latency (6.8 vs 10.5 ms)
- Segmentation models range from 40.3 AP (Nano, 3.4 ms) to 49.9 AP (2XL, 21.8 ms) on COCO
- All latency measured on NVIDIA T4 with TensorRT 10.4, CUDA 12.4, FP16, batch size 1
- RF100-VL results demonstrate strong domain-shift generalization across 100 diverse datasets
Expand Down
31 changes: 10 additions & 21 deletions docs/learn/export.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ description: Export RF-DETR models to ONNX, TensorRT, and TFLite (FP32/FP16/INT8
- Export to TFLite (FP32, FP16, INT8) for mobile and edge deployment
- TensorRT conversion delivers lowest latency on NVIDIA GPUs (2.3 ms for Nano)
- INT8 quantization requires calibration data from your dataset for accurate results
- Custom input resolutions supported (must be divisible by 14)
- Custom input resolutions supported (must be divisible by `patch_size × num_windows`, which varies by model variant)

RF-DETR supports exporting models to ONNX and TFLite formats, enabling deployment across a wide range of inference frameworks, edge devices, and hardware accelerators.

Expand Down Expand Up @@ -63,14 +63,15 @@ The `export()` method accepts several parameters to customize the export process
| `quantization` | `None` | TFLite quantization mode: `None`/`"fp32"`, `"fp16"`, or `"int8"`. Only used when `format="tflite"`. |
| `calibration_data` | `None` | Calibration data for TFLite export. Image directory, `.npy` file path, NumPy array, or `None`. See [TFLite Export](#tflite-export). |
| `max_images` | `100` | Maximum number of images to load from a calibration directory for TFLite INT8 quantization. Ignored for other calibration data formats. |
| `infer_dir` | `None` | Path to an image file to use for tracing. If not provided, a random dummy image is generated. |
| `simplify` | `False` | Deprecated and ignored. ONNX simplification is no longer run by `export()`. |
| `infer_dir` | `None` | Optional directory of sample images for inference validation during export tracing. If not provided, a random dummy image is generated. |
| `backbone_only` | `False` | Export only the backbone feature extractor instead of the full model. |
| `opset_version` | `17` | ONNX opset version to use for export. Higher versions support more operations. |
| `verbose` | `True` | Whether to print verbose export information. |
| `force` | `False` | Deprecated and ignored. |
| `shape` | `None` | Input shape as tuple `(height, width)`. Each dimension must be divisible by the selected model's block size (`patch_size * num_windows`). If not provided, uses the model's default resolution. |
| `batch_size` | `1` | Batch size for the exported model. |
| `dynamic_batch` | `False` | If `True`, export with a dynamic batch dimension so the ONNX model accepts variable batch sizes at runtime. |
| `patch_size` | `None` | Backbone patch size override. Defaults to the value from `model_config.patch_size`. Must match the instantiated model's patch size when provided. |
| `notes` | `None` | Optional user-defined metadata (string, dict, list, or any JSON-serialisable value) to embed in the exported ONNX model under the `"rfdetr_notes"` metadata property. |

## Advanced Export Examples

Expand All @@ -84,18 +85,6 @@ model = RFDETRMedium(pretrain_weights="<path/to/checkpoint.pth>")
model.export(output_dir="exports/my_model")
```

### Deprecated: Export with Simplification

The `simplify` flag is deprecated and ignored:

```python
from rfdetr import RFDETRMedium

model = RFDETRMedium(pretrain_weights="<path/to/checkpoint.pth>")

model.export(simplify=True) # Deprecated: same result as model.export()
```

### Export with Custom Resolution

Export the model with a specific input resolution. For example, `RFDETRMedium` expects dimensions divisible by `32` (`patch_size=16`, `num_windows=2`):
Expand Down Expand Up @@ -201,9 +190,9 @@ pip install "rfdetr[onnx,tflite]"
=== "Object Detection"

```python
from rfdetr import RFDETRBase
from rfdetr import RFDETRSmall

model = RFDETRBase()
model = RFDETRSmall()

model.export(format="tflite", output_dir="output")
```
Expand Down Expand Up @@ -266,10 +255,10 @@ Prepare calibration data as a NumPy array and save it to a `.npy` file:
```python
import numpy as np
from PIL import Image
from rfdetr import RFDETRBase
from rfdetr import RFDETRSmall

model = RFDETRBase()
target_resolution = model.resolution
model = RFDETRSmall()
target_resolution = model.model_config.resolution

# Load representative images from your dataset
images = []
Expand Down
5 changes: 3 additions & 2 deletions docs/learn/run/detection.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,9 +117,10 @@ These examples use OpenCV for decoding and display. Replace `<SOURCE_VIDEO_PATH>

model = RFDETRMedium()

video_capture = cv2.VideoCapture("<WEBCAM_INDEX>")
WEBCAM_INDEX = 0
video_capture = cv2.VideoCapture(WEBCAM_INDEX)
if not video_capture.isOpened():
raise RuntimeError("Failed to open webcam: <WEBCAM_INDEX>")
raise RuntimeError(f"Failed to open webcam: {WEBCAM_INDEX}")

while True:
success, frame_bgr = video_capture.read()
Expand Down
13 changes: 10 additions & 3 deletions docs/learn/train/advanced.md
Original file line number Diff line number Diff line change
Expand Up @@ -292,12 +292,19 @@ model.train(dataset_dir="path/to/dataset", aug_config={})

### Gradient Checkpointing

For large models or high resolutions, enable gradient checkpointing to trade compute for memory:
For large models or high resolutions, enable gradient checkpointing to trade compute for memory.

!!! warning "Constructor parameter — not a `train()` parameter"

`gradient_checkpointing` is a `ModelConfig` field and must be passed to the **model constructor**, not to `train()`. Passing it to `train()` will raise a `ValidationError` because `TrainConfig` has `extra="forbid"`.

```python
from rfdetr import RFDETRMedium

model = RFDETRMedium(gradient_checkpointing=True)

model.train(
dataset_dir="path/to/dataset",
gradient_checkpointing=True,
batch_size=2, # May be able to increase with checkpointing
)
```
Expand Down Expand Up @@ -355,7 +362,7 @@ These are automatically configured and don't require manual setup.
If you encounter CUDA out of memory errors:

1. Reduce `batch_size`
2. Enable `gradient_checkpointing=True`
2. Enable `gradient_checkpointing=True` (pass to the model constructor, not `train()`)
3. Reduce `resolution`
4. Increase `grad_accum_steps` to maintain effective batch size

Expand Down
Loading
Loading