Fine-tunes vision models - object detection (D-FINE, RT-DETR, DETR, YOLOS), image classification (timm/Transformers), and SAM/SAM2 segmentation - on Hugging Face Jobs GPUs with dataset validation and mAP/accuracy metrics.
---
name: huggingface-vision-trainer
description: Train and fine-tune vision models on Hugging Face Jobs cloud GPUs - object detection (D-FINE, RT-DETR v2, DETR, YOLOS), image classification (timm and Transformers classifiers), and SAM/SAM2 segmentation. Use when fine-tuning detectors, classifiers, or segmentation models on custom datasets.
---
# Vision Model Training on Hugging Face Jobs
Train detection, classification, and SAM/SAM2 segmentation models on managed cloud GPUs; results save to the Hub automatically.
## Workflow
1. Verify prerequisites: a paid HF plan (Jobs require it), a write-scoped token passed in job secrets, and an existing Hub dataset.
2. Confirm dataset shape - detection needs an `objects` column (bbox/category, xywh or xyxy auto-detected); classification needs `image` + `label`; SAM needs `image` + `mask` + a bbox/point prompt.
3. Always validate unknown datasets first with `uv run scripts/dataset_inspector.py --dataset <id> --split train` and estimate cost with `estimate_cost.py`.
4. Configure Albumentations augmentation and the right metrics (mAP/mAR for detection, accuracy for classification, DiceCE for SAM).
5. Submit the training job to HF Jobs with Trackio monitoring; report job ID and monitoring URL, and ensure the trained model is pushed to the Hub.
Full skill & source: https://github.com/huggingface/skills/tree/ea9a24f38a98baf2430dab3bcb31a89753ae9e6d/skills/huggingface-vision-trainer