Training finishes and the terminal shows an mAP50 (mean average precision at an IoU threshold of 0.50) north of 0.9. The next day you go out to the field: the model calls a tree shadow a target, and the mannequin it spotted at 20 meters in one frame disappears in the next. The first reflex is almost always the same — bigger model, more epochs, more aggressive augmentation.

In our experience the problem was almost never there. In the first version of the dataset we built for SUAS 2026, the overwhelming majority of frames were ground-level photos scraped from the web; aerial frames made up less than 15% of the set, and the two classes were imbalanced roughly 7:1. A validation split carved randomly out of that set produced a high number, but that number told us next to nothing about flight performance.

In this post we treat the dataset like a software component: it has a contract (the labeling rules), it has test coverage (the validation set), and it can break silently (leakage — information bleeding from training into test). The numeric targets come from the Ultralytics documentation; the structural decisions come from our own time in the field.

Anatomy of a good dataset

Ultralytics' "Tips for Best Training Results" guide recommends at least 1500 images per class and at least 10,000 labeled instances per class for production quality. For a student team those numbers are usually out of reach; what matters is knowing which axis you are closing the gap on.

Priority matters too: for us, 500 well-labeled aerial frames turned out to be worth more than 5000 badly labeled web photos.

Labeling: tight boxes, written rules

Ultralytics states three ground rules plainly: every instance of every class in an image must be labeled, boxes must wrap the object with no slack, and no object may be left unlabeled. Partial labeling does not work, because an unlabeled object tells the model "there is nothing here."

The label file is a .txt file carrying the same name as the image; each line is one object, and the coordinates are x_center y_center width height normalized to the 0-1 range. Class indices start at zero.

# labels/train/flight_042_0187.txt
0 0.412500 0.633333 0.031250 0.055556
1 0.784375 0.211111 0.048437 0

# labels/train/flight_042_0187.txt
0 0.412500 0.633333 0.031250 0.055556
1 0.784375 0.211111 0.048437 0

The format is not the hard part — consistency is. If two people label the same frame and draw different boxes, the model learns noise. So we collected our decisions on a single written rules page:

Every week of labeling you spend without a rules page comes back later as a week of relabeling.

Class balance: 7:1 punishes you quietly

Our class ratio was roughly 7:1. When that happens, the overall mAP is dominated by the performance of the majority class and you never notice that the minority class is doing badly. The simplest diagnostic: always read your metrics per class, and look at the per-class rows in the val output.

We went with targeted collection first and moderate oversampling second; merging classes was off the table because of the mission requirements.

Hard negatives: why you need frames with nothing in them

A hard negative is a frame the model mistakes for an object but that contains no target at all — a trash bag on grass, an awning that looks like a tent, a road sign shaped like a human silhouette. You add these frames to the dataset unlabeled: background images need no label file.

Ultralytics recommends background images at 0-10% of the total dataset; for reference, COCO contains roughly 1000 background frames, which is about 1% of the set. Higher ratios can cut false positives noticeably in some domains, but that is entirely domain-specific and only works with carefully chosen frames — piling on background images blindly will cost you recall.

The most valuable backgrounds are not collected at random; they are mined from the model's own mistakes. The workflow:

from ultralytics import YOLO

model = YOLO("runs/detect/train/weights/best.pt")
# Scan flight recordings we know contain no targets
results = model.predict("flights/clean_pass/", conf=0.25, stream=True)

for r in results:
    if len(r.boxes):                     # there should be no detection here -> FP
        r.save(f"hard_negatives/{r.path.stem}.jpg")
# These frames go into the dataset WITHOUT a label file.

from ultralytics import YOLO

model = YOLO("runs/detect/train/weights/best.pt")
# Scan flight recordings we know contain no targets
results = model.predict("flights/clean_pass/", conf=0.25, stream=True)

for r in results:
    if len(r.boxes):                     # there should be no detection here -> FP
        r.save(f"hard_negatives/{r.path.stem}.jpg")
# These frames go into the dataset WITHOUT a label file.

The key point: change this ratio by measurement too. If adding backgrounds doesn't reduce the FP count on your validation set, the frames you added weren't "hard" in the first place.

Splitting properly: a random split is a trap

The usual advice is 70/20/10 or 70/15/15 (train/validation/test). The ratio itself is secondary; what really matters is what you treat as the unit of the split.

Consecutive frames pulled out of a video are near-copies of one another. Split them randomly at the image level and frame 187 lands in training while frame 188 lands in validation. The model has already seen your validation frame; what you are measuring is memorization, not generalization. The literature describes this as leakage arising from spatio-temporal correlation in video-derived datasets, and it inflates metrics systematically.

The fix is to split groups, not frames: the same flight, the same video, the same scene all go to one side.

from sklearn.model_selection import GroupShuffleSplit

# groups: the flight/video ID each image belongs to
gss = GroupShuffleSplit(n_splits=1, test_size=0.30, random_state=42)
train_idx, rest_idx = next(gss.split(images, groups=groups))
# rest_idx is later split into val/test with the same logic

from sklearn.model_selection import GroupShuffleSplit

# groups: the flight/video ID each image belongs to
gss = GroupShuffleSplit(n_splits=1, test_size=0.30, random_state=42)
train_idx, rest_idx = next(gss.split(images, groups=groups))
# rest_idx is later split into val/test with the same logic

If you also have class imbalance, StratifiedGroupKFold preserves the class ratios while keeping the groups disjoint.

The validation set has to represent the environment you'll work in

A metric is only as meaningful as the distribution it measures. A high score on a validation set full of web photos tells you nothing about the scene an 81° HFOV gimbal camera sees from 100 meters of altitude.

Once we rebuilt the validation set from aerial frames only, the value we measured was mAP50 0.950. What makes that number valuable isn't its size but the fact that we know what it measures: the same model detected two targets from roughly 20 meters of altitude in a field test and completed an autonomous release. With a web-heavy validation set we could never have drawn that connection.

Practical rule: the validation and test sets must represent flight conditions — altitude range, time of day, ground texture, camera. Lock the test set the day you build it. The moment you look at it once to make a decision, it has become training data.

Data hygiene: duplicates and corrupt labels

When training starts, Ultralytics scans the dataset and prints a summary in the form ... images, ... backgrounds, ... corrupt; it also warns you about non-normalized or out-of-bounds coordinates and duplicate labels. Don't skip past that line — it can be hiding hundreds of silently dropped frames.

Our checklist:

# 1) Images with no label / labels with no image
# 2) Out-of-bounds coordinates (0-1 range violations)
python - <<'PY'
from pathlib import Path
for f in Path("labels/train").glob("*.txt"):
    for i, line in enumerate(f.read_text().split("\n")):
        if not line.strip(): continue
        c, *v = line.split()
        if any(not (0.0 <= float(x) <= 1.0) for x in v):
            print("OUT OF BOUNDS:", f.name, "line", i + 1)
PY

# 1) Images with no label / labels with no image
# 2) Out-of-bounds coordinates (0-1 range violations)
python - <<'PY'
from pathlib import Path
for f in Path("labels/train").glob("*.txt"):
    for i, line in enumerate(f.read_text().split("\n")):
        if not line.strip(): continue
        c, *v = line.split()
        if any(not (0.0 <= float(x) <= 1.0) for x in v):
            print("OUT OF BOUNDS:", f.name, "line", i + 1)
PY

A file hash is enough for byte-identical frames. For near-duplicates, FiftyOne Brain's compute_exact_duplicates() and compute_near_duplicates() functions make the job easy. When one copy of a scene sits in training and another sits in validation, your measurement loses its credibility.

Common traps

Practical summary

1. Pick the unit of the split first, the ratio second. Group by flight/video; use GroupShuffleSplit or StratifiedGroupKFold. 70/15/15 is a reasonable starting point.

2. Build the validation and test sets from flight imagery only. Web photos can stay in training; they can't stay in your measurement.

3. Put the labeling rules on a single page and validate them by having two people label the same frame and comparing the result. Boxes should wrap the object with no slack.

4. Add hard negatives at 1-5% of the total, mine them from the model's own false positives, and raise the ratio only on measured evidence (ceiling around 10%).

5. Run a cleanup pass before every training run: duplicate hash check, out-of-bounds coordinate scan, and a corrupt count of zero in the Ultralytics scan summary.

Sources: Ultralytics — Tips for Best Training Results, Ultralytics — Detection Datasets, Ultralytics — Data Collection and Annotation, Ultralytics — Preprocessing Annotated Data, scikit-learn — Cross-validation, FiftyOne Brain