Object detection

Many real applications need more than a label for the whole image. A self-checkout needs to find every item on the counter; a quality system must point to the scratch; a safety camera must count people in a danger zone. Object detection answers two questions at once: what objects are present, and where each one is.

A detector’s output is a list like this:

[
  {"class": "house", "confidence": 0.93, "box": [26, 73, 96, 86]},
  {"class": "car",   "confidence": 0.88, "box": [198, 136, 96, 44]},
  {"class": "tree",  "confidence": 0.81, "box": [147, 44, 58, 114]}
]

Each box is usually written as its top-left corner plus width and height (or as two corners), in pixels.

Is this box right? Intersection over union

A predicted box is never pixel-perfect. To decide whether it counts, we measure how much it overlaps the true (“ground-truth”) box:

IoU is 1 for a perfect match and 0 for boxes that do not touch. A common rule is that IoU ≥ 0.5 counts as a correct detection.

How detectors see an image

Early detectors ran a classifier over thousands of windows at different positions and sizes — accurate but slow. Modern single-stage detectors such as the YOLO (“you only look once”) family do everything in one pass: the network divides the image into a grid, and every cell predicts boxes, a confidence score and class probabilities at the same time.

Lower the threshold to 0.2 and noise appears: background cells with slightly higher confidence start to count. Raise it to 0.9 and even the tree disappears. Choosing the threshold is a business decision about which mistake — a false alarm or a miss — is more expensive.

Removing duplicates: non-maximum suppression

A detector typically fires several times for the same object — neighboring cells and differently shaped anchor boxes all see part of it. Non-maximum suppression (NMS) cleans this up.

Bounding box

The rectangle that encloses an object, given by corner coordinates or by center, width and height.

Confidence score

The detector’s belief that the box contains an object of the predicted class.

Anchor boxes

Predefined box shapes the network adjusts; different anchors specialize in tall, wide or small objects.

NMS threshold

The IoU above which a lower-scoring box is treated as a duplicate and removed.

mAP

Mean average precision: the area under the precision–recall curve, averaged over classes (and often over IoU thresholds from 0.5 to 0.95).

Measuring a detector

Detection is scored with precision and recall at a chosen IoU threshold. Sweeping the confidence threshold traces a precision–recall curve; the area under it is the average precision for a class, and averaging over classes gives mAP. The widely used COCO benchmark averages mAP over IoU thresholds from 0.5 to 0.95, rewarding tightly fitting boxes as well as correct ones.

from ultralytics import YOLO

model = YOLO("yolov8n.pt")                    # a small pretrained detector
results = model("street.jpg", conf=0.5)       # confidence threshold
for box in results[0].boxes:
    print(model.names[int(box.cls)], float(box.conf), box.xyxy.tolist())

You now know how computers turn pixels into numbers, detect patterns with convolutions, learn filters in CNNs, train robust classifiers and find objects. The rest of the course goes further — segmenting every pixel, reading poses and faces, understanding video, vision transformers and the diffusion models that generate images.