Object Tracking with OpenCV

Object tracking means locating the same object across successive frames of a video. Detection finds it once. Tracking follows it over time.

Why not just detect every frame? Speed. Tracking algorithms are faster than detection algorithms because they already know what the object looks like and where it was last seen. They use appearance, location, and motion to predict where it'll be next.

I tested every OpenCV tracker on a Charlie Chaplin clip -- tracking his face as he dances and turns. A tough test because the face disappears briefly when he turns away.

Setup

brew install opencv
pip3 install numpy

import cv2

tracker_types = ['BOOSTING', 'MIL', 'KCF', 'TLD', 'MEDIANFLOW', 'CSRT', 'MOSSE']

Select the region of interest (the face) in the first frame:

bbox = cv2.selectROI(frame, False)

The trackers, compared

BOOSTING

Based on online AdaBoost (the algorithm behind Haar cascade face detectors). Uses the initial bounding box as a positive example, surrounding patches as negatives. Each frame, it scores nearby pixels and moves the box to the highest score.

Verdict: Outdated. No reason to use it when MIL and KCF exist. Fails when Chaplin's face disappears. Doesn't know when it's lost the target.

MIL

Similar idea to BOOSTING, but smarter about training data. Instead of treating only the exact current location as positive, it samples a neighborhood of patches and treats the whole bag as potentially positive. This is Multiple Instance Learning -- only one sample in the "positive bag" needs to actually contain the object.

Pros: Better accuracy than BOOSTING. Handles partial occlusion. Doesn't drift as much.

Cons: Doesn't reliably report tracking failure. Can't recover from full occlusion.

The closest result I got on the Chaplin clip, though the tracker still loses the face during the turn.

KCF (Kernelized Correlation Filters)

Builds on MIL's approach but exploits the overlapping regions between positive samples for mathematical efficiency. Faster and more accurate than MIL.

Pros: Best balance of speed and accuracy. Reports tracking failure better than BOOSTING and MIL. My recommendation for most applications (OpenCV 3.1+).

Cons: Can't recover from full occlusion.

409 FPS -- the fastest tracker. Loses the object during occlusion but excels in real-time scenarios.

TLD (Tracking, Learning, Detection)

Splits the problem into three parts: short-term tracking, learning from errors, and detection to recover when tracking fails. Can re-acquire a lost target.

Pros: Handles occlusion across multiple frames. Tracks through scale changes.

Cons: Lots of false positives. Tends to jump to similar-looking objects. Almost unusable in crowded scenes.

MEDIANFLOW

Tracks in both forward and backward directions and measures the discrepancy. If forward tracking and backward tracking disagree, it knows the track is unreliable.

Pros: Excellent at reporting failure. Very reliable when motion is predictable and smooth.

Cons: Fails under large, sudden motion.

Acceptable speed-to-accuracy trade-off. Not as accurate as MIL, not as fast as KCF.

MOSSE (Minimum Output Sum of Squared Error)

Uses adaptive correlation filters initialized from a single frame. Robust to lighting changes, scale, pose, and non-rigid deformation. Detects occlusion via peak-to-sidelobe ratio, allowing it to pause and resume.

Pros: Extremely fast (450+ FPS). Easy to implement.

Cons: Sacrifices accuracy for speed. Loses objects even during normal movement. Lags behind deep learning trackers on accuracy.

2671 frames processed, but with frequent tracking loss.

CSRT (Discriminative Correlation Filter with Channel and Spatial Reliability)

Uses a spatial reliability map to adjust the filter to non-rectangular regions. Works with standard features (HoGs and color names). Slower (25 FPS) but more accurate.

Pros: Highest accuracy among OpenCV trackers. Handles non-rectangular objects well.

Cons: Slowest of the group. Not suitable for real-time applications that need high FPS.

Which one to pick

There's no universally best tracker. It depends on your constraints:

• Need speed? KCF or MOSSE.
• Need accuracy? CSRT.
• Need occlusion recovery? TLD (accept the false positives).
• Need reliable failure detection? MEDIANFLOW.
• General purpose? Start with KCF.

For anything beyond these classical approaches, deep learning-based trackers (GOTURN, SiamFC) offer better accuracy at higher computational cost.

Source Code: https://github.com/ehsangazar/OpenCV-Object-Tracking