What is AI player tracking?

How it actually works

Three stages, and understanding them tells you exactly where it will break.

1. Detection

A neural network looks at a single frame and returns boxes around the things it recognises as people, plus a confidence score. It does this independently for every frame, with no memory of the frame before.

Detection is the mature part. On reasonable footage it works well, and it is mostly not where problems come from.

2. Association, or tracking proper

Boxes in frame 1 have to be matched to boxes in frame 2, so that player 7 stays player 7. This is done by predicting where each player should have moved and matching on position, plus appearance cues like kit colour.

This is where things go wrong. When two players overlap, the system has to decide which box is which when they separate. Get it wrong and the identities swap — an ID switch, the characteristic failure of every tracking system ever built.

3. Mapping to the pitch

Detections are positions in a camera image, not positions on a pitch. To say anything about real distances, the system must work out the relationship between the two, usually by finding pitch lines and markings and solving for the camera's viewpoint — a homography.

When that succeeds, image positions become metres. When it fails — no visible lines, unusual angle, heavy occlusion — an honest system reports positions without world coordinates rather than guessing, because a wrong distance is worse than no distance.

Where it reliably fails

Any vendor who will not tell you this is not being straight with you.

Players in the same kit crossing

The hardest problem in the field. Appearance is the main way to tell teammates apart, and teammates look identical by design. Two players who cross and separate can come back with their labels swapped, and no amount of model quality fully solves it.

Crowds and pile-ups

A ruck, a maul, a goalmouth scramble, a crowded netball circle. When bodies fully occlude each other for a sustained period, there is genuinely no information in the image to work with.

Low, shaky or zoomed footage

Tracking quality is largely a function of footage quality. A high, wide, static shot where players are well separated tracks well. Handheld footage from pitch level, where players overlap constantly and the camera moves, tracks badly. This is the single biggest factor you control.

The ball

Much harder than players. It is small, fast, frequently occluded, and at a distance it can be a handful of pixels indistinguishable from a bald head or a puddle of light. Ball tracking is best understood as tracing a path when it can, not as a continuous record.

What it is genuinely good for

Set aside the marketing and the useful applications are narrower and real.

Making one player followable

The best use by a distance. Asking a squad to "watch the right-back" on a wide shot mostly fails, because the eye follows the ball. A marker that stays attached to that player through the phase makes off-ball work visible — and off-ball work is both the most coachable thing in team sport and the least watched.

Showing team shape as a shape

Connecting tracked players with lines turns eleven individuals into a visible structure. You can see the back four compress and stretch, the gap between the lines open, the shape distort towards the ball. That is difficult to see unaided and obvious once drawn.

Following the ball on a small screen

On a wide shot viewed on a phone, the ball is a few pixels. Tracing its path keeps a long switch or a cross followable, which matters because the phone is where players actually watch.

Rough positional context

When pitch mapping succeeds, you get approximate real-world positions: how deep a line sat, how much width a team used. Treat these as indicative rather than measured, and be sceptical of any grassroots product presenting them as precise.

What it is not

Three things tracking is regularly implied to be and is not.

It is not tactical understanding. The system does not know what a press is, or that a run was badly timed. It produces positions. Every tactical conclusion is drawn by you, and a product claiming to identify tactical patterns automatically is doing something considerably less reliable than its detection.

It is not a data source you should trust unconditionally. Distance covered and sprint counts derived from broadcast-style footage carry real error, particularly when players leave frame. Professional tracking uses calibrated multi-camera rigs or wearables for a reason. Numbers from a single club camera are indicative, and presenting them to players as fact is a mistake.

It is not instant. Tracking is computationally heavy and runs on GPU hardware billed by the second. Expect it to be queued and processed rather than immediate — in ClipItCoach a tracking request becomes a background job that notifies you when the clip is ready. Anyone offering unlimited instant tracking at no cost has not thought about the bill.

Questions to ask before paying for it

• What footage was the demo shot on? Compare it honestly with what you record. A demo on broadcast footage tells you little about your handheld touchline video.
• Which sports is it trained for? Models are trained on particular sports, and performance drops on ones they have not seen. Field sports with clear markings are the well-served case.
• What happens when it gets it wrong? Can you correct an ID switch by hand, or is the output take-it-or-leave-it?
• Is it limited, and how? By clip length, by number of runs per month, by plan. GPU time costs money, so there is always a limit somewhere.
• Does it produce positions or conclusions? Positions are honest. Automatically generated tactical insight, at this price point, usually is not.

If you want the surrounding vocabulary, our video analysis glossary covers detection, occlusion, homography and the rest in one page.

Common questions

How does AI player tracking work?

In three stages. A neural network detects people in each individual frame and draws boxes around them. A tracking algorithm then associates those boxes across frames using predicted motion and appearance, so each player keeps a consistent identity. Finally the system may map image positions onto a pitch model using visible line markings, which is what allows positions to be expressed in metres.

How accurate is AI player tracking?

It depends far more on your footage than on the software. A high, wide, static camera where players are well separated tracks reliably. Handheld footage from pitch level tracks poorly. The characteristic failure is an ID switch, where two players in the same kit cross and come back with their labels exchanged, which no system fully avoids.

Can AI tracking follow the ball?

Partially. The ball is small, fast and frequently hidden by players, and at distance it can be only a few pixels across. Ball tracking works best understood as tracing the path where it can be seen rather than as a continuous record. It is still useful, because on a wide shot viewed on a phone the ball is otherwise very hard to follow.

Does player tracking work for all sports?

No. Models are trained on specific sports and their performance drops on unfamiliar ones. Field sports with clear line markings and separated players are the best-served case. Sports with sustained player pile-ups, such as rugby rucks, or crowded small areas, such as a netball goal circle, are genuinely hard for any tracking system.

Is AI tracking the same as telestration?

No. Telestration is manual drawing on footage, where you decide what to mark and the annotation carries your coaching judgement. Tracking is automatic detection and following of players, which keeps a marker attached to someone as they move. They work well together: tracking shows who and where, telestration says what should have happened.