The Edge Cases That Have Delayed Autonomous Driving for Years

When an autonomous vehicle enters traffic, it spends 99.9% of its life doing the same things over and over: moving along its lane, stopping at red lights, keeping its distance from the car ahead. But here is the real question: what happens in those strange moments that occur in the tiny sliver of remaining time, once in millions of kilometers? These are exactly the moments that delay the "ready tomorrow" promise of autonomous driving - and in this article I will explain why they still have not been solved.
What Is an Edge Case, and Why Is the Long Tail Something Else?
First let's clarify the terminology, because these two concepts are often confused. An edge case is a rare scenario that falls outside an autonomous driving system's training data: a ladder tipped over on the highway, an unusual traffic officer's hand signal, an extraordinary weather condition. These can be defined in advance, at least in theory.
The long tail, on the other hand, is a more chilling concept: the sum of everything you could not even think of. Roadworks that change overnight, a mattress rolling off a pickup truck in the rain, that scenario at the very end of the tail that no one ever thought to write on a problem list. While an edge case is a puzzle, the long tail is like a box where you do not know how many pieces the puzzle has.
Rarity comes in varieties, too. The first is rare classes: objects such as donkeys or ambulances that are rarely seen on the roads. The second, and more insidious one, is rare scenarios: the objects themselves are perfectly ordinary, but their arrangement is strange - a pedestrian suddenly emerging from behind a truck, or a cyclist carrying a ladder. Finding the second type is much harder, because every component is already defined in the system; the problem is how they come together.
Experts divide edge cases into four main categories: environmental (weather, lighting conditions), object-based (rare objects), behavioral (unpredictable road users) and sensor-specific (hardware failures, blind spots). Each category is a headache on its own; together, they explain why the tail is so long.
Highlights
- A system working with 99% accuracy can encounter a dangerous scenario it cannot solve roughly once every 10,000 miles of driving.
- While a common dataset has 493,322 annotations for cars, there are only 49 annotations for ambulances.
- The 30 rarest object classes each appear fewer than 250 times in 344,966 frames of publicly available labeled data.
- If autonomous systems drove like humans, they could prevent only about one third of all accidents.
Rarity in Numbers: Why Is This Tail So Long?

Grasping the scale of rarity requires a bit of math. 99% accuracy sounds great, doesn't it? But do the math: at that level of accuracy, a system falls into a dangerous scenario it cannot solve roughly once in 10,000 miles of driving. Even for daily use, this number is huge - and for full autonomy it is catastrophic, because the safety threshold requires ultra-low error rates.
The picture on the data side is even more striking. The SearchAD benchmark, a joint work by Mercedes-Benz and Esslingen University, examined 423,798 frames from 11 driving datasets: 90 rare categories, 513,265 manual annotations. The result? Each of the 30 rarest classes appears fewer than 250 times in 344,966 frames of publicly available labeled data. At a specialist firm that manually searched for these rare cases, the job took eight months, an average of more than one minute was spent on each frame, and every frame was reviewed at least twice.
| Object Class | Number of Annotations in the Sample Dataset |
|---|---|
| Car | 493,322 |
| Stroller | 1,072 |
| Ambulance | 49 |
This table shows the skewness of the world itself: cars are everywhere, ambulances are almost nowhere. And the problem is not just discovery; label-based search only returns predefined classes. If no one labeled a pedestrian wearing a garbage bag in the rain, no query searching for that data can find it. Aggregate success metrics can also hide regressions in rare classes inside a strong overall score.
Can't We Just Collect More Data?
The answer is a resounding no for the vast majority, and the reason is a one-sentence truth: data skew is the skew of the world. The overwhelming majority of driving data collected from roads is filled with mundane behavior - lane keeping, obeying traffic lights, straight roads. When you double the dataset, you have mostly added twice as many mundane examples; the rare ones get buried even deeper in the queue. This is the classic diminishing returns problem: scaling compute and data does not deliver sufficient yield at ultra-low error rates.
So what if we recreated rare scenarios on stage? That's where the second wall stands: most rare cases are dangerous or impossible. Choreographing a pedestrian who will emerge from behind a truck is out of the question. Moreover, the queue is open-ended - it has no end point, meaning the moment to say "we've completed them all" never comes.
Simulation is therefore a brilliant assistant, but not a savior. On the real road, no amount of testing can guarantee safety on its own; besides, artificially generating rare events is itself a hard engineering problem. Waymo's approach is remarkable here: the team reconstructs collision scenarios that proved fatal in the real world and places the Waymo Driver in place of the vehicle that initiated the collision. In simulation, it avoided all the collisions. Impressive - but succeeding on defined scenarios does not mean exhausting the entire queue.
How Are Manufacturers Approaching the Same Problem?
Mobileye: AI that hunts down failures
The two vehicles Mobileye introduced in 2026 bring a systematic solution to the problem. Meteor, a multi-agent AI data analyst, scans millions of hours of driving video using vision-language model embeddings and automated reasoning. And what it is looking for is interesting: singular miracles called black swans, but repeatable failures - recurring patterns like partially obscured pedestrians, ambiguous road users. Discovery, hypothesis, scenario querying, validation and training data chain is processed automatically. The companion system Genario turns every validated failure into photorealistic synthetic scenarios and multiplies them with night/day, rain, snow, ice and fog variations. Behind all this lies more than 25 years of experience, over 230 million vehicles, and daily data uploads from 8 million vehicles.
Waymo: Limited area, strict rules
Waymo's strategy is not to shorten the queue but to shrink the area the queue can enter. Its so-called Operational Design Domain (ODD); is defined by geography, road type, speed range, weather and time of day. According to the company's regulatory filings, its target area includes roads with speed limits of up to 65 mph. The vehicles operate in full autonomy; remote human operators do not drive the vehicle with a joystick, they only provide high-level guidance. In October 2026, it also published a new framework for incident management drills to prepare for unexpected situations. The message from the management side is clear: autonomous vehicles can dramatically reduce accidents but cannot reach 100%, and it is necessary to speak openly and honestly about this. Company data shows that the number of accidents per million miles of its robotaxis is much lower than in previous years. Another observation is that ODD definitions and safety concepts can be reused across multiple cities - this flexibility also calls into question the boundaries of the field.
Tesla and NVIDIA: Two different bets
Tesla has bet that millions of vehicles and billions of miles of real-world data will, over time, illuminate the queue. In Full Self-Driving (Supervised) releases, users report fewer interventions, and according to the company, FSD reduces the likelihood of collisions caused by human error by more than 80% - but this is the company's own data, and independent verification is disputed; Tesla has never published official disengagement data. User reports, though anecdotal, are full of clues: a forced disengagement in front of a lane-changing truck, or a system approaching a construction truck stopped in the left lane at high speed.
NVIDIA, on the other hand, looks in the opposite direction: reasoning through unknown situations with simulation and generalizable intelligence. The Alpamayo-R1 project showed that the model, trained on reasoning data, improved planning accuracy in difficult cases by up to 12% and reduced near-collisions by 35%. Neither philosophy - just like Mobileye's hypothesis-driven data mining and Wayve's end-to-end AI approach - has yet been proven as a definitive solution. No one has seen the end of the queue.
Remote Human Intervention: Safety Net or Vulnerability?
The practical assurance of robotaxi companies is teleoperation: a remote human operator stepping in when the system is strained or fails. A US Senate hearing revealed that robotaxi companies rely on precisely this. There was already an industry-wide belief that Cruise robotaxis frequently required intervention from remote operators.
But this safety net has a ladder. In the Waymo model, the remote operator provides only high-level guidance; it does not take over the wheel. In Tesla's Austin robotaxi launch, it is stated that remote support staff can take temporary control at speeds of 2 mph and below and can drive the vehicle remotely, and a 2025 report from a state transportation agency documents that Tesla has adopted remote operation for rare edge cases and challenging environments. Low-latency 5G and 6G connections make these models technically possible. The question is: is it a real solution for humans to solve the scenarios the system cannot, or are we simply moving the problem into an invisible queue?
What Does the Law Say About the Gap Between Promise and Reality?
Here things get serious, because a crash in a rare edge case raises not only a technical but a legal question. In the US, NHTSA published an update to autonomous driving system rules in April 2026. On the legal front, there is an interesting distinction: the vehicle complying with safety standards does not make it flawless; a negligence claim is an entirely different legal path, and plaintiff lawyers play precisely there. According to a legal guide prepared in Texas, the argument used in lawsuits is the difference between the company's promise and what the car actually does. The technical risk highlighted by Brookings ties into the same point: unpredictable errors in perception and prediction systems can cause the vehicle to do the wrong thing at the wrong time.
The levels in SAE's J3016 standard also lay the promise-versus-reality question on the table. On the scale extending from Level 0 to Level 5, the overwhelming majority of vehicles on the roads are still at Level 1 or Level 2. Level 3 creates a major shift of burden, moving responsibility from the driver to the manufacturer. As for Level 5, some experts think the way some experts do: vehicles that work everywhere a human can drive will never exist. Even a 2026 scientific paper proposes an additional context- and situation-based classification, arguing that the current levels do not reflect real-world capability.
Can the Queue Be Consumed? The Honest Answer The Honest Answer
Let me tell you in the clearest terms: Edge cases are statistically extremely rare but unpredictable scenarios that delay autonomous driving from reaching its promised full autonomy; neither endless data collection nor simulation alone can consume this open-ended queue. The optimistic side has real crashes in hand: Waymo data shows the number of crashes per million miles has genuinely dropped, and Mobileye's synthetic scenario generation systematically narrows the queue. The pessimistic side has facts in hand too: if you drive like people do, you can prevent only about a third of crashes; according to some safety organizations, full autonomy hasn't arrived yet - there are still regular failures at basic tasks. And the next debate is already at the door: if we somehow solve edge cases, how will we guarantee the accuracy of generalized AI models this time?
My read is this: autonomous driving is not a race, it's a conveyor-belt elimination process. Driving thousands of kilometers without a hitch every day impresses no one; what makes the difference is what you do at that strange moment that comes once in a million miles. And what you'll do at that moment isn't known until the very day you imagine that scenario. Do you think robotaxis will beat this queue, or will it always stay one step ahead? I'm curious to hear your take in the comments.
Sources
- Diagnosing the long tail: how Mobileye turns edge cases into targeted training (Mobileye Blog)
- Why AV Models Miss Rare Edge Cases (and How to Find Them) | Voxel51
- Edge Cases in Autonomous Driving: Detection and Handling Guide | Kognic
- Waymo simulated driving behavior in reconstructed fatal crashes (Scanlon et al., 2021, ScienceDirect)
- Preparing for the Unexpected: A New Framework for AV Incident Management Exercises (Waymo Blog, Oct 2026)
- Full Self-Driving (Supervised) Vehicle Safety Report (Tesla)
- Solving the long-tail with e2e AI (Wayve)
- SAE Levels of Driving Automation (SAE International)
- Self-driving vehicles could struggle to eliminate most crashes (IIHS)
- STATE DOT Policy and Regulatory Approaches (NCDOT, 2025)