Executive Summary: A rundown of different approaches for validation of self-driving and driver assist systems, and a recommendation to Tesla and others to have countermeasures to detect drivers not watching the road, and permanently disable their Autopilot if they show a pattern of inattention.

The recent fatality for a man who was allowing his car to be driven by the Tesla "autopilot" system has ignited debate on whether it was appropriate for Tesla to allow their system to be used as it was.

Tesla's autopilot is a driver assist system, and Tesla tells customers it must always be supervised by an alert driver ready to take the controls at any time. The autopilot is not a working self-driving car system, and it's not rated for all sorts of driving conditions, and there are huge numbers of situations that it is not designed to handle and can't handle. Tesla knows that, but the public, press and Tesla customers forget that, and there are many Tesla users who are treating the autopilot like a real self-driving car system, and who are not paying attention to the road -- and Tesla is aware of that as well. Press made this mistake as well, regularly writing fanciful stories about how Tesla was ahead of Google and other teams.

Brown, the driver killed in the crash, was very likely one of those people, and if so, he paid for it with his life. In spite of all the warnings Tesla may give about the system, some users do get a sense of false security. There is debate if that means driver assist systems are a bad idea.

There have been partial self-driving systems that require supervision since the arrival of the cruise control. Adaptive cruise control is even better, and other car companies have released autopilot like systems which combine adaptive cruise control with lane-keeping and forward collision avoidance, which hits the brakes if you're about to rear end another car. Mercedes has sold a "traffic jam assist" like the Telsa autopilot since 2014 that only runs at low speeds in the USA. You can even go back to a Honda demo in 2005 of an autopilot like system.

With cruise control, you might relax a bit but you know you have to pay attention. You're steering and for a long time even the adaptive cruise controls did not slow down for stopped cars. The problem with Tesla's autopilot is that it was more comprehensive and better performing than earlier systems, and even though it had tons of things it could not handle, people started to trust it with their lives.

Tesla's plan can be viewed in several ways. One view is that Tesla was using customers as "beta testers," as guinea pigs for a primitive self-drive system which is not production ready, and that this is too much of a risk. Another is that Tesla built (and tested) a superior driver assist system with known and warned limitations, and customers should have listened to those warnings.

Neither is quite right. While Tesla has been clear about the latter stance, with the knowledge that people will over-trust it, we must face the fact that it is not only the daring drivers who are putting themselves at risk, it's also others on the road who are put at risk by the over-trusting drivers -- or perhaps by Tesla. What if the errant car had not gone under a truck, but instead hit another car, or even plowed into a pedestrian when it careened off the road after the crash?

At the same time, Tesla's early deployment approach is a powerful tool for the development and quality assurance of self-drive systems. I have written before about how testing is the big unsolved problem in self-driving cars. Companies like Google have spent many millions to use a staff of paid drivers to test their cars for 1.6 million miles. This is massively expensive and time consuming, and even Google's money can't easily generate the billions of miles of testing that some feel might be needed. Human drivers will have about 12 fatalities in a billion miles, and we want our self-driving cars to do much better. Just how we'll get enough verification and testing done to bring this technology to the world is not a solved problem.

While we don't want a repeat of the things that led to the death of this driver, there may be great value to the field in having fleets of volunteers helping verify these vehicles and if it can be done safely, the world probably needs it. But even if there is no way to do volunteer testing of systems that are still being refined -- and they will still be in the being refined state for many years to come -- there is still the question of whether you can have systems that need human supervision and intervention that get so good that humans ignore warnings and stop bothering to supervise them.

Here's a look at a variety of approaches to testing, verifying, refining and learning:

In Simulator

The only way many billions of miles will be done is in simulator, especially since you want to re-test all new software builds as much as you can before deploying. In simulator you can test all sorts of risky situations you could never readily test in the real world. That's why many years ago I laid out a plan for an open source simulator because I think this would be a great thing for everybody to contribute to. An open source simulator would soon feature every crazy situation anybody has even seen or thought of, and it would make all cars safer.

But simulation is just simulation, and it only features what people have thought of. You need more.

On test tracks

The most conservative, but most expensive and limited approach, is a private test track. And almost everybody starts on a closed testing ground, because when you need to when things are just getting going. They give you the confidence to eventually go out on the road -- but go out on the road you must. You can really only do a fraction of a percent of your testing on test tracks, even with robot slave cars to drive along with. You just won't encounter enough variation in terrain and driving situations, and you can only test a few cars at a time.

On the plus side, on test tracks you can use unmanned cars, reducing some costs, and you can run 24 hours/day.

There is a hybrid of simulator and test track called "Vehicle in the Loop" which puts the car on a dyno in a building while robots whiz around to provide relative motion. That requires less real estate and is safer, but automated test track operation is probably better at this point.

On the road with trained professional safety drivers

All companies eventually need to go out on real roads and deal with real situations. They do it by having paid professionals, usually with special training, who sit behind the wheel watching the vehicle and ready to take the controls at any sign of doubt. It is also common to have a second person who is monitoring the computer systems and sensor scans to make sure all is operating properly, and to tell the safety driver to take over if something is wrong. In fact, safety drivers are required for testing self-driving cars (but not autopilots/cruise software) by most of the state laws about self-driving cars.

Companies like Google tell their safety drivers to take control at any sign of doubt. They don't want them to let the car keep driving to see if it would have braked properly for the children crossing the road. Later, they can use the simulator to replay what the car would have done if the safety driver had not intervened. If the car would have done something wrong, it's a bug to be fixed at high priority.

Still, even trained safety drivers can make mistakes. Google has reported one accident where their systems made a mistake. The system decided that a bus it was sharing the lane with would let it in to traffic, but the bus did not. The safety driver didn't intervene because he also judged the bus would let the car in.

In addition, when the systems are very good, safety drivers can also become complacent and not intervene as quickly as they should.

On the road with regular drivers but limited function

You could test a self-driving system by having it do only some of the driving. For example, you could make a system which steers but does not control the brakes. A driver who knows that she has to hit the brakes when approaching something is not going to stop paying attention. With this system you can note when the human hit the brakes, and see if the system also would have, and who was right. Chances are any surprise braking would be combined with taking the wheel. Of course you would also learn when the driver grabbed the wheel.

You could also offer only steering and leave both throttle and brakes to the driver. Or you could do mild braking -- the kind needed to maintain speed behind a car you're following -- but depend on the driver for harder braking, with hard braking also requiring taking of the wheel.

You can also do it in reverse -- the driver steers but the computer handles the speed. That's essentially adaptive cruise control, of course, and people like that, but again, with ACC you have the wheel and are unlikely to stop paying attention to the road. You will get a bit more relaxed, though.

Supervised autopilot with "countermeasures"

Some cars with autopilots or even lanekeeping take steps to make sure the driver is paying attention. One common approach is to make sure they touch the steering wheel every so often. Tesla does not require regular touches as some cars do, but does in certain situations ask for a touch or contact. In some cases, though, drivers who want to over-trust the system have found tricks to disable the regular touch countermeasure.

A more advanced countermeasure is a camera looking at the driver's eyes. This camera can notice if the driver takes his eyes from the road for too long, and start beeping, or even slowing the car down to eventually pull off the road.

Of course, those who want to over-trust the system hate the countermeasures, and find they diminish the value of the product. And most people who over-trust do get away with it, if the system is good, which reinforces that.

Disable the autopilot if it is not used diligently

One approach not yet tried in privately owned cars would be countermeasures which punish the driver who does not properly supervise. If a countermeasure alert goes off too often, for example, it could mean that the autopilot function is disabled on that car for some period of time, or forever. People who like the function would stay more diligent, afraid of losing it. They would have to be warned that this was possible.

Drivers who lose the function may or may not get a partial refund if they paid for it. Because of this risk, you would not want people who borrowed a car to be able to use the function without you present, which might mean having a special key used only by the owner which enables the autopilot, and another key to give to family, kids or friends.

Of course, for many, this would diminish the value of the autopilot because they were hoping to not have to be so diligent with it. For now, it may be wisest to diminish the value for them.

Deliberate "fake failures" countermeasure

Cars could even do a special countermeasure, namely a brief deliberate (but safe) mistake made by the system. If the driver corrects it, all is good, but if they don't, the system corrects it and sounds the alert and disables autopilot for a while -- and eventually permanently.

The mistakes must of course be safe, which is challenging. One easy example would be drifting out of the lane when there is nobody nearby in the adjacent lane. The car would correct when it got to the line (extra safe mode) or after getting a few inches into the unoccupied adjacent lane.

This would keep drivers trained to watch for steering errors, but you also want to have them watch for braking and speed errors as well. This presents some risk as you must get "too close" to a car ahead of you. To do that safely, the normal following distance would need to be expanded, and the car would every so often crowd to a closer, but still safe distance. It would also be sure it could brake (perhaps harder than is comfortable) if the car in front were to change speed.

The problem is that the legal definition of too close -- namely a 2 second headway -- is much wider a gap than most drivers leave, and if you leave a 2 second gap, other drivers will regularly move into it if traffic gets thick. Most ACCs let you set the gap from 1 to 2.5 seconds, but sometimes to even less. You might let the user set a headway, and train to know what is "too close" and demands an intervention.

To do this well, the car might need better sensors than an mere autopilot demands. For example, the Tesla's rear radars are perhaps not good enough to reliably spot a car overtaking you at very high speed, and so a fully automated lane change is not perfectly safe. However, in time the sensors will be enough for that. The combination of radars and blindspot-tracking ultrasonics are probably good enough to know it's safe to nudge a few inches into the other lane to test the driver.

This method can also be used to test the quality of professional safety drivers, who should be taken off the job and retrained if they fail it.

Rather than fake failures, you could just have a heads-up display which displays "intervene now" in the windshield for a second or two. If you don't take over -- beep -- you failed. That's a safe way to do this, but might well be gamed.

Not letting everybody try out the autopilot

A further alternative would be to consider that systems in Tesla's class -- which is to say driving assist systems which can't do everything but which do so much that they will lull some people into misusing them -- should perhaps only be offered to people who can demonstrate they will use them responsibly, perhaps by undergoing some training or test, and of course losing the function if they set off the countermeasure alarm too often.

Make customers responsible

Tesla used some countermeasures, but mostly took that path of stressing to the customer that their system was not a self-drive system and drivers should supervise it. Generally, letting people be free to take their own risks is the norm in free societies. After all, you can crash a car in 1,000 ways if you don't pay attention, and people are and should be responsible for their own behaviour.

Except, of course, for the issue of putting others on the road at risk. But is that an absolute? Clearly not, since lots of car features enable risky behaviour on the part of drivers which put others at risk. Any high performance sports car (including Tesla's "Ludicrous" mode) and the ability to drive 200mph is a function which presents potential risk for other road users as well as the driver. The same is true of distractions in the car, such as radios, phone interfaces and more. Still, we want to hold risk to others to a higher, though not absolute standard.

Just watch

You can get some value from having a system which requires the driver to drive completely manually, but compares what the self-drive system would have done to that, and identifies any important differences. This is easy to do and as safe as other manual driving -- actually safer because you can implement very good ADAS in such a system -- but the hard part is pulling out useful information.

The self-drive system will differ from a human driver quite a lot. You need automated tools to pull out the truly important differences and have them considered and re-run in simulator by a test engineer. To make this workable, you need good tools to only bother the test engineer with worthwhile situations. If the driver brakes suddenly or swerves suddenly and the self-drive system did not want to, that's probably useful information.

This requires a car with a full suite of self-drive sensors, as well as possibly a set of all-around cameras for the test engineer to study. The problem is that this is expensive, and since it's not giving the car owner much benefit, the car owner is unlikely to pay for it. So you could deploy that to a subset of your cars if you have a lot of cars on the road.

Watching is still always valuable, no matter what, and also offers training data for neural networks and updates to map data, so you want to do it in as many cars as you can.

Real beta tests

Tesla would probably assert they were not "testing" their autopilot using customers. They, and all the others makers of similar products, would say that they believed their products to be as safe as other such functions, as long as they were properly supervised by an alert driver. And their current safety record actually backs up that this is broadly true.

Some smaller startups may actually desire to release tools before they are ready, actively recruiting customers as true beta testers. This is particularly true for small companies who have thought about doing retrofit of existing cars. Larger companies want to test the specific configuration that will go on the road, and thus don't want to do retrofit -- with retrofit, a customer car will go out on the road in a configuration that's never been tried before.

This is the most risky strategy, and as such the beta testing customers must be held to a high standard with good countermeasures. In fact, perhaps they should be vetted before being recruited.

Testing, validation and training

While I've used the word testing above, there are many things you gain from on-road operation. Tesla would say they were not testing their system -- they tested it before deployment. It is intended only for supervised operation and was judged safe in that situation, like cruise controls or lanekeeping. On the other hand, lots of supervised operation does validate a system and also helps you learn things about it which could be improved or changed, as well as things about the road. Most importantly, it informs you of new situations you did not anticipate.

That is the value of this approach, with diligent supervisors. You can keep improving an autopilot system and watch how well it does. Some day you will find that it doesn't need many interventions any more. Once it gets to 200,000 miles or more per intervention, you have well surpassed the average human driver. Now you can claim to have a self-driving system, and perhaps tell users they can stop supervising. This is the evolutionary approach car makers intend, while the non-car companies (like Google, Tesla and others) seek to go directly to the full self-driving car.

The approaches are so different that the car companies made sure there was a carve-out in many of the laws that were passed to regulate self-driving cars. Cars that need supervision, that can't operate without it, are exempted from the self-driving car laws. They are viewed by the law as just plain old cars with driver assist.

Conclusion

I think that for Tesla and the other companies marketing high-end driver assist autopilots that require supervision, the following course might be best:

• Have a sensor package able to reliably detect that it is safe to wander a short distance into adjacent lanes
• Have sufficient countermeasures to assure the driver is paying attention, such as an eye tracking camera
• Consider having a "strange event" countermeasure where the vehicle drifts a bit out of its lane (when safe) and if the driver fails to correct it, the car does and signals a countermeasure alert.
• Too many countermeasure alerts and the drive assist function is disabled long term, with or without refund as indicated in the contract.