Roadshow explains the SAE’s new self-driving testing guidelines – Roadshow

Unless you work in the automotive industry, odds are pretty good you’re not familiar with the Society of Automotive Engineers (SAE). That said, if you’ve ever been on the road in a car, you’ve been directly affected by the organization’s standards and guidelines.

The SAE isn’t a government arm, and as such, it doesn’t carry any legislative power. However, the organization’s guidelines are generally recognized and adhered to by the automotive and aerospace industries. If you’ve ever bought oil for your car, then you’ve likely experienced the fruits of their guidelines for the makeup of the detergents anti-wear chemicals in its additive packages.

These days, the SAE matters that get talked about the most are the ones related to self-driving cars. The most famous (infamous?) of these topics is “SAE Levels of Autonomy,” a scale we often reference here at Roadshow.

But the SAE doesn’t just seek to define the different levels of vehicular automation on the road to full autonomy. The organization has other ideas on the subject. Specifically, in 2015, the SAE published a paper with the totally unsexy title of J3018-201503. This document outlined what the SAE felt were the appropriate precautions and procedures for companies to follow when conducting self-driving vehicle tests on public roads.

As you might imagine, a lot has changed since 2015. Today, we have more self-driving vehicle developers than ever, thanks to a flood of money from tech companies, legacy car companies and venture capitalists all trying to be first in the race to commercialize self driving. More developers mean more testing, and more testing means more opportunities for things to go wrong, as they did with Uber’s infamous fatal crash in Arizona in 2018.

To help steer the auto and tech industries towards safer testing practices and safer vehicles, the SAE recently released a substantially updated version of its original J3018 document in September. We got our hands on a copy, and we’re going to simplify some of the language and shine a light on what the SAE believes AV testing should look like.

———-

Because J3018 is a technical paper aimed at the automotive industry and not at consumers, it uses a bunch of acronyms to help keep its size down to a relatively palatable 11 (yet extremely dry) pages. The abbreviations are handy, though, and worth breaking down, so that’s where we’ll start:

Automated Driving System (ADS): This is the SAE’s term for the hardware and software that work together to make varying levels of automation at Levels 3, 4 or 5 possible. We like this as a blanket term for all of this tech, as it otherwise tends to get complicated when broken out into individual systems or branded terms.

Dynamic Driving Task (DDT): This is what the SAE calls driving, but more than that, DDT encompasses all of the physical processes and calculations necessary for making driving safe. DDT gets further broken down into operational and tactical processes. Steering, acceleration, and braking are all operational tasks, for example, while maneuver planning and things like signaling and turning on headlights for more visibility are considered tactical.

Driver Support: These are the advanced driver assist systems (ADAS) that make up Level 1 and Level 2 hardware packages common on new cars today. Such equipment includes things like blind spot monitoring and adaptive cruise control.

Operational Design Domain (ODD): This is a big one. ODD is the SAE’s term for the total  operating conditions (e.g. geographic location plus weather conditions plus time of day, plus state of traffic, plus road condition, etc.)  in which an automated vehicle is meant to operate. Thus, the functionality and effectiveness of Level 3 and 4 systems are extremely dependant on their ODDs, with 4 being less so than 3. True Level 5 vehicles can operate in any ODD without human intervention.

In-vehicle Fallback Test Driver (IFTD): Nope, IFTD isn’t a Meyers-Briggs result gone wrong; it’s the SAE’s designation for a human safety driver. IFTD is a good designation because it’s much clearer as to what the test driver’s responsibility is and is not. The driver is a fallback, meaning that if the Automated Driving System packs it in, the vehicle can fallback onto the test driver to retake control. 

There are plenty of other terms and jargon that are featured in J3018, but these are the most important concepts, and we feel like the industry adopting a shared vocabulary like this one would be extremely helpful.

—————

Next, we should talk about the SAE’s ideas for what traits, skills and abilities an ideal IFTD possesses. Again, because this document calls for standardization in what is largely an underregulated industry, there are some pretty fundamental considerations listed, including said individual possessing a valid driver’s license, for example.

Other requirements are less straightforward, but still important. What’s more important still is the SAE’s call to have IFTDs broken down into two groups: Early-stage prototype test drivers and late-stage prototype test drivers. 

Early-stage test drivers are routinely asked to deal with vehicles and systems which are still in development, and therefore more prone to failure. These drivers need to have greater skill, reflexes and training so that they’re able to quickly and reliably take control of a vehicle in an unsafe situation, whether that situation is caused by the ADS or by outside forces (e.g. bad weather or other motorists). The SAE calls explicitly out that these IFTDs should be trained in emergency handling and evasive driving techniques.

Late-stage test drivers should still have the training to deal with retaking control during unsafe situations, but because a late-stage vehicle is much closer to the end of its development cycle, the likelihood of a driver needing to intervene should reasonably also be much less. The SAE doesn’t think late-stage IFTDs need the same kind of emergency handling training that early-stage drivers do.

Speaking of training, the SAE has more specific ideas on what this sort of curriculum should include as well. The document outlines that drivers should begin with classroom instruction with lots of testing to ensure comprehension, noting that this in-classroom training should cover defensive driving and driving etiquette, as well as situational awareness and specific training on the ADS that’s going to be tested.

Once the classroom portion is done, the SAE recommends simulator training to prepare drivers before sending them out on the road. Makes sense, right? From there, these IFTDs would be sent to a closed test track for in-vehicle training. If that goes well, training on public roads follows.

This kind of a standardized methodology could very well have prevented an incident like the one Uber experienced where its test operator was found to have been watching TV on their phone instead of monitoring their vehicle. That carelessness directly resulted in the death of a pedestrian.

While we’re on the subject of distracted test drivers, J3018 also states that drivers shouldn’t be expected to do things like capturing, analyzing, or annotating real-time data while driving. That’s too involved and can lead to a driver who may not be alert enough to take control during an incident. The document does say that if the task is simple, like a verbal command or a single button press, it’s reasonable to ask.

On the flip side, the paper also warns about drivers having too little to do during an ADS test. This lack of engagement can also cause a driver to be slow to react or put them in a situation like that of Uber’s test driver, where they turn to dangerous distractions just to stay awake.

We still have quite a way to go before widespread, commercially available Level 4 and 5 autonomy-capable vehicles will be on our roads. Until then, self-driving test vehicles are going to become more commonplace on our roads. That’s why it’s crucial to have a widely-agreed-upon set of best practices and unified terminology, so that even with a relative lack of government regulation, the public can feel safer sharing the road.

If you’re curious about the SAE’s guidelines for self-driving car testing, you can visit their website and purchase the updated version of J3018 for yourself. It’s worth it because there’s a lot more information where that came from.