Adam CookNov 21, 2022

Another #SystemsSafety Monday is here!

Today, I would like to talk about an article I saw last week in the New York Times concerning the #Tesla #FSDBeta program.

The article was entitled "What Riding in a Self-Driving Tesla Tells Us About the Future of Autonomy": https://www.nytimes.com/interactive/2022/11/14/technology/tesla-self-driving-flaws.html

We are already off to a bad start in terms of public safety because Tesla vehicles are not capable of "self-driving".

In fact, no consumer can purchase a self-driving vehicle today - perhaps ever.

🧵 ⬇️

1/6

What Tesla ‘Full-Self Driving’ Tells Us About the Future of Autonomy

We took a ride with Tesla’s experimental self-driving system. It was by turns intriguing, impressive and scary.

Show threadAdam CookNov 21, 2022

The article focuses on a FSD Beta human "tester" (which is not actually testing anything at all).

Despite its full name of "Full Self-Driving", an equipped Tesla vehicle requires a **human** driver fallback at all times while the partially automated system is active.

This means that, at all times, a human driver is there to act as a "safety net" for when the partially automated system makes a mistake.

Therefore, at all times, the human driver is driving the vehicle - **not** FSD Beta.

2/6

Show threadAdam CookNov 21, 2022

The human driver, at all times, has effectively the **same** driving responsibilities as if FSD Beta did not exist at all.

So, the other public safety issue in this article is to describe the situation as "Tesla is driving" in various places - which is not true at all (as pictured below).

FSD Beta or "the Tesla", the partially automated system, is **not** "driving" at any time.

The human driver featured is driving at all times.

3/6

Show threadAdam CookNov 21, 2022

I am coming off as repetitive because the **acute** understanding of the limitations of a partially automated system or #ADAS is an essential pre-condition to maintaining systems safety.

It does not matter how mature the underlying technology or engineered system is if the human driver **erroneously** believes that the system is capable of "self-driving" at any given time.

That misunderstanding makes the system **terminally** unsafe before the human driver even sits down in the vehicle.

4/6

Show threadAdam Cook

The misunderstanding by the article authors (and the human "test" driver featured) is understandable.

Because despite Tesla's dangerous, years-long marketing campaign on the subject, the steering wheel does **appear** to be rotating by itself at times.

The brake pedal appears to be depressing itself at times.

The accelerator pedal appears to be depressing itself at times.

But underneath all of those appearances, the human driver is the fallback, again, at all times.

5/6

Nov 21, 2022 at 12:49pmWeb
Show threadAdam CookNov 21, 2022

I would implore the #media to educate themselves on these subtleties for future articles.

I would also invite everyone to review Liza Dixon's seminal work on #autonowashing - a dangerous over-exaggeration of automated driving system capabilities: https://lizadixon.com/Autonowashing

Liza is on the 🐦 site here: https://twitter.com/lizadixon

Liza has always been steadfast at helping the media and the broader public understand these technologies from a #HumanFactors perspective.

That is it.

Happy #Monday!

6/6

Autonowashing — Liza Dixon

Autonowashing[ aw-ton-uh-wosh-ing ] verb. The practice of making unverified or misleading claims which misrepresent the appropriate level of human...

Show thread虚无的空间Nov 22, 2022
@adamjcook just got full self driving beta, eager to test it out. I would think of it as watching Tesla driving. It helps, although one cannot trust it blindly. Healthy dose of skepticism is quite useful to remind us this tech is not mature yet.
Show threadAdam CookNov 22, 2022

@charleston The key is that one must always realize that FSD Beta is **never** a "safety net" for the human driver.

The human driver is **always** the safety net for FSD Beta.

But that systems-level fact is very different from how Tesla/Musk describes it and how many in the Tesla Community (including Chuck Cook in this article) portray it.

There are other structural systems safety issues with FSD Beta that I will touch on in later threads, but the above is the Big One.

Drive safely!

Show thread虚无的空间Nov 22, 2022

@adamjcook I really appreciate your criticism. Watching autopilot already showed possible mistakes that the car makes. I would not trust Elon, especially when he sells it more than what it is. The system must be watched by the driver all the time.

Knowing how neural networks work also makes one highly skeptical of the technology. Big data is good, but it is still statistical. Even if it works 99 percent of time, that 1 percent could be fatal.

Humans need to babysit tech.