Tesla Is Crazy To Not Make Full Use Of Maps To Improve ‘F’ Self-Driving

Almost every self-driving car team makes use of pre-computed maps, combined with what their sensors see, to help them understand the road and drive correctly and safely on it. Tesla is a rare exception, deprecating detailed maps and trying to use mostly navigation maps to do the job. While Tesla hopes this approach lets them drive “everywhere” without the effort of making and maintaining maps, because it doesn’t work yet, it means instead that they try to drive everywhere, but do it badly.

Other teams have concluded it is better to spend a bit more effort to actually be able to do the driving task, then expand their territory rather than not being able to do the task anywhere. There are a variety of reasons behind this decision, which I go into in a new video on real self-driving and mapping.

Last month, in a review of Tesla’s “FSD” prototype version 10.8, I had to give it an “F” for the surprisingly large number of serious errors it made in a short distance, including 3 wrong turns, running 2 red lights and blocking traffic. In spite of this, many of the mistakes the system makes would not be made if it had good maps. A few of those mistakes are shown in the video. Many of its mistakes come from not understanding the lanes and what they mean, or the lights, or what’s about to happen up ahead that it can’t yet see. Maps help understand the road to greatly reduce such risk.

Given that everybody wants to get to safety as soon as they can, it is odd that a major team would deliberately avoid using such a valuable technique. It is commonly said that Tesla feels that they will some day make driving without a map work, and once they do, the effort building and maintaining maps would be a waste. Many people mistakenly thinking making maps is very expensive and stops a system from scaling up. They also think that it’s too hard to handle the way roads change and make maps out of date due to construction. In the video, and below, it is explained why those impressions are a mistake. In addition, even if Tesla’s bet that it can drive safely mapping on the fly works out, it’s explained why even getting halfway to that makes very useful mapping free — so there’s no reason not to do it.

The video contains visual examples, but for those who prefer text, an edited transcript follows. You can read this and or watch segments of the video (which have chapters to allow you to find them.)

Tesla’s bad decision to avoid maps

When people compare the approach of Tesla FSD with other self-driving teams, they often are drawn to the role of maps.  There are many other controversies, like LIDAR, and radar and and what a beta is and much more, and I've got articles on all of those, but in this video, I'll show you why Tesla is crazy to avoid using maps to make their car safer, sooner.

Tesla drives without first building detailed maps of the road as almost all other teams do.  Fans say that's the only way because now the Tesla can drive everywhere, and others only drive where they have maps.  They still think that mapping is too expensive and doesn't scale.

Key Points

There are 4 important points to consider:

•     That mapping lets you drive more roads, not fewer, by solving safety and perception problems
•     That mapping is not expensive and scales fine — in fact it can be free
•     That while maps of course get out of date, being surprised by an out of date map will be extremely rare, and
•     There are many ways to handle those rare situations, particularly if there are lots of details in your map

Don’t drive badly everywhere

The reality of mapping is not as you might imagine.  A Tesla must build a map "on the fly," but that doesn't actually work yet. So a Tesla can't drive everywhere, it can drive everywhere badly. The best other teams can drive well in the places they spend a rapidly shrinking amount of effort to make maps.   While hope springs eternal, there's no sign that mapping on the fly is about to work soon.   Even though humans can drive tolerably with no maps in an area they have never been, computers aren't nearly as smart as people at doing that, and they don't work the same way human brains work.   Nor should they.  Just because birds fly by flapping their wings doesn't mean that's how we build airplanes.

Mapping not only improves the quality of the drive greatly, it's not expensive at all — in fact now it can even be effectively free, which scales really well.

Most teams rely on mapping.  Tesla has declared that it will not use high-definition maps, which are maps that contain details on many features of the road including the position of lane lines, curbs, signs, traffic signals, some static objects and sometimes much more.    Tesla isn't entirely explicit about what they mean.   Like everybody they use basic navigation maps, and even lane-level navigation maps, though they don't appear to have full coverage.   It is widely speculated that they are using more detailed maps of certain areas of the road where they have problems, but don't admit it.

The Tesla FSD visualization shows the car building a map "on the fly" as it drives.   This map is often correct, but also very frequently wrong.  You often see it changing as the car advances, often getting better, but not always.  "Often" doesn't cut it in self-driving.

The obvious attraction of Tesla's goal is to be able to someday drive just about anywhere without spending extra money on mapping.  Cars that use maps tend to limit themselves to where they have made and certified their maps.  This is incorrectly often called "geofencing."   On top of that, everybody knows that the road sometimes changes, and the map becomes out of date thanks to construction zones or other situations.   Those who use a map must handle those changes, while the map-free car has to drive in that scary state all the time.

Maps can be close to free

People ask whether making maps and keeping them up to date scales, or if it's just too expensive or limiting.  This question also hinges on what type of vehicle is being made.   A consumer car needs to be sold over a very wide region.  A car that only drives itself in a few cities is a very tough sell for any car brand that needs to get sales everywhere to be profitable.    A robotaxi, on the other hand, is a perfectly viable business even if it has a limited service area.  You could run a successful robotaxi company just in Manhattan or San Francisco if you wanted to, and grow it from there.

Since driving almost everywhere is a must for a consumer car, mapping must be done and maintained everywhere.  People presume that would be much too expensive and can't scale.   That's not true because it can actually be free.  That's because a car that has any chance of building a map while it drives, is also a car that can build a better map for free.

When you map on the fly, you have to figure out the road from a distance, as you drive towards it.  There are lots of things you can't see, or see poorly.  When you get right up close, you'll see much more and build a better map, but it might be too late.   You can learn even more if you get to see things from another angle, or at different times of the day.

Mapping after the fact is much easier.   You get to look at everything right up close.  In daylight if you want.  Drive twice and see it all from the other side, or different lanes.    Nothing is blocked from your vision.   After you collect the data, you have all the resources of the cloud to bring out the big AI guns to make the map.  You can do much better than you can from a distance at night, needing an answer in real time with the tiny computer in your car.   You can remember what you, or your cousin cars have learned when they drove the road.

That's what the map is at its core.  Just having a memory.  Why would you want to forget and drive ignorant each time?

Examples maps can fix

In the video is a clip of my Tesla using FSD 10.8 to drive next to Apple HQ.  It doesn't like the right lane, so on this road, it usually finds itself in a lane that must turn left.   You must move to the right in order to go straight.   Sometimes when it drives this it figures it out, though usually a bit late, and makes the move.

A few times, though, it stayed in the lane and was in a no-win situation at the end.  It can't legally move right, if it goes straight it goes into a yellow zone, and left is a security gate.   Once it tried to go left, running a red light, and I had to intervene.

Other times it figured it out.  Once it drove closer to the lanes, it could see the left arrows and the rest of the road geometry.  It would barely figure it out in time, though.    After seeing it all and figuring it out, it should upload it into the map.   A map made by a car driving the other way might add perspectives to be merged in the cloud.  All the million Teslas driven by humans with the FSD hardware could be doing that.

When the FSD car drives this road, first it goes over a bridge.  It can't see the turn ahead, so it puts itself in the left lane.  If it still had that map, it would know that's the wrong lane to be in to go straight, and would put itself in the right lane, and have no problems at all, and not do the last minute lane change.   Instead, in the left lane it tries to figure it out from a distance again and takes time to figure out the left arrows on the road.  That leads it to a bad place where it makes bad mistakes.

Avoid a crash

Another recent cip in the video  involves a right lane which ends quickly after an intersection.  As FSD drove though this in moderately dense traffic, its on-the-fly map could not see the right lane was ending and it moved with full speed. When the car in front merged left it was going fast with no more road. I had to intervene to prevent a likely crash.

With a map it would have been able to know the lane was ending.  It would not have missed the signs, either.  We often say humans don't need maps but many humans find themselves stuck by surprise at suddenly vanishing lanes like this one.   People who have driven the road before know to try to get one lane over, or go with caution into the merge.

Where would it have gotten this map?  Well, on the way out, during twilight, it passed through this piece of road, scanning both sides.  Examination of an image from the car’s drive over this same road in the other direction minutes before reveals that it scanned and mapped this section of road and saw that 3 lanes turn into 2 at this point.  If only it had just remembered the map from before, or had access to the maps a thousand other cars built of this road in the past.

Crowdsourcing

That's just what several map companies are doing — "crowdsourcing" their map data from ordinary cars, driven by people, all over the world.   Some start with a drive or two from a dedicated car and rely on the crowd to confirm and discover changes in the map.  Others do it all with the crowd.

This type of mapping happens for free.  If, like Tesla, you have a million vehicles driving the roads, constantly trying to make maps, you get all these maps without human effort.  This is what MobilEye and Nvidia DeepMap do already.  MobilEye has an even bigger fleet than Tesla.   It scales just fine, to the whole world when you need it.

Handling construction zones

So what if, somehow, these lanes had been repainted or there was construction?  If your car has a map with details, it would quickly see they don't line up.  The lane markers are in the wrong place.  In that case, it would slow down a bit and drive like a  map-free car. 

In spite of some intuitions, it's actually quite rare to be surprised by construction.

We worry because we all see construction zones every day when we drive. Turns out that we're almost never the very first car to see that construction zone and get surprised by it.   Tens of Thousands of cars will pass the zone and only one will be the first.  When any car in the fleet notices a change and updates the map, nobody else is surprised by it.  Programs like Waze usually know about the construction zones before you get to them — and they require a human to type things in, it's not an AI doing the job.

Most construction zones are planned, and need permits, and are recorded in databases before work starts.  Here's an example of how the California road authority maps the construction on state roads.  The self-driving mappers all make use of that, so even the very first car to the construction is rarely surprised.  If a few laws were changed, it could be made so that construction crews don't get paid if they change the road without pulling out their smartphone first.  Surprises could be one in a million, but you can never say never.  

Once that first car in your fleet visits the construction zone, it should be able to  send data to update the map.   If the first car has nobody in it, it has a few choices.   It can drive it like a map-free car, and that will work as well as Tesla FSD, but with today's technology, that's only probably.  It can route around it until a human driven car happens to scan it.   Or it can just take it slow and stay safe.   Most construction zones require you to slow down anyway.   Would this be annoying to those behind the car?  A little.  But remember, this almost never happens.    Better to slow down, or even pull over, once in a blue moon than to be unsafe if you might figure out the road wrong.

Humans don’t drive down railway tracks

To learn about another trick, the video contains a clip by YouTuber “AI Addict” of an earlier release of Tesla FSD handling a turn in San Jose, California.   This street has light rail tracks and cars are not permitted to go there.   For many months, Teslas on FSD just couldn't figure this out, and kept wanting to drive down the train tracks.   After a lot of public shaming, they trained the car to get better at recognizing this particular situation, as they should be doing as bugs are reported.    Even after it got better, it changed from going down the tracks to using the bus-only lane.  A step up, I guess.

Maps would easily have fixed this, as they would fix probably more than half the mistakes I see FSD make in videos.    Free maps might not, though odds are that that mapping computer, with the chance to get input from many cars driving the full length of the road would have had a better chance to grasp what's going on.  But there's another trick.

Learning from other drivers

MobilEye does a clever mapping technique.   They add to their map information gleaned by watching tens of millions of human cars drive these roads.   Using that, you quickly notice that no cars are entering that train lane.  You will also learn that most (though not all) avoid the bus lane.   It's very clear without reading any signs or understanding what train tracks are.   Watching humans drive a particular road will observe the way they drive differently from what the lane lines say.  You can learn how they creep up to see cross traffic.  This road has a stop sign which is a bit confusing, and meant for traffic in another direction.  Tesla FSD often stops for that sign, but a map would show that human drivers don’t stop there.   These are the unwritten rules of the road and it's great to have on your map.

To repeat, driving without a map doesn't work everywhere.  It works everywhere badly.   With self-driving, "works" doesn't mean "sometimes works" or even "works 99.99% of the time."   This is what misleads so may people about FSD's development status.   With ADAS like Autopilot, what matters is what it can do, because the human takes up the slack.  With self-driving, what matters is what it can't do.   And if it doesn't do it all the time, it can't do it.

Not free, but worth it

One thing you can do with money is send out a fancier car with extra sensors to deliberately drive the road to create a base map.  Then you refine it and update it

There's no sign of map-free driving working soon.    Maps work now,  and they can work everywhere at a surprisingly low cost, but there is a cost.   Since when do you want to give up that much safety to save a little money?   

Free maps, made from examining the scans of each road from multiple angles and distances entirely by computer, can be pretty good.  If you really have zero budget, you might stop there.   Because everybody seeks the highest safety, and wants to be first to get there, most teams do spend some money on their maps, but it's a worthwhile spend.

In addition to automatic mapping, if there is something the computers can't figure out, you can call upon a human QA team to make sure it's right.

If you involve humans, to drive the road or to do the QA, that costs money and time.  People worry about whether that can scale if you need too much human effort.    Even if it is expensive, the cost per mile is not that bad.  And even if the cost per mile is high, that map will be used by thousands of cars a day driving over that stretch of road.   The eventual cost per trip becomes sub-penny, and barely worth noticing.

If you're still dreaming of a car that drives without a map, remember that such a car is also a free map-building car.  And a car that drives badly without a map can be good enough to provide the data for systems in the cloud to make that free map.   The map is a memory, and not remembering what you have learned is just silly.  That's just a cost of data processing.

That's important because remember that while Tesla hopes to drive without a map, today they can't.  They make all sorts of mistakes today, and many of them wouldn't happen if the car had a good map.  Making a perfect, "you bet your life" map on the fly is a stretch goal, one not yet close to being achieved.    Sure, drawing a good guess map most of the time is doable today, but a guess is a very long way from "bet your life."

Maps only get easier

In the future, this won't even be a problem.   As we get more robocars out there, people will think of mapping as an essential part of building or modifying any road.  Nobody will put up a sign without putting it on the map first — they won't get paid unless they do.  No steamroller will roll without a phone in it that records where it went.   Before long, that will always be how we roll.  Mapping will be a tiny part of the cost of road construction and maintenance, and it will be useful for far more than the robocars.   It's going to be easy, and it's going to help cars not just perceive the road but understand it.  They'll understand not just where  lanes or signals are but what they mean.  Meaning is one of those things that computers still need to work on, and we want to give them all the help we can get.   You bet your life.

Read/leave comments and see FAQs here