What’s left on Tesla’s FSD to-do list before they can robotaxi ?

I’m sure that somewhere at Tesla, there’s a big whiteboard (hopefully digital display), that shows the outstanding development items for the Autopilot / FSD team to work on and only once these are complete, could the robotaxi dream, finally be possible.

While Tesla isn’t about to make this list public, I think it’s worth spending time thinking hard on what this list contains. This also comes at a time when the $TSLA stock is depressed, down -46.23% in the past year and many, including CEO Elon Musk, are putting their hopes on future profits of the company coming largely from robotaxis, once their FSD software is complete and approved by regulators.

Given the regulation step is such an unknown, it makes sense we focus on the technical items outstanding before that regulatory approval could be sought for a SAE Level 4 or 5 autonomous system. Tesla also announced it is working on a next-gen vehicle, referred to as a robotaxi, which we imagine will at least have a version with no pedals or steering wheel. If this new model is slated for 2025/2026, the software has a long way to go in a short amount of time.

1. Pickup and Dropoffs

As you watch the latest v11.3/11.4 FSD beta videos, you’ll notice a commonality between all of them. Almost none of the drives start from the car being parked, in a parking spot, as would be the case, where riders entering a robotaxi.

When riders arrive at their destination, robotaxis should behave like regular taxi’s in that they park and let riders out safely near the curb. This may involved a reverse or angled park but currently FSD Beta will get you to the end of the routed address, or pinned position, then hand back to the driver.

In a robotaxi world, this simply isn’t going to fly.

It may sound like a small item, but there are a huge number of challenges to overcome to reach a point where rider dropoffs are done safely. Think about all the different types of parking signs that would need to be read and understood by the car.

While Tesla is getting much better about recognising lanes, you’d also have to consider what the car does next after a dropoff. If the riders exit the vehicle and then there’s no next job to go to, the car would sit in that location until there was. If this happened to be a no-parking or tow-away zone, that wouldn’t work.

There is a chance that Tesla develops an understanding of parking locations that cars could travel to when not in use, but this would add km/miles to the unpaid distances travelled per day by car.

While we imagine the interface will be somewhat restricted to passengers, not unlike valet mode, it’s likely customers would still need access to the frunk and truck for storage while travelling, how the car would know they need to remove storage before driving away, is also something that needs to be resolved.

2. Collision avoidance

If Tesla was to ship a vehicle or allow you to enrol yours into a robotaxi service, you’d likely want to be confident the car was robust enough to handle real-world encounters that had the potential to mean your car becomes damaged or stranded.

I see 3 main opportunities for this to occur, each of which would need to be addressed before robotaxis are practical.

Pothole avoidance

When you think of a road, you probably imagine bright, shiny new tarmac, well-marked and in perfect condition. The reality is, roads constantly degrade and potholes develop, particularly after severe weather events. What may start as a small breakup of the surface, rapidly deteriorates into a massive hole as cars repeatedly drive over the same area.

With potholes being a far too common occurrence on our roads, it’s easy to imagine that within just a few trips a Tesla robotaxi could drive over a severe pothole and damage a rim and tyre to the point where the vehicle could no longer drive.

To address this, Tesla would need to develop its computer vision technology to accommodate these stationary obstacles on the road. While these approach rapidly, I imagine the following distance on a robotaxi could be set hither to provide more reaction time and a smoother turn away from danger as to maintain ride comfort for passengers.

Detecting and avoiding potholes in good weather feels possible and the car would have much better reaction times than humans, but the question is, how would it be fair in poor weather where a pothole is filled with water?

Debris avoidance

Not all obstacles on the road are stationary and have a predictable path. Unfortunately far too many drivers do not secure their cargo properly and we end up with lots of weird and wonderful items on the road. There’s also a healthy amount of trees down on the roads after any storm.. obviously, this won’t impact concrete jungles, but for those who live outside the urban centres, it’s a practical reality.

These all vary in shape, size, weight and angle and have different trajectories based on what the item is. Imagine everything from paper that you could easily drive over without issue, to mattresses, ladders, a bag of nails and more.

It is possible that a more advanced version of the occupancy network could be the answer here and the car would quickly identify anything on a trajectory to intersect with its planned route, and then subsequently route around it. This could be high risk and there are times when there’s simply nothing any human or autonomous car could to do avoid it.

This means we’re unlikely to get to a 0% chance of an accident here, but it needs to be so low that if you run your car as a robotaxi every day of the year, you can expect it’ll make it home each day.

Animal avoidance

While vulnerable road users (aka humans) are a challenge, I feel Tesla has vehicle responses largely under control for the vast majority of risks associated with vehicle/human interactions.

Where things get a lot harder are animals. These are inherently unpredictable and their speed, capability, responses to headlights etc vary considerably.

Tesla’s approach to autonomy is to enable the cars to drive anywhere (talking sealed roads), so this means city and country driving. This means accommodating deer, kangaroos and other fast-moving animals is necessary to execute that vision.

It is conceivable that Tesla could model the characteristics of larger animals that would present an issue for the vehicle (sorry birds, snakes etc). This would require a significant investment in time and resources.

Many human-driven cars end up in repair shops with body and windscreen damage, or completely written off every week. Our expectations certainly shouldn’t be that this goes to absolute zero, but rather that autonomy reduces the number of incidents and if you’re the robotaxi owner, you’d love an option to avoid dangerous roads based on previous accident data in Tesla’s fleet.

3. Charging (nav to supercharger)

Right now, humans monitor the charge available in our vehicles and we make calculations about our planned destinations compared to the available energy. Where there’s not enough, we plot a route to charge the car, or ensure it gets charged overnight, ready for the next day.

When the car is in a robotaxi fleet, charging absolutely needs to be accommodated.

Firstly we can assume the car returns home to charge overnight (Tesla will have to find a home for their robotaxis). When the car is providing rides and making you revenue through the day, it should monitor the distance travelled (this will vary per day based on rider’s destination).

As the State of Charge (SOC) drops through the day, the car would be automatically calculating potential routes to charge and at the end of a ride. When it reached a critical threshold (i.e. 20%), the car would route to the Supercharger.. but not by itself.

Idea: There is an opportunity here, for Tesla to prompt a rider for help.. if they accept, the car would route not to their destination, but route to a charger and offer them either a free ride, or a significant discount for the inconvenience.

This means Tesla doesn’t need to wait for a fleet of Optimus robots, man the Superchargers with their own staff (paid by the hour), or build a robotic charging snake. Instead, Tesla could handle the navigation and parking of the car at a Supercharger stall and the only thing the human needs to do is plug in the charger! This seems like win-win, as long as 1 or 2 customers per day are willing to accept a 20-30 minute longer journey.

Something I think about a lot is Tesla’s capacity to roll out its own charging infrastructure. I could definitely see them standing up dedicated real estate for Supercharging robotaxi-enrolled vehicles, as adding thousands of vehicles to existing charging locations is unlikely to work.

4. Emergency vehicles

Simply because a vehicle is autonomous, does not avoid its obligation to comply with emergency vehicles and incidents. It could be a car crash with police directing traffic, or simply a matter of moving to the shoulder of a road in a tunnel to allow a tow truck to pass by, regardless of the reason emergency vehicles are around, robotaxis need to respond safely and predictably.

While the Tesla may not feature an external microphone to listen for sirens like we would, deaf people can drive as well, and the reason emergency vehicles have both visual and audible indicators to get your attention is that both work.

With Tesla’s Computer Vision approach, their surrounding cameras have plenty of opportunity to recognise and respond to flashing lights, even if it has no ears.

5. U-turns and reversing

There are some pretty crafty things you can do to route to locations that mean you never have to perform a U-turn or reverse the vehicle to reach your destination. Despite that, there are times when it will be necessary. and currently, Tesla’s FSD beta isn’t really engaging in these standard driving practices.

Imagine your drop-off point is at the end of a dead-end street. You’d need the car to nose into a driveway and complete a 3-point turn to exit and embark on its next journey.

Tesla’s hardware suite means (HW3), now in millions of vehicles, was decided back in the 2017/2018 timeline and they made a decision to rely on their rear-facing reversing camera feed to make determinations about the safety of reversing.

While there is ultrasonics on the rear bar of many Tesla vehicles, recently Tesla stopped shipping these, in favour of their vision-only approach. There are often blind spots that would make it difficult to reverse out into a stream of traffic.

If FSD wants to make it in Melbourne, it’d also need to accommodate hook turns, another challenge entirely, but would seem a solvable one, with turns of the data engine.

6. Reverse summon (drop you off and park)

Smart Summon today is not that smart and Tesla knows that. Back in October 2022, Elon Musk posted that Actual Smart Summon is almost done. Unfortunately, we’re still waiting and Smart Summon still requires your mobile to be within a fairly short range, and by the time you can see your car, using Summon to come to you, is rarely beneficial.

What Actual Smart Summon should do, is allow you to pick your destination and time and come pick you up. At the start, the locations for pickup may be restricted to parking lots, rather than on the side of roads, given the more complicated challenge of traffic, but there’s more to the smart summon offering.

Reverse Smart Summon is a concept where you’d get dropped to at the entrance of a shopping center and the car would then go find a carpark. This doesn’t exist today and while not strictly required for robotaxi, it is part of what’s been promised as part of the FSD package.

7. Multi-story carparks

There are times, when your car may need to park for extended periods of time. You could be at a shopping center and your car needs to park while you shop for 3 hours and at the mall, there’s a multi-story carpark.

Navigating its way up and down turn corners underground where GPS and cellular signals are often blocked is a challenge in itself, but there are others.

At the entry and exit to these parking garages, there are often boom gates that require you to take a ticket from a machine before the gate will open. Without a driver to do so, this seems like a great opportunity for things to go badly.

It is possible Tesla creates a partnership with parking garages and gets them to retrofit a solution for autonomous vehicles, but this seems like years away and can’t be a dependency for multi-story parking support.

8. Mobile app

The Tesla mobile app has seen many generations and many improvements over the past 5 years, but it needs to undergo perhaps its biggest refresh yet if Tesla is to deliver a robotaxi service.

Tesla recently added the ability for the owners of non-Tesla EVs to use the Tesla app, add a credit card and use a limited set of Superchargers.

I would expect that the same Tesla app would also be used for robotaxi and anyone looking for a ride would download, setup an account and request rides, very much like Uber.

The front end is one challenge, but there’s a whole back end to a robotaxi service that would need to be built. Tesla has never delivered a revenue share before and while the terms of robotaxi won’t be known for some time, owners who enrol their cars in the service will get remunerated for doing so, while Tesla will take a percentage for running the network and bringing them customers.

What is critical to user adoption of a Tesla-powered robotaxi is the time to ride, meaning there needs to be an appropriate level of available cars, to ensure riders are only waiting a few minutes. A lower cost (minus the cost of the driver) could encourage some to wait slightly longer than Uber, but there will be a limit to this, it certainly can’t be double the time, even if it was half the cost.

I expect Tesla will break out their latest AI to manage the challenge of job allocation based on the availability of vehicle types and the requirement of the rider or riders. Of course, there’s a bit of a turo element to deal with as well. If riders damage or dirty the vehicle, then alerts should be sent to the vehicle owner for remedy and in some circumstances, may mean the vehicle is offline for a period of time.

There are literally months and months of work involved in standing this up. Maybe Tesla is banking on using the time they have as they seek regulatory approval, but they are certainly showing no signs of doing what ARK invest suggests, which is to get their stand up now for a human-powered ride-share service ahead of delivering autonomy.

9. Right-hand drive support (Hi from Australia)

To date, the only countries that have FSD beta, the latest and greatest software builds from Tesla, are the US and Canada, both Left-hand drive countries. While the majority of the world is LHD, Tesla has sold FSD across the world and while I don’t think this is the largest issue to overcome, it is something that we are yet to see and that would need to be achieved before a robotaxi service was available here.

RHD is a little more complex than just flipping things, there’s the language changes here, meaning that having the character recognition models and potential voice models in the car for customer requests, absolutely nailed to deliver the best experience and in some instances, to ensure the car follows the rules and doesn’t end up with a traffic or parking violation.

10. Others

There are likely still many other development items on the to-do list in Tesla Engineering, but here are a few more, if you’ve got some, drop them in the comments below.

• Weather (detect, visualise, route around storms)
• Trams / Trains (frequent stopping, passengers offboarding)
• Railroad crossings (signs, signals etc)
• School buses (40km/hr when lights flashing)
• Digital signs (above and roadside – recent improvements, but more work to do)
• Right turn on red / yield on red
• Lane recognition (parking lane)
• Distance to bike riders (ie. min distance of 1m up to 60km/hr, or 1.5m at over 60km/hr)
• Other Tesla vehicle support (Cybertruck, Semi etc)

When autonomy? When FSD?

Predicting when something will be finished in technology is a fool’s game, particularly if it’s something hard, that’s never been done before and unbound autonomy is certainly that in spades.

Many talk about AI accelerating the path to autonomy and that may be true, but regardless of improvements, it’s the user facing-features that really show and feel like true progress is made, rather than architectural rewrites.

If I had to guess, I’d say we’re still a couple of years away, given the volume of challenges in the list above, many of which haven’t even been started. Let us know what you think in the comments.