Overnight, a new video went live from the Computer Vision and Pattern Recognition Conference (CVPR), where Tesla’s Head of Autopilot Software Ashok Elluswamy presented.
Elluswamy provided some great insights into new development Tesla is making regarding their computer-vision-based autonomous software, known as Full Self Driving. While the name has non-Tesla owners confused, the insights into Tesla’s efforts to understand the world around the vehicle and safely navigate through it should leave no doubt in people’s minds, they absolutely plan on making their cars autonomous.
During the talk, we saw new strategies from Tesla to tackle the challenges associated with achieving this. We’ve seen Tesla start with lane lines as the bounding parameter of driving, which is still in use today in Autopilot, but their FSD Beta moved to use Drivable space as the limiting factor, enabling the car to cross lines where appropriate, moving through complex environments in a more human-link way.
Tesla also makes depth calculations from the vision system, to determine if objects are solid and should therefore be avoided, but this has challenges like being able to predict through occlusion as well.
Occupancy networks
Elluswamy introduces us to Occupancy Networks as the solution to these challenges. This is a new technique to take in inputs from all 8 cameras to construct a 3D representation of the volumetric occupancy around the car. This means it doesn’t just identify curbs to avoid when cornering, but actually understands the vehicle and the space available around it (this may be incredibly beneficial for Semi to avoid objects like overhead signs and even overhanging trees).
Effectively this translates to space being occupied or not, meaning the car cannot enter occupied space, regardless of what is occupying it. It could be a permanent object like a mailbox, or a temporary object like a parked car, either way, the car understands it can’t enter that space and navigates around it, while still applying the constraint of general road rules.
Things are just static in this Occupancy Network, Tesla treats all objects as if they could move, so works to predict their flow (or movement) over time to understand if they are about to occupy the space the Tesla is attempting to drive to.
What was great to hear and see in the demo, was Tesla’s ability to detect objects it has no 3D asset for and still understand it is occupying space that is unavailable to it. This means you may not see an item in the visualization, but that does not mean the car hasn’t seen and accommodated for it. I expect there’s some interesting discussions inside Tesla on how to represent this, given the amount of 3D assets in the software can’t be infinite.
One of the most interesting sections of the presentation is around 18 minutes in when Elluswamy discusses occlusion. Parts of the environments we drive in, can at times be hidden from the view of the cameras. The represents a real challenge for us to understand how Tesla could overcome such an issue.
An example of this is the unprotected left known as ‘Chuck’s Turn‘ which is expected to be addressed in FSD Beta v10.69 dropping today. In the presentation, we see an example intersection, where oncoming traffic is occluded from vision.
Another great example of these challenging intersections is this image from Lisa, showing trees to the left of the vehicle.
To date, Tesla has used a pretty basic implementation of creep, which edges the car forward to increase the angle of incidence so the vision system and path planning have a chance to route the car through an intersection.
What we learnt today is that Tesla’s brain in the car, has the capability to understand where it has issues seeing (i.e. cars appearing from nowhere), and adapt to that lack of visibility. Think of this as smart creep, understanding the distance from the front of the car that cars are passing by, to appreciate the space available to creep into. With the right amount of creep, the visibility to oncoming traffic improves and enables the car to navigate through the environment safely.
I believe this is what is included in FSD Beta v10.69, making this a fairly big architectural change to the way decisions are being made in the car.
The uncrashable car
In the back half of the presentation, we learned about Tesla’s capability when it comes to Collision Avoidance. We’ve seen a little bit on this previously, in regards to smart shift. In the refreshed Model S and Model X, Smart Shift offers owners the ability to automatically select the direction of driving based on what the camera sees.
This Collision Avoidance technology is now going way beyond and sounds like it will come to all Teslas. This accommodates for the mistakes human drivers make in the incorrect application of the accelerator, which can result in vehicles crashing into buildings, or other vehicles, or even worse, people.
Leveraging the spatial ‘Occupancy’ data as discussed above, Tesla then translates that into a collision avoidance probability. That is, if you’re facing a wall and press the accelerator, the probability you’re going to crash would be almost 100%, so the car wouldn’t let you do that.
The image above is from a simulation where a distracted driver presses the accelerator and offers no steering input. The car detects a crash would occur and plots a path to navigate the car safely through the environment. Unlike the modest lane correction we have in cars today, turning the wheel just a few degrees, in this example, the Tesla would change trajectory dramatically, crossing to the correct lane and continuing on safely.
Elluswamy wraps the presentation by explaining that if they can successfully implement all the techniques discussed, they could produce a car that does not need to crash, ever.
This job is clearly not done and solving it, will take more effort, with the last slide of the presentation, a pitch to engineers to come to join Tesla to build a car that never crashes!
This will provide a very interesting challenge for Tesla, now they are in the insurance space. There is still the possibility that other cars run into a Tesla, so insurance is still likely to be necessary, however, imagining a day where all Tesla’s with HW3 and above get a software update that makes them uncrashable (from a result of the driver input) is wild to think about.
