NVIDIA Instant NeRF software leverages AI and neural radiance fields (NeRF) to approximate how light behaves in the real world, enabling users to reconstruct a 3D scene from a handful of images taken at different angles.
NeRF is a type of deep learning model that can be trained to represent the 3D shape and appearance of an object from a set of images. It does this by learning a function that maps from a 3D point to a colour value. This function can then be used to render the object from any viewpoint, even if the object was not seen from that viewpoint in the training data.
In as little as an hour, you can compile the codebase, prepare your images, and train your first NeRF. Unlike other NeRF implementations, Instant NeRF only takes a few minutes to train a great-looking visual.
Jonathan from the YouTube channel EveryPoint has released a great hands-on video, to walk you through the ins and outs of making your first NeRF.
While the process of creating the scene isn’t too difficult, there are a number of pre-requisites that are required, and Jonathan walks you through the steps to get started. Finally, the theory is put into practice with a demonstration of how you can create an animation from the 3D scene, generated from images.
Compiling the codebase
The codebase is straightforward to compile for experienced programmers and data scientists. Beginners can easily follow the detailed instructions provided in bycloudai’s fork from the main GitHub repository.
Here are a few additional tips that helped with the installation process:
- Re-install Visual Studio 2019.
- Re-install CUDA Toolkit with the latest version.
- Install Python 3.9. If you are new to Python, this is not the latest version.
- Use CMake 3.22 and make sure that Python 3.9 is used to compile the codebase.
Capturing imagery for Instant NeRF
The pipeline accepts both photo and video input for Instant NeRF generation. The first step in the Instant NeRF generation pipeline uses COLMAP to determine camera positions.
Due to this fact, you must follow basic principles of photogrammetry with respect to overlapping and sharp imagery. The video shows you example imagery from an ideal capture.
Launching the GUI and training your first NeRF
When the images’ positions are prepared for your first Instant NeRF, launch the graphical user interface through Anaconda using the included Testbed.exe file compiled from the codebase. The NeRF automatically starts training your NeRF.
You will find a majority of visual quality gained in the first 30 seconds; however, your NeRF will continue to improve over several minutes. The loss graph in the GUI eventually flattens out and you can stop the training to improve your viewer’s framerate.
The GUI includes many visualization options, including controls over the camera and debug visualizations. I cover several different options in the GUI in the hands-on demo video.
Tip: save your commonly used command-line prompts in Notepad for future reference.
Creating an animation
NVIDIA provides an easy-to-use camera path editor with the GUI. To add keyframes, navigate through the scene and choose Add from Cam.
The GUI generates a camera trajectory with Bézier curves.
To preview your animation, choose Read. When you are happy with the animation, save your camera path and render a full-quality video with the render script in your scripts folder.
Conclusion
With any new technology like this, it will take a lot of trial and error to get the right results, but NVIDIA’s Instant NeRFs have a lot of great potential.
The speed at which you can produce results means you can quickly iteration what works for image capturing. This walkthrough from EveryPoint is fantastic and will really help those new to the field get up to speed quickly.
Now that you know how to create an Instant NeRF, watch this new tutorial on how to crop out the background:
With a 3D environment created, Jonathan from EveryPoint took it to the next level and decided to put it to the test using the Meta Quest Pro VR headset (A$1,729.99).
