NVIDIA NeRF allows you to create stunning 3D animations from images

    NVIDIA Instant NeRF software leverages AI and neural radiance fields (NeRF) to approximate how light behaves in the real world, enabling users to reconstruct a 3D scene from a handful of images taken at different angles.

    NeRF is a type of deep learning model that can be trained to represent the 3D shape and appearance of an object from a set of images. It does this by learning a function that maps from a 3D point to a colour value. This function can then be used to render the object from any viewpoint, even if the object was not seen from that viewpoint in the training data.

    In as little as an hour, you can compile the codebase, prepare your images, and train your first NeRF. Unlike other NeRF implementations, Instant NeRF only takes a few minutes to train a great-looking visual.

    Jonathan from the YouTube channel EveryPoint has released a great hands-on video, to walk you through the ins and outs of making your first NeRF.

    While the process of creating the scene isn’t too difficult, there are a number of pre-requisites that are required, and Jonathan walks you through the steps to get started. Finally, the theory is put into practice with a demonstration of how you can create an animation from the 3D scene, generated from images.

    Compiling the codebase

    The codebase is straightforward to compile for experienced programmers and data scientists. Beginners can easily follow the detailed instructions provided in bycloudai’s fork from the main GitHub repository.

    Here are a few additional tips that helped with the installation process:

    1. Re-install Visual Studio 2019.
    2. Re-install CUDA Toolkit with the latest version.
    3. Install Python 3.9. If you are new to Python, this is not the latest version.
    4. Use CMake 3.22 and make sure that Python 3.9 is used to compile the codebase.

    Capturing imagery for Instant NeRF

    The pipeline accepts both photo and video input for Instant NeRF generation. The first step in the Instant NeRF generation pipeline uses COLMAP to determine camera positions.

    Due to this fact, you must follow basic principles of photogrammetry with respect to overlapping and sharp imagery. The video shows you example imagery from an ideal capture.

    Photos with callouts: Avoid changes in lighting, use a gimbal to ensure sharpness, and take 50-150 overlapping images.
    Figure 1. A few tips on input images to improve the quality of the NeRF output

    Launching the GUI and training your first NeRF

    When the images’ positions are prepared for your first Instant NeRF, launch the graphical user interface through Anaconda using the included Testbed.exe file compiled from the codebase. The NeRF automatically starts training your NeRF.

    You will find a majority of visual quality gained in the first 30 seconds; however, your NeRF will continue to improve over several minutes. The loss graph in the GUI eventually flattens out and you can stop the training to improve your viewer’s framerate.

    Screenshot with a callout indicating where to find the Flattened loss graph within the GUI.
    Figure 2. Snapshot of the GUI within the Instant NeRF software highlights the flattened loss graph

    The GUI includes many visualization options, including controls over the camera and debug visualizations. I cover several different options in the GUI in the hands-on demo video.

    Tip: save your commonly used command-line prompts in Notepad for future reference.

    Figure 3- Share your commonly used command prompts in a notepad for ease of use later on.
    Figure 3. Command-line prompts within the software

    Creating an animation

    NVIDIA provides an easy-to-use camera path editor with the GUI. To add keyframes, navigate through the scene and choose Add from Cam.

    The GUI generates a camera trajectory with Bézier curves.

    To preview your animation, choose Read. When you are happy with the animation, save your camera path and render a full-quality video with the render script in your scripts folder.

    Screenshot of the static images rendering into a 3D scene.
    Figure 4. Instant NeRF renders the static images into a 3D scene


    With any new technology like this, it will take a lot of trial and error to get the right results, but NVIDIA’s Instant NeRFs have a lot of great potential.

    The speed at which you can produce results means you can quickly iteration what works for image capturing. This walkthrough from EveryPoint is fantastic and will really help those new to the field get up to speed quickly.

    Now that you know how to create an Instant NeRF, watch this new tutorial on how to crop out the background:

    With a 3D environment created, Jonathan from EveryPoint took it to the next level and decided to put it to the test using the Meta Quest Pro VR headset (A$1,729.99).

    Jason Cartwright
    Jason Cartwright
    Creator of techAU, Jason has spent the dozen+ years covering technology in Australia and around the world. Bringing a background in multimedia and passion for technology to the job, Cartwright delivers detailed product reviews, event coverage and industry news on a daily basis. Disclaimer: Tesla Shareholder from 20/01/2021

    Leave a Reply


    Latest posts


    Related articles