# Advance World Simulation With 3D Gaussian Splatting for Large-Scale Environment Reconstruction

## Abstract

Learn how to reconstruct a large scene for robotics testing using NVIDIA Omniverse NuRec Gaussian-based reconstruction technologies to perform multi-GPU training, object segmentation, and extraction. This lab will walk through core concepts with a step-by-step workflow for data capture, reconstruction, and object level-integration for robotics simulation in NVIDIA Isaac Lab. Prerequisite(s): Omniverse NuRec, Isaac Sim / Lab Explore training opportunities from the NVIDIA Deep Learning Institute  (DLI). Access a wide range of self-paced courses  or instructor-led virtual workshops  designed to build key skills in AI, HPC, graphics & simulation, and more.   Validate your skills with NVIDIA certification  and distinguish yourself in the industry.

## AI Summary

- The lab focused on creating 3D Gaussian Splat models from 2D images using the NVIDIA NuRec toolset, specifically the fVDB library.
- Participants learned how to capture high-quality data, including best practices such as ensuring high overlap between frames and avoiding motion blur.
- The lab covered the use of COLMAP to generate a sparse point cloud and camera poses, which are essential for initializing the Gaussian Splat.
- Segmentation techniques, including the GARField method, were introduced to help extract and clean up objects from the 3D Gaussian Splat.
- The lab concluded with a demonstration of how to import the Gaussian Splat into Isaac Sim for robotic simulation, including creating a USD file and a physical mesh.
- Additional resources and next steps were provided for further exploration of Gaussian Splatting and its integration into larger robotics pipelines.

## Transcript

**[00:00:10 – 00:00:14]** to making it to Thursday afternoon.

**[00:00:17 – 00:00:21]** things you'll do here, so hopefully we make it exciting

**[00:00:21 – 00:00:24]** and meaningful for everybody.

**[00:00:31 – 00:00:34]** The general field of ratings fields and more recently,

**[00:00:34 – 00:00:35]** Gaussian Splatting.

**[00:00:35 – 00:00:38]** So I'm excited to be your lead TA today.

**[00:00:38 – 00:00:42]** I wanted to introduce Zoe.

**[00:00:42 – 00:00:43]** Zoe, wave hello.

**[00:00:43 – 00:00:44]** Yes.

**[00:00:44 – 00:00:46]** Zoe will be your instructor.

**[00:00:46 – 00:00:51]** She's done the lion's share of work on this lab, and I think you'll.

**[00:00:51 – 00:00:54]** You'll be lucky to hear her walk through all these different

**[00:00:54 – 00:00:59]** techniques today, and she, like myself, also has a lot

**[00:00:59 – 00:01:05]** of experience in this area, and a degree in imaging science, so...

**[00:01:05 – 00:01:06]** Welcome, Zoe.

**[00:01:06 – 00:01:09]** So I don't know how many people have taken labs.

**[00:01:09 – 00:01:12]** We're at the end of the week, so probably a few of you have been to

**[00:01:12 – 00:01:16]** past GTCs or other training labs.

**[00:01:16 – 00:01:19]** But if you haven't, you'll see kind of like this sea

**[00:01:19 – 00:01:21]** of blue cups in front of you.

**[00:01:21 – 00:01:23]** Don't put drinks in them.

**[00:01:23 – 00:01:29]** But they're sort of our mechanisms for folks to get help from our our

**[00:01:29 – 00:01:31]** staff of teaching assistants today.

**[00:01:31 – 00:01:34]** So if you see anybody with this polo on, wave

**[00:01:34 – 00:01:37]** everybody, we've got a few.

**[00:01:37 – 00:01:41]** And you have a question or you're stuck or you need need

**[00:01:41 – 00:01:44]** some assistance, the idea is red cup, right?

**[00:01:44 – 00:01:47]** So hopefully things go smoothly and we don't see a sea of

**[00:01:47 – 00:01:52]** red cups, but but please yeah, make sure to use those and then

**[00:01:53 – 00:01:56]** It always happens where, once you get answered, go ahead

**[00:01:56 – 00:01:58]** and go back to blue so we don't come back to you.

**[00:01:58 – 00:02:03]** So that's just some logistics of how this goes.

**[00:02:03 – 00:02:07]** The session will end about 20 minutes early, and we'll hopefully

**[00:02:07 – 00:02:08]** have a little Q&A session.

**[00:02:09 – 00:02:13]** We do have special prizes up front for good questions, so just

**[00:02:13 – 00:02:16]** keep that in mind, don't be shy.

**[00:02:16 – 00:02:19]** There's some swag to be had.

**[00:02:19 – 00:02:20]** Anything else on my checklist?

**[00:02:20 – 00:02:24]** So, I guess, in general, I think that's everything I'm supposed

**[00:02:24 – 00:02:30]** to cover, and just one last thing of please turn off your cell

**[00:02:30 – 00:02:35]** phones so we don't interrupt Zoe's flow, but with that, I'm going

**[00:02:35 – 00:02:40]** to go ahead and hand it off to Zoe, and we'll kind of get things going,

**[00:02:40 – 00:02:46]** and you guys can follow along as she walks through the course.

**[00:02:46 – 00:02:50]** So if you haven't already, please go to the DLI course page.

**[00:02:50 – 00:02:52]** If you need the Wi-Fi, it's the same password that it's

**[00:02:52 – 00:02:53]** been all week, GTC2026.

**[00:02:53 – 00:02:59]** We're going to go ahead and dive right in.

**[00:03:05 – 00:03:08]** Okay, so we're all here for Advancing World Simulation

**[00:03:08 – 00:03:09]** with 3D Gaussian Splatting.

**[00:03:09 – 00:03:12]** But before we jump in, please enter the course code and

**[00:03:12 – 00:03:14]** start launching the notebook.

**[00:03:14 – 00:03:15]** It might take it a second.

**[00:03:15 – 00:03:18]** We've got a lot of moving parts in this DLI, so we just

**[00:03:18 – 00:03:19]** want you to get it started.

**[00:03:19 – 00:03:21]** So I'll give you just a couple seconds to type

**[00:03:21 – 00:03:23]** that course code in.

**[00:03:24 – 00:03:26]** And just make sure to click Start on that Jupyter Notebook

**[00:03:26 – 00:03:29]** so it starts spinning up in the background when we

**[00:03:29 – 00:03:32]** start getting started.

**[00:03:37 – 00:03:40]** As Brent mentioned, we have a great team of people here

**[00:03:40 – 00:03:41]** to help you today.

**[00:03:41 – 00:03:44]** Just, if you need help at any point, switch to the red

**[00:03:44 – 00:03:46]** cup and one of our TAs will be more than happy to come

**[00:03:46 – 00:03:48]** over and give you a hand.

**[00:03:48 – 00:03:52]** And if you still need the course code, it's down here, by the way.

**[00:03:54 – 00:03:58]** We're one of the last labs this week, I think the second

**[00:03:58 – 00:04:01]** to last lab of the whole week, but we are one of the first

**[00:04:01 – 00:04:04]** steps in this sort of robotics pipeline, this robotics path

**[00:04:04 – 00:04:08]** that we've had going on throughout some of our DLIs this week.

**[00:04:08 – 00:04:10]** So NuRec is this, that we're going to be talking about

**[00:04:11 – 00:04:13]** today, is an NVIDIA

**[00:04:13 – 00:04:16]** A set of tools that allow us to reconstruct real-world scenes.

**[00:04:16 – 00:04:18]** That's what we'll be diving into today, but it connects

**[00:04:18 – 00:04:20]** to a lot of our other DLIs.

**[00:04:20 – 00:04:23]** There was a Cosmos one yesterday, and then, of course, all the

**[00:04:23 – 00:04:28]** robotic simulation that you can do with the scene once you have it.

**[00:04:34 – 00:04:37]** NVIDIA has a lot of open models, frameworks, and stuff for robotics,

**[00:04:38 – 00:04:41]** and we're just going to be touching on a small portion of it today

**[00:04:41 – 00:04:46]** on the synthetic scene creation digital twin aspect, but we have a

**[00:04:46 – 00:04:49]** lot of pipelines for what we want, and we highly encourage you to

**[00:04:49 – 00:04:53]** look into whatever things we have to offer beyond just this today.

**[00:04:53 – 00:04:58]** We are specifically focusing on creating

**[00:04:58 – 00:05:03]** Simulations from real-world scenes using real-world data,

**[00:05:03 – 00:05:05]** and the reason we want to do this is, in the past, we've

**[00:05:05 – 00:05:06]** had specialist robots.

**[00:05:06 – 00:05:09]** Think of your robots that are on an assembly line, really

**[00:05:09 – 00:05:12]** good at tightening a bolt, and that's all they do.

**[00:05:12 – 00:05:16]** And then we've shifted to this more generalist, broad-scope

**[00:05:16 – 00:05:18]** robot that can do a lot of things, but maybe it doesn't

**[00:05:18 – 00:05:19]** do them necessarily well.

**[00:05:19 – 00:05:22]** It's not an expert, and where we want to go is this generalist,

**[00:05:22 – 00:05:25]** specialist robot that can do a lot of things, and it

**[00:05:25 – 00:05:27]** does all of those things well.

**[00:05:27 – 00:05:32]** But in order to do that, you're going to need a lot of data.

**[00:05:32 – 00:05:41]** And it is very cost-heavy, time-intensive to capture a

**[00:05:41 – 00:05:42]** lot of data to train these robots.

**[00:05:42 – 00:05:46]** What we can do is we can use synthetic data creation, we

**[00:05:46 – 00:05:49]** can use Gaussian Splatting like we're going to today to make

**[00:05:49 – 00:05:54]** 3D scenes from real-world images, and then use things like Cosmos to

**[00:05:54 – 00:05:59]** increase our synthetic data amount.

**[00:06:02 – 00:06:05]** Okay, now we're actually diving into this lab, now that we've

**[00:06:05 – 00:06:10]** got through all the Omniverse robotics pre-stuff, this is what

**[00:06:10 – 00:06:11]** we're actually focusing on today.

**[00:06:11 – 00:06:16]** We want to take 2D images, in this case a lovely video of

**[00:06:16 – 00:06:18]** our data set we call Safety Park.

**[00:06:18 – 00:06:21]** It's like a little pseudo-town for first responder training.

**[00:06:21 – 00:06:23]** We're going to take that video and we are going to turn it

**[00:06:23 – 00:06:27]** into a 3D model, a Gaussian Splat.

**[00:06:27 – 00:06:29]** And Reality Capture in general is this idea of taking the

**[00:06:29 – 00:06:35]** real world and bringing it into some sort of 3D representation.

**[00:06:35 – 00:06:37]** There's lots of ways you can do this.

**[00:06:37 – 00:06:39]** Today we're going to focus with just 2D imagery, but

**[00:06:39 – 00:06:43]** you can use LiDAR scans and stuff like that as well.

**[00:06:46 – 00:06:50]** Okay, so our full workflow, the fun stuff.

**[00:06:50 – 00:06:51]** The first thing we're gonna tell you about is how to

**[00:06:51 – 00:06:52]** capture your scene.

**[00:06:52 – 00:06:55]** This is the most important step of this entire pipeline.

**[00:06:55 – 00:06:58]** Then we're gonna do structure for motion because we're using

**[00:06:58 – 00:07:01]** just 2D images, no LiDAR point scans or anything like that.

**[00:07:01 – 00:07:05]** We're gonna need to run this to initialize our 3D Gaussian scene.

**[00:07:05 – 00:07:09]** And then, for this GTC, we've added a fun new thing, segmentation,

**[00:07:09 – 00:07:12]** where we can start ripping out parts of our scenes and

**[00:07:12 – 00:07:13]** playing around with them.

**[00:07:13 – 00:07:16]** And in order to actually use them in simulation and robotics,

**[00:07:16 – 00:07:18]** we're going to have to format and crop our scenes so they're

**[00:07:18 – 00:07:20]** ready for Isaac Sim.

**[00:07:20 – 00:07:22]** And then eventually we're going to mess around with our

**[00:07:22 – 00:07:26]** own little simulation in Isaac Sim.

**[00:07:29 – 00:07:34]** And this whole pipeline is powered by Omniverse NuRec.

**[00:07:34 – 00:07:38]** NuRec has the Python library that we're going to be using

**[00:07:38 – 00:07:41]** to do the actual reconstruction and segmentation of our scene,

**[00:07:41 – 00:07:45]** but it also has the renderer in Isaac Sim that the team built

**[00:07:45 – 00:07:50]** to render the splats efficiently and quickly for robotic simulation.

**[00:07:53 – 00:07:56]** NuRec has three groups of sort of what it consists of

**[00:07:56 – 00:07:59]** that are gonna mesh with our three lab objectives today.

**[00:07:59 – 00:08:02]** The first thing, it's got reconstruction APIs.

**[00:08:02 – 00:08:06]** We have a NuRec container that takes in images and LiDAR scans.

**[00:08:06 – 00:08:11]** We have 3D GROOT. We now have the 3D GS, the GSplot

**[00:08:11 – 00:08:13]** library on GitHub.

**[00:08:13 – 00:08:17]** There's also generative enhancement and segmentation models.

**[00:08:17 – 00:08:20]** Things like the Asset Harvester are a part of NuRec's fixer

**[00:08:20 – 00:08:22]** In order to try to remove some,

**[00:08:22 – 00:08:24]** Artifacts you might get from Gaussian Splatting, and then

**[00:08:24 – 00:08:29]** finally we have the rendering APIs where we can actually display.

**[00:08:29 – 00:08:35]** Specifically, we were going to be using FEDB from the

**[00:08:35 – 00:08:36]** reconstruction part.

**[00:08:36 – 00:08:38]** It's a Python library. You can pip install it.

**[00:08:38 – 00:08:41]** We're going to be doing segmentation, and then finally,

**[00:08:41 – 00:08:44]** of course, our rendering.

**[00:08:47 – 00:08:49]** First step of our pipeline I mentioned is the most important.

**[00:08:49 – 00:08:51]** It is how to capture data.

**[00:08:51 – 00:08:55]** If you do not have good data, you are not gonna get good results.

**[00:08:55 – 00:08:57]** It's important to note that Gaussian Splatting is not

**[00:08:57 – 00:08:58]** a generative method.

**[00:08:58 – 00:09:00]** It is reconstructing what you saw.

**[00:09:00 – 00:09:03]** So if you never took an image of it, you are never gonna reconstruct

**[00:09:03 – 00:09:05]** it because it was not there.

**[00:09:05 – 00:09:09]** There's some best practices we can do to get good data.

**[00:09:09 – 00:09:11]** High overlap between frames, specifically when you're just

**[00:09:11 – 00:09:14]** using 2D images and you're gonna wanna run COLMAP, you want about 60

**[00:09:14 – 00:09:18]** to 80% overlap between your frames.

**[00:09:18 – 00:09:21]** Quality lighting, you're going to, the lighting in your scene

**[00:09:21 – 00:09:25]** is going to be reflected in your final 3D Gaussian splat.

**[00:09:25 – 00:09:29]** If you're flying, you want to make sure you avoid motion blur.

**[00:09:29 – 00:09:32]** It's going to be harder for us to do downstream tasks

**[00:09:32 – 00:09:34]** if our images are blurry, and there's a couple of ways

**[00:09:34 – 00:09:38]** we can actually go about collecting aerial data if we're doing these

**[00:09:38 – 00:09:40]** large-scale geospatial captures.

**[00:09:40 – 00:09:41]** For example,

**[00:09:41 – 00:09:46]** If you have an oblique camera, so say a plane equipped with

**[00:09:46 – 00:09:48]** five cameras, one down, and then in each of the four cardinal

**[00:09:49 – 00:09:53]** directions, you can kind of do this raster pattern and

**[00:09:53 – 00:09:57]** keep the camera positions that correspond to the arrows.

**[00:09:57 – 00:09:59]** So in the center you're keeping all five, here you're just

**[00:09:59 – 00:10:02]** keeping the one that faces towards the center of your scene.

**[00:10:02 – 00:10:03]** You can also orbit.

**[00:10:03 – 00:10:06]** This is what Safety Park does.

**[00:10:06 – 00:10:09]** It's the camera's focused down and towards the center of the

**[00:10:09 – 00:10:10]** scene, and then you just orbit it.

**[00:10:10 – 00:10:14]** If you've got a larger scene, as we have here, you can do several

**[00:10:14 – 00:10:17]** of those overlapping orbits.

**[00:10:19 – 00:10:24]** Now, once we have data, that's not all we need to start making

**[00:10:24 – 00:10:24]** our Gaussian splat.

**[00:10:24 – 00:10:26]** We need to know a couple things.

**[00:10:26 – 00:10:29]** Where our cameras were in space when they took the photos, where

**[00:10:29 – 00:10:32]** they were pointing, and we need a sparse point cloud of our scene.

**[00:10:32 – 00:10:35]** And this is going to be used to initialize our Gaussian splat,

**[00:10:35 – 00:10:38]** but since we didn't use LIDAR or anything, we don't have that data.

**[00:10:38 – 00:10:40]** We just have our images, but that's no problem because

**[00:10:40 – 00:10:42]** we can run COLMAP.

**[00:10:42 – 00:10:45]** Are people here familiar with COLMAP?

**[00:10:47 – 00:10:49]** For the most part, yes. If you don't know, it's a structure

**[00:10:49 – 00:10:52]** from motion, multi-view stereo pipeline, and it's going to

**[00:10:52 – 00:10:55]** allow us to basically map out this information that we need.

**[00:10:55 – 00:10:58]** So we're going to start with our set of images and we're

**[00:10:58 – 00:11:02]** going to look for features, things in our images that

**[00:11:02 – 00:11:05]** are distinct and hopefully exist across several of them.

**[00:11:05 – 00:11:08]** So, for example, the corner of this building here is in

**[00:11:08 – 00:11:09]** several of our frames.

**[00:11:09 – 00:11:12]** So we're going to find hundreds, thousands of those features,

**[00:11:12 – 00:11:16]** and then we're going to use trig to basically triangulate

**[00:11:16 – 00:11:18]** where they were in 3D space.

**[00:11:18 – 00:11:22]** And we're going to slowly do this for like add one image at a time

**[00:11:22 – 00:11:25]** and do this for all of our images.

**[00:11:25 – 00:11:28]** Until eventually we get the sparse point cloud, which

**[00:11:28 – 00:11:32]** is the center thing here, all these points are safety park, and then

**[00:11:32 – 00:11:36]** our ring of cameras are these red guys, so we've got the positions

**[00:11:36 – 00:11:38]** and the poses of our cameras, and that sparse point cloud we

**[00:11:38 – 00:11:43]** need, all with some triangulation.

**[00:11:43 – 00:11:47]** And now, we can finally actually make our splat.

**[00:11:47 – 00:11:50]** So, are people here familiar with Gaussian Splatting?

**[00:11:50 – 00:11:52]** It's been around for a couple of years.

**[00:11:53 – 00:11:56]** If you're not, think of a Gaussian Splatting seen as

**[00:11:56 – 00:12:00]** a point cloud, but instead of the points being just little

**[00:12:00 – 00:12:02]** dots, they're three-dimensional Gaussian functions.

**[00:12:02 – 00:12:03]** So they're like,

**[00:12:03 – 00:12:07]** Blobs is usually what I like to think of them as, and this is going

**[00:12:07 – 00:12:09]** to be a trained representation.

**[00:12:09 – 00:12:10]** And the way we're going to train this is we're going

**[00:12:10 – 00:12:14]** to first take our nice output that we just got from COLMAP,

**[00:12:14 – 00:12:17]** and we are going to convert those sparse points into Gaussians.

**[00:12:17 – 00:12:20]** We're just going to randomly initialize our scene by converting

**[00:12:20 – 00:12:23]** them to three-dimensional Gaussian functions, and we

**[00:12:23 – 00:12:25]** now know where all the cameras are.

**[00:12:25 – 00:12:29]** So from the same locations in space as our original cameras,

**[00:12:29 – 00:12:30]** we're going to render an image.

**[00:12:30 – 00:12:32]** We're going to get a predicted image, and then we're going

**[00:12:32 – 00:12:36]** to compare it to that ground truth image that we actually have.

**[00:12:36 – 00:12:37]** And we're going to see how would they differ.

**[00:12:37 – 00:12:41]** And based on that loss, we can update our 3D Gaussians.

**[00:12:41 – 00:12:45]** And we basically just do this over and over again, backdrop, until

**[00:12:45 – 00:12:49]** eventually those renderings, our predicted images, look exactly the

**[00:12:49 – 00:12:51]** same as our ground truth images.

**[00:12:51 – 00:12:54]** And as a result, the views in between where our cameras

**[00:12:54 – 00:12:57]** originally were are pretty good.

**[00:12:57 – 00:12:59]** Any questions?

**[00:13:00 – 00:13:03]** Okay, sweet.

**[00:13:08 – 00:13:11]** So now, we can get into how we actually do this with New

**[00:13:11 – 00:13:16]** Rec, and specifically, we're going to be using fvdb.

**[00:13:16 – 00:13:20]** So, fvdb does a lot more than Gaussian Splatting.

**[00:13:20 – 00:13:25]** It's really good at doing large-scale spatial intelligence,

**[00:13:25 – 00:13:28]** and the reason that it is is because it is based off

**[00:13:28 – 00:13:32]** of, we built it off of NanoVDB, which was this GPU-accelerated,

**[00:13:32 – 00:13:36]** voxel-based database structure that was used for special effects.

**[00:13:36 – 00:13:38]** Think of like the Marvel movie special effects.

**[00:13:38 – 00:13:41]** You do this really quickly on the GPU with NanoVDB, but what

**[00:13:41 – 00:13:44]** if you added AI operations to that?

**[00:13:44 – 00:13:47]** And now you can have the best in class spatial intelligence

**[00:13:47 – 00:13:52]** where you can do cool things with all this volumetric data we have

**[00:13:52 – 00:13:55]** and do it quickly and efficiently.

**[00:13:55 – 00:13:59]** So fvdb does a ton of things beyond Gaussian Splatting.

**[00:13:59 – 00:14:02]** Oops, sorry, technical difficulties.

**[00:14:02 – 00:14:03]** There we go.

**[00:14:03 – 00:14:05]** And it's meant to be developer friendly.

**[00:14:05 – 00:14:07]** It's meant to make your life easier.

**[00:14:07 – 00:14:10]** Because before, if you wanted to do something like Gaussian

**[00:14:10 – 00:14:13]** Splatting and eventually segment it and bring it into Isaac

**[00:14:13 – 00:14:15]** Sim, you're going to be grabbing a ton of tools from different

**[00:14:15 – 00:14:18]** places and everything scattered.

**[00:14:18 – 00:14:23]** Fvdb brings everything together to create this unified deep learning

**[00:14:23 – 00:14:27]** on volumetric data framework that you can just pip install.

**[00:14:27 – 00:14:30]** And it's meant to be both easy to use for you out of

**[00:14:30 – 00:14:32]** the box if you just want to do a simple reconstruction,

**[00:14:32 – 00:14:35]** you can do it on a command line in like five minutes, or if you're

**[00:14:35 – 00:14:38]** a developer and want to build on it, it's very well documented and

**[00:14:38 – 00:14:41]** it's very easy to plug and play.

**[00:14:43 – 00:14:46]** For Gaussian Splatting specifically, the fvdb

**[00:14:46 – 00:14:50]** team has optimized the data structure, does tiled image

**[00:14:50 – 00:14:53]** loading, sparse pixel training, and has tuned the parameters

**[00:14:53 – 00:14:58]** so it works pretty decently out of the box on any data set.

**[00:14:58 – 00:15:02]** And all of these speedups make it so we can run really large-scale

**[00:15:02 – 00:15:07]** scenes that could be thousands upon thousands of images large across

**[00:15:07 – 00:15:10]** multiple GPUs without immediately running out of our memory.

**[00:15:10 – 00:15:16]** So, for example, this scene covers about seven square kilometers,

**[00:15:16 – 00:15:20]** and 800 million Gaussians make up this scene, trained

**[00:15:20 – 00:15:22]** across four GPUs, so it's huge.

**[00:15:22 – 00:15:26]** You can get as big as you need to get.

**[00:15:27 – 00:15:29]** On the other hand, if you have more unique scenes, we can

**[00:15:29 – 00:15:31]** also do underwater environments.

**[00:15:32 – 00:15:35]** So this is a submarine sitting on the bottom of the ocean floor.

**[00:15:35 – 00:15:37]** It's a video with all the little fishes swimming around,

**[00:15:37 – 00:15:41]** and we were able to reconstruct the nose of that submarine

**[00:15:41 – 00:15:42]** sitting on the ocean floor.

**[00:15:42 – 00:15:47]** So don't think yourself as limited to just normal outdoor imagery.

**[00:15:47 – 00:15:50]** We can do underwater, we can do indoor, we can do

**[00:15:50 – 00:15:52]** large-scale aerial.

**[00:15:53 – 00:15:57]** Before we get training, does anyone have any questions?

**[00:15:57 – 00:15:59]** I know I kind of talk fast sometimes.

**[00:15:59 – 00:16:03]** I was told that VREC creates the mesh as well as the

**[00:16:03 – 00:16:05]** Gaussian Splatting.

**[00:16:05 – 00:16:09]** Yes, we will be meshing them in the next part.

**[00:16:11 – 00:16:16]** Okay, if there are no other questions, oh.

**[00:16:22 – 00:16:26]** I believe the slides should be on the course page.

**[00:16:26 – 00:16:29]** If anyone wants access to them, they should be towards

**[00:16:29 – 00:16:32]** the bottom, I believe.

**[00:16:35 – 00:16:39]** Once you are ready, feel free to launch the notebook.

**[00:16:39 – 00:16:43]** And then the first part of this lab is going to be going through how

**[00:16:43 – 00:16:45]** you train your own Gaussian Splat.

**[00:16:45 – 00:16:48]** So once you launch, go to this Gaussian Splatting Notebook.

**[00:16:48 – 00:16:52]** It's going to walk you through all the steps.

**[00:16:52 – 00:16:55]** There's a lot of parameters you can play with if you want.

**[00:16:55 – 00:16:57]** Feel free to just learn and play.

**[00:16:57 – 00:17:00]** And if you have any questions or concerns, let us know.

**[00:17:00 – 00:17:03]** And once again, it's just this 01 Gaussian Splatting

**[00:17:03 – 00:17:06]** Notebook in the environment.

**[00:17:06 – 00:17:10]** And sometimes training can take a hot second.

**[00:17:10 – 00:17:13]** So we have created all the files that you will need for you.

**[00:17:13 – 00:17:15]** So if something doesn't train to completion or it's taking too

**[00:17:15 – 00:17:17]** long, you can just stop the kernel.

**[00:17:17 – 00:17:20]** You can stop the cell and you will still have everything

**[00:17:20 – 00:17:22]** you need to go to the next step.

**[00:17:22 – 00:17:24]** So don't worry about that.

**[00:17:27 – 00:17:30]** Thank you.

**[00:17:31 – 00:17:32]** Okay, how are we all doing?

**[00:17:32 – 00:17:35]** Did we all get through the notebook?

**[00:17:35 – 00:17:39]** Do you have beautiful splats?

**[00:17:39 – 00:17:41]** Oh, gosh, yeah, I love technology.

**[00:17:42 – 00:17:52]** Okay, we'll quickly sort of run through what we just learned.

**[00:17:56 – 00:17:58]** Once you guys are done with this stuff, if you could please,

**[00:17:59 – 00:18:01]** for me, kill your kernel.

**[00:18:01 – 00:18:03]** The second notebook tends to not play nice with the first

**[00:18:03 – 00:18:07]** notebook file, so if you could just go up to the top, go to kernel,

**[00:18:07 – 00:18:14]** and restart it, that should make the second part flow much nicer.

**[00:18:17 – 00:18:20]** Okay, and so in this section, we learned a few things.

**[00:18:21 – 00:18:23]** We learned how to capture good data.

**[00:18:23 – 00:18:26]** We learned how to create the input we need for Gaussian

**[00:18:26 – 00:18:28]** Splatting from just two images.

**[00:18:28 – 00:18:31]** You guys messed around with the basics of FEDV reality

**[00:18:31 – 00:18:35]** capture, and were able to train your own Gaussian Splatting scene.

**[00:18:35 – 00:18:38]** Does anyone have any questions on any parts of that first

**[00:18:38 – 00:18:41]** module of the lab?

**[00:18:42 – 00:18:46]** The question was why we're not using QSFM or something

**[00:18:46 – 00:18:50]** like that, and that is, you can.

**[00:18:51 – 00:18:55]** We are working on, right after GTC, expanding on documentation

**[00:18:55 – 00:18:58]** on other ways to get the structure for motion input that's not

**[00:18:58 – 00:19:01]** just from COLMAP, but COLMAP, of course, has been the most widely

**[00:19:01 – 00:19:04]** documented way to get this input.

**[00:19:04 – 00:19:06]** Is there a minimum number of images?

**[00:19:06 – 00:19:10]** Usually for a scene that's like one focus, you want between

**[00:19:10 – 00:19:11]** 100 and 200 images.

**[00:19:11 – 00:19:16]** So, I think Safety Park is like 120, 150, something like that.

**[00:19:16 – 00:19:19]** For bigger, large aerial reconstructions, you can

**[00:19:19 – 00:19:23]** get on the magnitude of thousands if you needed.

**[00:19:26 – 00:19:30]** Yeah, go ahead. Yeah, so, how does this reconstruction

**[00:19:30 – 00:19:37]** and then building the 3D model work for, like, a ground vehicle?

**[00:19:37 – 00:19:40]** Yeah, so...

**[00:19:40 – 00:19:45]** We have a Nurek container that is really good with autonomous

**[00:19:45 – 00:19:48]** vehicles, so it's focused more on that ground imagery, and

**[00:19:48 – 00:19:52]** it uses both LiDAR and 2D images as input, and that one's really

**[00:19:52 – 00:19:55]** built for that sort of ground from a car level imagery, so I'd

**[00:19:56 – 00:20:00]** highly recommend checking that out.

**[00:20:00 – 00:20:03]** I believe it's in an open beta.

**[00:20:05 – 00:20:08]** Oh, yes, we've got a microphone.

**[00:20:08 – 00:20:10]** There's a downsample command.

**[00:20:10 – 00:20:14]** What do you guys downsample to?

**[00:20:14 – 00:20:18]** So you can choose to, your reconstruction is going to run

**[00:20:18 – 00:20:20]** faster if your images are smaller.

**[00:20:20 – 00:20:24]** So you can just put in like two, four, and it will half,

**[00:20:24 – 00:20:25]** quarter size your images.

**[00:20:25 – 00:20:26]** And it'll run faster.

**[00:20:26 – 00:20:29]** You'll have less resolution, but you'll get a model

**[00:20:29 – 00:20:31]** out a lot quicker.

**[00:20:31 – 00:20:34]** Just the number of the density of splats would reduce?

**[00:20:34 – 00:20:39]** It's the input images are actually being down sampled.

**[00:20:39 – 00:20:41]** So how does it affect the output?

**[00:20:41 – 00:20:44]** So there's going to be less...

**[00:20:45 – 00:20:47]** When we're rendering back our splats and comparing and

**[00:20:47 – 00:20:50]** doing our loss, there's going to be a lot less detail in

**[00:20:50 – 00:20:52]** there because the images are smaller, so as a result, there's

**[00:20:53 – 00:20:56]** going to be a lot less detail in your splats, but depending on

**[00:20:56 – 00:20:59]** your task, you might not need all that detail and rather have your

**[00:20:59 – 00:21:04]** splats back in a minute instead of five minutes, if that makes sense.

**[00:21:09 – 00:21:14]** Can you speak to the effects of in-scene movement, like

**[00:21:14 – 00:21:17]** in this safety park example, if there's like a truck that

**[00:21:17 – 00:21:21]** moved around or maybe a bird that flew underneath the drone?

**[00:21:21 – 00:21:26]** Yeah, so it's best practice that anything moving in your

**[00:21:26 – 00:21:29]** scene, you kind of...

**[00:21:30 – 00:21:34]** If it's not masked out, COLMAP is going to try to use that as

**[00:21:34 – 00:21:37]** a feature, and then if that feature moves, you're not going to be able

**[00:21:37 – 00:21:38]** to triangulate back accordingly.

**[00:21:38 – 00:21:41]** So what we did for this scene is we masked out all the fish

**[00:21:41 – 00:21:44]** before we ran COLMAP, and they were also masked out

**[00:21:44 – 00:21:48]** when we trained our Gaussian Splat, so we sort of reconstruct

**[00:21:49 – 00:21:52]** around the fish, and we still get a pretty good reconstruction, even

**[00:21:52 – 00:21:55]** though there were like 100 fish making their home in the submarine.

**[00:21:55 – 00:21:57]** Does that answer your question?

**[00:21:58 – 00:22:01]** What do you mean by mask the map?

**[00:22:01 – 00:22:06]** COLMAP can literally take a folder of images that are just mask

**[00:22:06 – 00:22:10]** images, so for each image, it's like, you can use this information,

**[00:22:10 – 00:22:12]** but you can't use this information.

**[00:22:12 – 00:22:15]** And it's just a black and white image.

**[00:22:17 – 00:22:20]** Exactly, image segmentation.

**[00:22:21 – 00:22:28]** Traditional machine learning techniques.

**[00:22:28 – 00:22:32]** On the segmentation, but there's script in there saying I just don't

**[00:22:32 – 00:22:34]** use this black segmented thing.

**[00:22:34 – 00:22:36]** Yeah, COLMAP, yeah.

**[00:22:36 – 00:22:38]** COLMAP can just take a folder of images.

**[00:22:38 – 00:22:39]** Oops, sorry, I'm standing in front of the speaker.

**[00:22:39 – 00:22:42]** Let me get you this before you, no one else can hear you.

**[00:22:42 – 00:22:43]** Oh, okay. Yeah.

**[00:22:43 – 00:22:46]** So it's segmented out, but just automatically it's in

**[00:22:46 – 00:22:49]** there that it just won't use the black segmented part.

**[00:22:49 – 00:22:52]** Yeah, if you give COLMAP a folder of images, if you give FADB the

**[00:22:52 – 00:22:56]** folder of images that are the mask, it knows automatically what to

**[00:22:56 – 00:22:58]** use and what not to use from that.

**[00:22:59 – 00:23:01]** I did have to generate the mask myself, but I just created a simple

**[00:23:01 – 00:23:03]** algorithm and be like, is fish?

**[00:23:03 – 00:23:05]** Mask.

**[00:23:06 – 00:23:10]** So I'm pretty new to this, but do you need the camera

**[00:23:10 – 00:23:13]** location for each of these, like as it's flying around,

**[00:23:13 – 00:23:15]** or does it just deduce that?

**[00:23:15 – 00:23:19]** Like, do you need GPS for every single image?

**[00:23:19 – 00:23:23]** So, COLMAP will figure that information out for you, but

**[00:23:23 – 00:23:27]** if you do have GPS information, you can feed that to COLMAP,

**[00:23:27 – 00:23:31]** and both COLMAP and FEDB can use that to make your scene

**[00:23:31 – 00:23:33]** scaled right, so like a meter is about a meter because it

**[00:23:33 – 00:23:36]** has that GPS information, and when you throw it into

**[00:23:36 – 00:23:40]** Isaac Sim, then a meter is still a meter and it's all great, But

**[00:23:40 – 00:23:45]** it might not be enough information, like accurate enough, to actually

**[00:23:45 – 00:23:50]** use them as camera positions for training, if that makes sense.

**[00:23:50 – 00:23:54]** Kind of to piggyback off of that question, could you explain

**[00:23:54 – 00:23:59]** the importance of the intrinsic camera parameters on the quality

**[00:23:59 – 00:24:02]** of the solve and constraining the solver with that kind

**[00:24:02 – 00:24:03]** of prior information?

**[00:24:03 – 00:24:06]** For COLMAP?

**[00:24:06 – 00:24:10]** For COLMAP or RealityCapture or any solver.

**[00:24:10 – 00:24:12]** But right now, for COLMAP, the way we run it is that

**[00:24:12 – 00:24:17]** we get in a set of images that are distorted back as if they were

**[00:24:17 – 00:24:20]** taken from an ideal pinhole camera.

**[00:24:20 – 00:24:25]** So as long as COLMAP can take those intrinsic parameters and knows how

**[00:24:25 – 00:24:30]** to undistort and bring back to this perfect world, we're good to go.

**[00:24:40 – 00:24:47]** Does FEDB optimize Gaussians in time and space, like NuRec?

**[00:24:48 – 00:24:51]** Right now, just in space.

**[00:24:51 – 00:24:54]** We're not doing any, like, moving Gaussians or anything

**[00:24:54 – 00:24:57]** like that at the moment in FEDB.

**[00:24:57 – 00:25:03]** Okay. We are going to move on to our second section here.

**[00:25:03 – 00:25:06]** We are going to actually start segmenting our splats.

**[00:25:06 – 00:25:10]** So let's get...

**[00:25:11 – 00:25:12]** There we go.

**[00:25:12 – 00:25:15]** Okay, so we're still gonna use FEDB for this.

**[00:25:15 – 00:25:18]** There's also like the Asset Harvester and NuRec, but FEDB

**[00:25:18 – 00:25:23]** has been working on implementing various segmentation methods

**[00:25:23 – 00:25:25]** for Gaussian Splatting, and the one we're gonna use today

**[00:25:25 – 00:25:27]** is called GARFEDB.

**[00:25:27 – 00:25:31]** It's an adaptation of a paper called GARField, like the

**[00:25:31 – 00:25:37]** cat, made to work with this sort of voxel-based data structure.

**[00:25:37 – 00:25:39]** And with it, we're able to query our scene at some scale.

**[00:25:39 – 00:25:42]** Like, let's say I want objects that are about 20% of my scene

**[00:25:42 – 00:25:45]** scale, and then I can get all that segmentation out.

**[00:25:45 – 00:25:48]** So we're going to briefly go over how this works.

**[00:25:48 – 00:25:52]** If you'd like to know more, there's some links inside the notebooks

**[00:25:52 – 00:25:54]** that you'll be going through for segmentation, and you can really

**[00:25:54 – 00:25:56]** dive into that paper if you'd like.

**[00:25:56 – 00:26:01]** So the first thing we do for GarField is we are going to

**[00:26:01 – 00:26:04]** use Segment Anything to get 2D masks from our original images.

**[00:26:04 – 00:26:08]** So our original images, and then we get a bunch of masks,

**[00:26:08 – 00:26:10]** but our C's in 3D.

**[00:26:10 – 00:26:14]** So we've got to instill those back to 3D, basically project them

**[00:26:14 – 00:26:17]** back onto our 3D Gaussian splat.

**[00:26:17 – 00:26:20]** And that might kind of seem like we're done, but the issue

**[00:26:20 – 00:26:23]** is we have one segment here for the front of the building,

**[00:26:23 – 00:26:26]** and we're going to have one for the back and one for the side,

**[00:26:26 – 00:26:31]** and you're going to have a ton of segments for just one object.

**[00:26:31 – 00:26:34]** So we need a way to get these segments a little bit nicer,

**[00:26:34 – 00:26:36]** a little bit cleaner.

**[00:26:37 – 00:26:39]** Before we start training anything, though, we also

**[00:26:39 – 00:26:40]** need one more thing.

**[00:26:40 – 00:26:44]** It's gonna be a set of sparse voxel grids.

**[00:26:44 – 00:26:48]** So let's take a little pseudo scene here where we have three

**[00:26:48 – 00:26:50]** whole splats in our scene, the red, the green one, and

**[00:26:50 – 00:26:54]** the pink one, and we are going to create a sparse voxel grid.

**[00:26:54 – 00:26:59]** The sparse here just means if there is nothing in these places,

**[00:26:59 – 00:27:00]** why would we put a voxel there?

**[00:27:00 – 00:27:04]** We're only gonna put voxels where our data are to save time.

**[00:27:04 – 00:27:07]** And space, and be efficient, because FEDB is really good

**[00:27:07 – 00:27:10]** with voxels, so we can do this very efficiently.

**[00:27:10 – 00:27:13]** And we're going to make these at various resolutions.

**[00:27:13 – 00:27:16]** In this example, we're just using two, but in actual training

**[00:27:16 – 00:27:19]** I think it's about 16 or something.

**[00:27:19 – 00:27:21]** What do these voxels actually mean?

**[00:27:21 – 00:27:23]** So they're going to store...

**[00:27:24 – 00:27:28]** Information about what lives there in the 3D scene that we're

**[00:27:28 – 00:27:31]** going to use when we curate our model to get out of segmentation.

**[00:27:31 – 00:27:33]** So it's going to have some sort of information about

**[00:27:33 – 00:27:37]** what goes on here in 3D space.

**[00:27:39 – 00:27:41]** Now let's take a look at our actual model.

**[00:27:41 – 00:27:43]** As I mentioned earlier, it gives a scale.

**[00:27:43 – 00:27:45]** It can be in meters or it can be like some percentage

**[00:27:46 – 00:27:49]** of your total scene size.

**[00:27:49 – 00:27:52]** And then you're going to append all of the features in those

**[00:27:52 – 00:27:53]** voxels for a given splat.

**[00:27:53 – 00:27:55]** So we're looking at this red splat.

**[00:27:55 – 00:27:58]** So whatever information's in this voxel, I'm gonna send

**[00:27:58 – 00:28:01]** that in, and then I'm going to do the same for the other resolutions,

**[00:28:01 – 00:28:03]** and append that all in.

**[00:28:03 – 00:28:08]** And then it goes through a multi-layered Receptron, back

**[00:28:08 – 00:28:10]** to like intro to AI.

**[00:28:10 – 00:28:14]** And we're going to get from this a feature.

**[00:28:14 – 00:28:19]** Some embedding, some representation Of.

**[00:28:20 – 00:28:29]** Now that doesn't actually give us any segmentations

**[00:28:29 – 00:28:31]** yet, so what we're going to do is we're going to run that

**[00:28:31 – 00:28:33]** on all the splats in this scene.

**[00:28:33 – 00:28:36]** So each splat is going to have some sort of vector at

**[00:28:36 – 00:28:39]** this scene scale that we queried.

**[00:28:39 – 00:28:44]** Gaussians that have similar feature vectors are probably

**[00:28:44 – 00:28:48]** part of the same object, so they should be segmented together, and

**[00:28:48 – 00:28:51]** we can run something like k-beans or any other clustering algorithm

**[00:28:51 – 00:28:55]** to group these together, and then we get all of those segmentations,

**[00:28:55 – 00:28:58]** but we totally glossed over how we actually train this

**[00:28:58 – 00:29:02]** thing, so we are learning both what goes in these boxes and what goes

**[00:29:02 – 00:29:07]** in this multilayer perceptron when we are training our Garfield model.

**[00:29:07 – 00:29:11]** So we're going to learn at different resolutions what

**[00:29:11 – 00:29:14]** kind of lives in 3D space here, and then we're going

**[00:29:14 – 00:29:16]** to learn how to actually create these features for each splat

**[00:29:16 – 00:29:19]** in the multi-layer perceptron here.

**[00:29:19 – 00:29:24]** It's a bit complicated, so I recommend if you want to

**[00:29:24 – 00:29:28]** learn more to read the GARField paper, but at the end of the

**[00:29:28 – 00:29:31]** day we're getting out features that we just cluster together and we get

**[00:29:31 – 00:29:36]** all of our segmented objects out.

**[00:29:38 – 00:29:42]** Now, this part is going to be in Segmentation Notebook

**[00:29:42 – 00:29:45]** 02, so you guys can go to that and there will be steps

**[00:29:45 – 00:29:52]** walking you through how to use FEDB to segment out our objects and then

**[00:29:52 – 00:29:55]** actually look at those segments and then we can rip them out so

**[00:29:55 – 00:29:58]** we can use them later in Isaac Sim.

**[00:29:58 – 00:30:00]** Any questions?

**[00:30:02 – 00:30:04]** Yes, do we have the microphone around still?

**[00:30:04 – 00:30:06]** Thank you.

**[00:30:12 – 00:30:15]** Thank you for explaining this.

**[00:30:15 – 00:30:20]** May I ask how we chose the voxel grid?

**[00:30:20 – 00:30:24]** How we chose the... Yes, so our voxel grid is... Let me

**[00:30:24 – 00:30:26]** go back to our little diagram here.

**[00:30:26 – 00:30:28]** We're going to place voxels based on the means of

**[00:30:28 – 00:30:30]** our Gaussian splats.

**[00:30:30 – 00:30:33]** So wherever there are splats, we'll try to place them.

**[00:30:34 – 00:30:37]** So the topology might change based on how many voxels we're

**[00:30:37 – 00:30:40]** using, There are a number of examples that are

**[00:30:40 – 00:30:46]** We're going to create 16 of various sizes, so it's like 16

**[00:30:46 – 00:30:52]** predetermined resolutions of sparse voxel grids based on the location

**[00:30:52 – 00:30:55]** of the means of the Gaussians.

**[00:31:03 – 00:31:06]** We always make all of 16, and then the scale parameter is going

**[00:31:07 – 00:31:09]** to be an input into the model.

**[00:31:09 – 00:31:12]** The resolution of your voxel grids and the scale parameter

**[00:31:12 – 00:31:15]** you query with are unrelated.

**[00:31:18 – 00:31:25]** I have another question, but it's related to the first notebook.

**[00:31:25 – 00:31:27]** Yeah, go ahead.

**[00:31:27 – 00:31:33]** After doing the segmentation on the fishes,

**[00:31:35 – 00:31:41]** I mean, if a fish is in front of the sea, that's not related

**[00:31:41 – 00:31:44]** to the submarine, so it's fine.

**[00:31:44 – 00:31:49]** But what if the fish is in front of a submarine?

**[00:31:49 – 00:31:52]** How do we handle such occlusions?

**[00:31:52 – 00:31:57]** Yeah, because we have views from many different angles, if a fish

**[00:31:57 – 00:32:00]** is blocking a good chunk of the submarine in one angle, hopefully

**[00:32:00 – 00:32:02]** in the next one, we kind of...

**[00:32:02 – 00:32:04]** He's not, if that makes sense.

**[00:32:04 – 00:32:09]** Because we have all these angles, we can kind of expect

**[00:32:09 – 00:32:11]** that we have enough data about each point in space that we

**[00:32:11 – 00:32:16]** can reconstruct it, even if a fish is blocking it in some of them.

**[00:32:18 – 00:32:25]** In FEDB, it actually sets the loss gradient to zero

**[00:32:25 – 00:32:29]** for those pixels during that one particular back propagation.

**[00:32:29 – 00:32:33]** So that's how the, that image doesn't contribute to moving

**[00:32:33 – 00:32:36]** the blocked Gaussians.

**[00:32:36 – 00:32:40]** But then, as Zoe was saying, the other angles do, so hopefully

**[00:32:40 – 00:32:43]** that answers your question.

**[00:32:51 – 00:32:55]** Okay, feel free to get started in going through

**[00:32:55 – 00:32:58]** the segmentation notebook.

**[00:32:58 – 00:33:01]** If you have any questions, you can use the CUP system or flag one

**[00:33:01 – 00:33:03]** of us down, we're happy to help.

**[00:33:03 – 00:33:05]** Again, this is just going to go through how to segment,

**[00:33:05 – 00:33:09]** how to train a GARField network and how to get segments out

**[00:33:09 – 00:33:11]** from that, and then how to actually rip those segments out from

**[00:33:11 – 00:33:15]** our scene so we can use them later.

**[00:33:17 – 00:33:21]** Have we all gotten some nice segmentations out?

**[00:33:22 – 00:33:28]** So just for a brief recap of what we've done here.

**[00:33:30 – 00:33:33]** Of course, input all the things we're gonna need.

**[00:33:33 – 00:33:36]** There's a big stretch of code that just makes the viewer work.

**[00:33:36 – 00:33:40]** But then you actually trained your GARField model here.

**[00:33:40 – 00:33:45]** And then you were able to query that model for some given scale and

**[00:33:45 – 00:33:47]** visualize all of those segments.

**[00:33:47 – 00:33:51]** And if you'd like to rip out your own segments to throw

**[00:33:51 – 00:33:53]** into Isaac Simulator, feel free.

**[00:33:53 – 00:33:57]** With this little script here at the bottom, you can choose what number

**[00:33:57 – 00:34:02]** of cluster ID you want, and we'll save these so you can access them

**[00:34:02 – 00:34:05]** on the Isaac Lab compute later.

**[00:34:05 – 00:34:07]** So feel free to do that if you want.

**[00:34:07 – 00:34:12]** But we also have some free generated for you as well.

**[00:34:15 – 00:34:20]** Okay, let me get back to my slides.

**[00:34:28 – 00:34:32]** Okay, sorry, technical difficulties.

**[00:34:35 – 00:34:37]** There we go.

**[00:34:38 – 00:34:44]** Okay, FADB is that GARField is not the only segmentation

**[00:34:44 – 00:34:45]** model that they're working on.

**[00:34:45 – 00:34:48]** Right now they're also working on LangSplat and they plan

**[00:34:48 – 00:34:49]** on implementing others.

**[00:34:49 – 00:34:52]** This one, instead of just like, find me all the segments in this

**[00:34:52 – 00:34:55]** scene at this scale, this one you query it with some sort of prompt.

**[00:34:55 – 00:34:59]** In this case we can ask it to please find me the truck.

**[00:34:59 – 00:35:01]** And then you get out the segmentation of the truck

**[00:35:01 – 00:35:05]** from your 3D scene and can follow the same workflow that we just did

**[00:35:05 – 00:35:08]** to rip out the individual splats.

**[00:35:09 – 00:35:11]** This is still in development, but you're welcome to take

**[00:35:11 – 00:35:14]** a look and try it out.

**[00:35:14 – 00:35:16]** We have links on the course page that will bring you to

**[00:35:16 – 00:35:19]** this so you can try it.

**[00:35:22 – 00:35:26]** Okay, in quick review, we know about the segmentation

**[00:35:26 – 00:35:29]** things FEDB is working on, and you trained your own model,

**[00:35:29 – 00:35:33]** and we can visualize and pick out segments that we can use for later.

**[00:35:33 – 00:35:36]** Specifically, we're gonna use them for robotic simulation.

**[00:35:36 – 00:35:38]** So back on Omniverse, as I mentioned earlier, it's

**[00:35:38 – 00:35:42]** this collection of open libraries APIs for industrial

**[00:35:42 – 00:35:45]** and physical AI applications.

**[00:35:45 – 00:35:47]** We are specifically going to use

**[00:35:47 – 00:35:50]** Isaac Sim, we're going to sort of use this digital twin

**[00:35:50 – 00:35:53]** physical AI use case.

**[00:35:53 – 00:35:56]** You can do lots of things with it, you know, robotic fleet

**[00:35:56 – 00:35:58]** simulation, process simulation.

**[00:35:58 – 00:36:01]** We're going to specifically just have a little spot robot

**[00:36:01 – 00:36:03]** wander around our seat.

**[00:36:03 – 00:36:09]** In Isaac Sim, he was actually trained, his locomotion policy was

**[00:36:09 – 00:36:12]** trained on safety parks, so that's where he learned how to walk.

**[00:36:12 – 00:36:15]** We used Isaac Lab to do the training, and we're gonna use Isaac

**[00:36:15 – 00:36:17]** Sim to do the actual simulation where you guys can move him around

**[00:36:17 – 00:36:20]** yourself and explore the scene.

**[00:36:21 – 00:36:24]** And of course, this is all powered once again by Nurek,

**[00:36:24 – 00:36:29]** our rendering API that actually allows us to see Gaussian Splat.

**[00:36:29 – 00:36:34]** We are gonna take our images, we train to Gaussian Splat, and that's

**[00:36:34 – 00:36:39]** gonna go into Isaac Sim, into our Omniverse ecosystem, and then you

**[00:36:39 – 00:36:45]** can plop a robot if you have some sort of CAD or URDF model, and then

**[00:36:45 – 00:36:47]** you can use them for simulation.

**[00:36:47 – 00:36:51]** So this is, all this stuff has led up to here.

**[00:36:53 – 00:37:01]** Now, the splats are usually formatted as ply files, PLY files.

**[00:37:01 – 00:37:03]** Isaac Sim does not speak that kind of language.

**[00:37:03 – 00:37:05]** Omniverse is usually OpenUSD.

**[00:37:05 – 00:37:09]** Now, this is a standard for industrial digitalization

**[00:37:09 – 00:37:11]** and physical AI workloads.

**[00:37:11 – 00:37:15]** It is a standard across companies, Adobe uses it, Pixar uses

**[00:37:15 – 00:37:18]** it, we use it, everyone uses it, and I also just recently

**[00:37:18 – 00:37:22]** created a standard for Gaussian Splat scenes to be represented

**[00:37:22 – 00:37:23]** in USD specifically.

**[00:37:23 – 00:37:28]** So our renderer uses that standard in order to work.

**[00:37:28 – 00:37:31]** So we need to make one of these.

**[00:37:31 – 00:37:35]** But we also need to make some sort of physical surface for

**[00:37:35 – 00:37:37]** our robot to walk on.

**[00:37:37 – 00:37:41]** So we can put the Gaussian splat in a USD, but it still

**[00:37:41 – 00:37:42]** doesn't have physics.

**[00:37:42 – 00:37:45]** At the end of the day, it's a splat smushed into a USD file.

**[00:37:46 – 00:37:48]** We need to make a mesh so the robot can walk.

**[00:37:48 – 00:37:53]** And we are going to do that with a super cool algorithm that

**[00:37:53 – 00:37:56]** one of our FADB team members made.

**[00:37:56 – 00:37:58]** We're just going to do like stereo vision.

**[00:37:58 – 00:38:01]** We're going to render images from our Gaussian splat and

**[00:38:01 – 00:38:04]** use them to do stereo vision and get the depth that way.

**[00:38:04 – 00:38:07]** And then we're able to quickly and efficiently make a pretty

**[00:38:07 – 00:38:09]** highly accurate mesh.

**[00:38:09 – 00:38:11]** But we just spent all this work making these beautiful

**[00:38:11 – 00:38:12]** high fidelity splats.

**[00:38:12 – 00:38:15]** So in Isaac Sim, we're going to hide the mesh.

**[00:38:15 – 00:38:17]** The robot knows it's there. He can physically interact

**[00:38:17 – 00:38:18]** with that surface.

**[00:38:18 – 00:38:21]** And we're going to keep our splats for the high fidelity

**[00:38:21 – 00:38:24]** visualization part.

**[00:38:27 – 00:38:30]** Okay, so now this is the fun part where we're actually going to be

**[00:38:30 – 00:38:35]** able to play around with robots, so please start notebook number three.

**[00:38:35 – 00:38:38]** It's going to go through how to create the USD files,

**[00:38:38 – 00:38:40]** how to create the mesh, and just the little script

**[00:38:40 – 00:38:43]** that basically aligns things.

**[00:38:43 – 00:38:45]** And make sure that when we put them into Isaac Sim, it's

**[00:38:45 – 00:38:47]** going to be right-side up.

**[00:38:47 – 00:38:48]** Just make your life a little bit easier.

**[00:38:48 – 00:38:52]** And then you're going to actually launch the environment in

**[00:38:52 – 00:38:53]** which we're going to run Isaac Sim.

**[00:38:53 – 00:38:56]** So you have a little virtual desktop, and you can construct

**[00:38:56 – 00:38:59]** the scene yourself, and then you can eventually run Isaac

**[00:38:59 – 00:39:04]** Lab and move your robot around the scene in simulation.

**[00:39:08 – 00:39:13]** Okay, any questions on that part?

**[00:39:25 – 00:39:27]** An important note, just in case you don't catch it in the notebook,

**[00:39:27 – 00:39:31]** the instructions for the Isaac Lab, Isaac Sim portion are going to

**[00:39:31 – 00:39:36]** be back on this course page under robotic simulation in Isaac Sim and

**[00:39:36 – 00:39:38]** that's just going to go through all the steps for you to create your

**[00:39:38 – 00:39:40]** scene and then run the simulation.

**[00:39:40 – 00:39:44]** Please let us know if you have any issues at all.

**[00:39:46 – 00:39:49]** Okay, hopefully everyone has gotten to this point where

**[00:39:49 – 00:39:52]** you've got your little spot robot in Isaac Sim and you

**[00:39:52 – 00:39:54]** can start moving stuff around.

**[00:39:54 – 00:39:57]** We've still got time, so we are by no means kicking you out.

**[00:39:57 – 00:39:58]** Please keep working.

**[00:39:58 – 00:40:02]** But I just wanted to open the floor up to any questions.

**[00:40:02 – 00:40:05]** And I also want to make sure you guys have our Next Steps

**[00:40:05 – 00:40:10]** line so you can scan any QR codes that you might want.

**[00:40:10 – 00:40:15]** And there's also, on the course page, there's a Next Steps tab.

**[00:40:15 – 00:40:22]** So please, if you'd like to learn more, scan, go to those pages.

**[00:40:22 – 00:40:27]** We've got how to do co-map on your own data, so you can

**[00:40:27 – 00:40:30]** train your own Gaussian Splat, some information about the

**[00:40:30 – 00:40:32]** connection between Nurek and Isaac Sim, because you can really

**[00:40:32 – 00:40:34]** get into the nitty-gritty there.

**[00:40:34 – 00:40:38]** FEDV just released their 4.0 version, which has got things

**[00:40:38 – 00:40:41]** like Gaussian Splatting with MCMC.

**[00:40:41 – 00:40:44]** And some other improvements that you can check out and

**[00:40:44 – 00:40:47]** the PIP wheels have been made so you can PIP install it today and

**[00:40:47 – 00:40:53]** of course you can dive deeper into all of our Omniverse ecosystem.

**[00:40:56 – 00:40:58]** Thank you.

**[00:40:59 – 00:41:03]** If you guys have any questions, please let us know in this

**[00:41:03 – 00:41:10]** last five or ten minutes or so, we're happy to answer them.

**[00:41:12 – 00:41:16]** Yes, the paper has grabbed our attention.

**[00:41:20 – 00:41:21]** So, we are looking into it.

**[00:41:22 – 00:41:25]** FEDB is always open to, if you think there is something

**[00:41:25 – 00:41:29]** in their library, in their tooling, that would make your life

**[00:41:29 – 00:41:32]** better, is a great thing to have.

**[00:41:32 – 00:41:34]** They would love to know, so they can look into implementing it

**[00:41:34 – 00:41:39]** fast, and to scale across multiple GPUs, so it's open for everybody

**[00:41:39 – 00:41:41]** to use effectively, instead of in sort of like a research

**[00:41:41 – 00:41:46]** code mode, if that makes sense.

**[00:41:47 – 00:41:51]** I also wanted to say that I have swag here at the last Q&A session,

**[00:41:51 – 00:41:55]** so if anyone has questions, this group, everyone's been asking great

**[00:41:55 – 00:41:58]** questions, I usually have to like,

**[00:41:58 – 00:42:01]** you know, bribe everybody, but we do have some swag and I'm happy

**[00:42:01 – 00:42:06]** to give out the rest of these.

**[00:42:06 – 00:42:12]** So how, when do we see some of the generative stuff, right

**[00:42:12 – 00:42:15]** now Gaussian is helping all the generative stuff today.

**[00:42:16 – 00:42:19]** When do we start seeing some of your generative research

**[00:42:19 – 00:42:21]** coming back outside of New Rec?

**[00:42:21 – 00:42:24]** Like I think, I think Extractor is a really

**[00:42:24 – 00:42:26]** good example of that, right?

**[00:42:26 – 00:42:32]** Where it's, but when do we start seeing it integrated into FEDB?

**[00:42:35 – 00:42:39]** Okay, so on the segmentation side is probably what we're

**[00:42:39 – 00:42:41]** going to see at first.

**[00:42:41 – 00:42:45]** This first implementation was based on the GARField, but like I said,

**[00:42:45 – 00:42:50]** we have something called LangSplat coming, which is more on the...

**[00:42:50 – 00:42:54]** The generative AI side, and then you'll probably see some

**[00:42:54 – 00:42:58]** integrations coming to help clean up the splats, right?

**[00:42:58 – 00:43:02]** So, I don't know if you've been tracking DiffX 3D+, that

**[00:43:02 – 00:43:07]** is using the Cosmos diffusion backbone to help clean up

**[00:43:07 – 00:43:11]** splats during the training process.

**[00:43:12 – 00:43:19]** Defects3D+, that's kind of like the research side and

**[00:43:19 – 00:43:22]** as those kind of get proven out they migrate into FEDB

**[00:43:22 – 00:43:27]** as a productized implementation.

**[00:43:32 – 00:43:36]** As a research part, that was great, but I think there was

**[00:43:36 – 00:43:39]** vagueness on licensing for actually trying it out, anything

**[00:43:39 – 00:43:43]** beyond research, so who do we follow up with if you have

**[00:43:43 – 00:43:47]** any specific questions like that?

**[00:43:54 – 00:44:00]** In this lab, we produced a Gaussian splat out in the cloud somewhere.

**[00:44:00 – 00:44:07]** Is there guidelines or help to figure out what kind of machine

**[00:44:07 – 00:44:12]** is adequate for us to do that sort of thing, given the kind of out

**[00:44:12 – 00:44:14]** of memory issues we're seeing here?

**[00:44:14 – 00:44:21]** I mean, maybe like, you know, number of images, resolution,

**[00:44:21 – 00:44:23]** that kind of thing.

**[00:44:24 – 00:44:26]** Yeah, that's actually a good idea.

**[00:44:26 – 00:44:28]** We should probably have like a little guide with the, next

**[00:44:29 – 00:44:33]** to the, so Zoe also put, here we have, tip it this way,

**[00:44:34 – 00:44:40]** Zoe also put up a guide on the structure for motion portion.

**[00:44:40 – 00:44:43]** But yeah, we should probably look into putting kind of

**[00:44:43 – 00:44:47]** like general guidelines on how much VRAM is required for each step.

**[00:44:47 – 00:44:51]** I would say some of the challenges with the out of memory is

**[00:44:51 – 00:44:54]** Like you wouldn't probably run into it as much if you wrote your own

**[00:44:54 – 00:45:02]** code because the Jupyter Notebook kernels love holding on to VRAM.

**[00:45:13 – 00:45:16]** Yes, if you run this in like a Python script, you won't

**[00:45:16 – 00:45:18]** have as much issue.

**[00:45:18 – 00:45:22]** Some of it is just the kernel management on Jupyter.

**[00:45:22 – 00:45:25]** But it is it is a good idea because we we have demonstrated

**[00:45:25 – 00:45:28]** running this on really large images like 150 megapixel images and that

**[00:45:28 – 00:45:32]** kind of thing So we could probably provide a few recommendations

**[00:45:32 – 00:45:34]** on like the training process across those different types

**[00:45:34 – 00:45:40]** of image scales Yeah, we can get the information between a couple

**[00:45:40 – 00:45:43]** hundred images and training across four GPUs on like thousands of 50

**[00:45:43 – 00:45:48]** megapixel images It's a great idea

**[00:45:50 – 00:45:53]** So it kind of glosses over this.

**[00:45:53 – 00:46:00]** I'm looking at the code. But to get the environment we

**[00:46:00 – 00:46:04]** Gaussian splatted into Isaac Lab so that we can operate,

**[00:46:04 – 00:46:05]** it's like a custom script here.

**[00:46:05 – 00:46:08]** But like.

**[00:46:09 – 00:46:12]** What is the guidance there on like, okay, I've created

**[00:46:12 – 00:46:16]** my own, and now I want to run it in Isaac Lab as an environment.

**[00:46:16 – 00:46:18]** Is it just to copy the script?

**[00:46:18 – 00:46:23]** Or is it like, Isaac Lab Arena, I saw now you can

**[00:46:23 – 00:46:25]** have different environments.

**[00:46:25 – 00:46:28]** Is that an easier way to load it?

**[00:46:28 – 00:46:31]** I'm not too well-versed into the actual robotics part,

**[00:46:31 – 00:46:33]** but you're more than welcome to use this script.

**[00:46:33 – 00:46:37]** It's fairly on the simple side, I will say, because

**[00:46:37 – 00:46:39]** at the end of the day, this is more of a Gaussian Splatting

**[00:46:39 – 00:46:41]** Lab rather than an Isaac Lab.

**[00:46:41 – 00:46:44]** But it's a good jumping-off point to see, at the bare bones,

**[00:46:44 – 00:46:48]** how can I do this, basically, and then you can scale up and out.

**[00:46:49 – 00:46:54]** Also, the USD format is now more officially supporting

**[00:46:54 – 00:46:58]** splatting, so I would imagine we'll start to see more, it's becoming

**[00:46:59 – 00:47:03]** more of an actual standard, so there'll be more conversion scripts

**[00:47:03 – 00:47:06]** and other tooling coming out too.

**[00:47:11 – 00:47:14]** So thank you so much for having this training lab.

**[00:47:14 – 00:47:18]** I've been trying to get up to speed on Gaussian Splatting and listen

**[00:47:18 – 00:47:24]** to a podcast with Ru Long Yi a few months ago, and he was giving the

**[00:47:24 – 00:47:27]** update on Nerf Studio and G-Splat.

**[00:47:27 – 00:47:31]** So I don't remember, but are those still being supported?

**[00:47:31 – 00:47:32]** Are those being deprecated?

**[00:47:32 – 00:47:36]** And should we go to, I guess, UREC?

**[00:47:37 – 00:47:40]** Yeah, so I like to think of gSplat as constantly adding in

**[00:47:40 – 00:47:43]** the newest, latest research paper.

**[00:47:43 – 00:47:44]** It's constantly improving.

**[00:47:44 – 00:47:48]** FEDB is going to take what's useful from the research world and make

**[00:47:48 – 00:47:53]** it run fast across multiple GPUs at large scales in an easy-to-produce

**[00:47:53 – 00:47:58]** developer way, and NVIDIA is currently supporting both of those.

**[00:47:58 – 00:47:59]** Okay, thank you.

**[00:48:00 – 00:48:05]** Hi, I know Nurek has a mono and a stereo path in terms of capturing.

**[00:48:05 – 00:48:08]** Do we know at what point, I know the mono path uses

**[00:48:08 – 00:48:16]** like 3D gut, do we plan on having like FEDB involved in that

**[00:48:16 – 00:48:18]** or kind of revamping that flow?

**[00:48:18 – 00:48:24]** Yeah, that is actually, let me back away from the speaker, 3D gut,

**[00:48:24 – 00:48:27]** like those secondary ray effects are something that FEDB is working

**[00:48:27 – 00:48:30]** on implementing because they do seem to be something that people

**[00:48:30 – 00:48:36]** really want to make these Gaussian splats a lot more expressive.

**[00:48:36 – 00:48:38]** Well, thank you all for the great questions, and thank

**[00:48:38 – 00:48:40]** you all for coming.

**[00:48:40 – 00:48:42]** You should have access to the course for a while, and

**[00:48:42 – 00:48:44]** the slides are on there.

**[00:48:44 – 00:48:47]** There's a list of next steps that have a bunch of links

**[00:48:47 – 00:48:50]** for whatever direction you sort of wanna take this work.

**[00:48:50 – 00:48:52]** And thank you all for coming. It was great to have you here.