[Guide] Label Gaussian Splats with Segments.ai

2 min read -
Avatar photo
- May 7th, 2024 -

Gaussian splatting is a technique to render detailed 3D scenes. Instead of representing a scene as a set of meshes, a Gaussian splat represents the scene as a sort of point cloud, where each point is a 3D Gaussian. In 2023, Kerbl, Kopanas, et al. published a paper titled “3D Gaussian Splatting for Real-Time Radiance Field Rendering”. In it, they show how a 3D Gaussian splat can be trained from a set of images and how the resulting splat can be rendered in real-time.

We’re excited to announce that you can now upload, view, and label .splat files on Segments.ai. You can both annotate vector objects (cuboids, polygons, polylines, and keypoints) in splats, and create segmentation labels by annotating the different 3D Gaussians directly. In the rest of this blog post, we’ll show you exactly how you can label your Gaussian splats.

How many photos do you need for Gaussian splatting?

To create a Gaussian splat, you’ll need a set of around 20 to 200 images. Ideally, these images should be crisp, well-lit, and free from motion blur, bokeh, and image artifacts. The images can be in PNG or JPG format.

 

How to Label Gaussian Splats

1. Create a Gaussian splat

You can use a commercial service such as Polycam or Luma AI to create a gaussian splat from your images. Alternatively, you can also use the open-source project Nerfstudio to train Gaussian Splats. After you’ve created your splat, download it as a .ply file.

2. Convert the splat to the .splat format

Next, you’ll have to convert your .ply file to the .splat format (a file format designed specifically for Gaussian splats). If you only have a few scans, you can use SuperSplat, an online splat editor. If you want to process multiple scans programmatically, you can use the open-source point-cloud-tools library.

3. Upload the splat to Segments.ai

To upload the splat to Segments.ai, you first have to create a dataset. You can do this by pressing the “New dataset” button. To create a dataset for Gaussian splats, select “Pointcloud” as the data type and select the labeling task you want (cuboid, vector, or segmentation). In the pop-up, you can also find the instructions for creating datasets using the Python SDK.

Next, you can simply drag-and-drop your .splat files to the platform. Alternatively, you can also press the “Add samples” button, where you can also find the instructions for uploading data and adding samples using the Python SDK.

Before labeling the samples, you can also define the categories you want to label (and optionally add attributes for a certain category or for the whole sample). To do so, go to the dataset settings and select the “Labeling” tab. Here, you will find the category editor. Don’t forget to save the new categories when you’re done.

4. Label the gaussian splat

At this point, you’re ready to label the uploaded samples. You can start by pressing the “Start labeling” button or by opening a sample from the list. In the 3D interface, you will see your Gaussian splat being loaded dynamically, after which you can start labeling.

We won’t go over how to use the labeling tools on Segments.ai in detail here, but we’ll instead refer you to our documentation: https://docs.segments.ai/how-to-annotate/label-3d-point-clouds

5. Download the label

After you’ve labeled the samples, you can create a release. A release is a snapshot of your dataset containing all of the samples and labels in a JSON format. The format of the label depends on the task type, so have a look at our documentation for more details: https://docs.segments.ai/reference/label-types#id-3d-point-cloud

To create a release, open the dataset, go to the releases tab, and press “Create a new release”. After the release is created, you can simply press the download button to download it.

You can also get the JSON label of a single sample by using the Python SDK.

Demo datasets

You can play around and test the Gaussian splat interface on Segments.ai.
If you already have an account, you can just click these links.

Don’t have an account yet? Create a free account (no-strings-attached) first.

That’s it! You’ve now successfully created and labeled a Gaussian splat. In case you have any more questions, feel free to reach us at support@segments.ai or via the chat widget on the platform.