Solving Multi-Sensor Labeling Challenges in Robotics and Automotive

How to build ground-truth data, specifically for autonomous systems in the robotics and automotive industries? Otto, CEO and co-founder of Segments.ai, spoke at Auto.ai Berlin ‘24 on this topic. Otto addressed the core problems faced by teams building autonomous systems and how to solve them, focusing on data annotation tools for multi-sensor use cases.

Parcel delivery robots on sidewalks, self-driving tractors, and self-flying drones generate data from cameras, lidar, radar, and other sensors. This data needs to be annotated accurately and consistently across all modalities.

This ground-truth data is essential for autonomous systems to function reliably. It forms the foundation for training, validating, and regulating machine learning models in many applications.

However, data annotation, especially for multi-sensor data, remains a challenge.

Variables like sensor types, class ontology and annotation specifications make complete automation impossible for these use cases.There’s no (foundational) model that can accommodate for the diverse set of guidelines requested by different teams. For example, should car side mirrors be included in a cuboid? Should objects in a crowd be labeled separately or jointly as a crowd instance? Should a tracked cuboid keep fixed dimensions? Such small decisions can have a big impact on the effectiveness of the automation pipeline.

Segments.ai takes a gradual, step-by-step approach to automation. We start with manual tools that are easy and efficient for annotators. We then introduce machine learning-powered assistive tools. Training our own domain-specific models can deliver additional speedups, yet models trained by customers will always provide the best automation results. Segments.ai focuses primarily on the latter, which is only successful when sufficient attention is given to dedicated QA autolabeling functionality.

Let’s dive into some core problems and how to solve them.

Some of our key customers require annotations across multiple sensors, including 3D point clouds and 2D camera data, and across multiple labeling modalities such as vector labeling and segmentation labeling.

The challenge is to ensure consistent labels across all sensors, with linked object IDs between various objects, such as parked cars, pedestrians, and road signs.

Typically, we face more complex use cases with multiple types of annotations required for each object:

• 3D cuboids on the point clouds
• 2D bounding boxes on camera images.
• Segmentation masks for precise pixel-level annotations in both 2D and 3D.

There are multiple approaches to solve this problem. One of the current step-by-step approaches is by solving the 3D cuboid problem first.

From 3D to 2D bounding boxes

We approach the problem by starting the annotations in 3D first. The team annotates objects using cuboids in the point clouds, defining the objects’ boundaries.

Next, we generate 2D bounding boxes from these 3D cuboids via a straightforward transformation method — projecting the 3D annotations onto the calibrated 2D space. This projection isn’t perfect, especially when dealing with imperfect calibration or imperfect (fisheye) distortion coefficients, so annotators manually correct these projections to ensure high accuracy. While these shape edits are still required, the projection step inherently ensures consistent tracking IDs and efficiency improvements.

From 3D cuboids to 3D segmentation

To kickstart 3D segmentation labeling, we can introduce a pre-filling step. An algorithm fills the cuboid’s interior points based on the annotated boundaries, greatly reducing manual effort. Human annotators then refine this pre-fill method to ensure precision, especially in complex or cluttered scenes. They either correct the pre-fills on each individual frame, or they use the merged point cloud mode to correct the labels on a single frame.

From 2D bounding boxes to 2D segmentation

To obtain 2D segmentations, we deploy a large-scale foundational segmentation model to predict the 2D segmentation masks based on the previously created bounding boxes. We’ve seen our dedicated model to empirically perform better than Meta’s SAM 2 and are investigating more advanced tracking mechanisms within the projection workflow.

This method saves time and ensures consistent annotation across the entire dataset, maintaining the same object IDs across 3D cuboids, 2D bounding boxes, and 2D segmentation masks.

From 3D segmentation to 2D segmentation

An alternative to obtaining 2D segmentation from 2D bounding boxes, might be to obtain them from 3D segmentation annotations instead.

However, this is less straightforward than one might expect. Projecting 3D segmentation masks onto the 2D plane yield very sparse 2D masks, which still need to be adjusted. This is a so-called in-painting problem which can be solved with diffusion models.

In this ECCV 2024 paper, we’ve shown that finetuned diffusion models using 3D segmentation masks as priors can successfully be used to obtain 2D segmentation autolabels, albeit currently only for sufficiently low-resolution images and sufficiently structured point clouds. The question remains how much time is then needed for QA purposes of these semi-autogenerated labels.

Teams that require labeled mapping data, often need to annotate lane lines, road boundaries, and drivable areas in both 2D images and 3D point clouds. The perception teams rely on these annotated maps to train models that understand road layouts, including relationships between e.g. vehicles, lane markings, and traffic signs.

Merging multi-frame data for high-resolution annotation

Instead of annotating each frame separately, you can merge all sequence frames on Segments.ai into a single high-resolution point cloud.

By combining data from multiple frames, annotators work with a denser and more complete representation of the scene, making it easier to annotate stationary objects like lane lines and traffic signs.

A lane line can also be annotated for the entire road sequence at once, rather than being limited to the point cloud in the current frame.

This technique also captures subtle details that might not be visible in individual frames due to sensor noise or low density in certain areas.

Creating relationships between annotated objects

A critical requirement in this mapping use case involves not just annotating individual objects but also establishing relationships between them.

For example, labeling a car stopped at a traffic light requires a connection between the car’s position, the lane line it occupies, and the traffic light it responds to. These relationships are key to training models that understand how different objects in the environment interact.

Projection back to 2D

Once the 3D point cloud is annotated, the platform projects the annotated lane lines and drivable areas onto the 2D camera images. This back-projection ensures perfect consistency between 2D and 3D modalities, allowing the client to train models incorporating data from both sensor types.

Automating parts of this process and allowing for manual correction where needed enables the mapping team to generate high-quality annotated maps efficiently.

Another challenge is to not just annotate objects but also assign detailed attributes to those objects over time. This use case involves annotating the state of vehicles, pedestrians, and other road users. Specifically, you need to know not only where a vehicle is, but also whether it is merging into a lane, exiting a highway, or stopping at a traffic light.

Dynamic object states

This first challenge is tackled by implementing a system for annotating time-based attributes. For example, a car that initially moves may begin merging into another lane over a series of frames.

Annotators mark the exact point where this change occurs and track it as the vehicle completes the maneuver. The system also allows annotating states like occlusion (whether part of an object is hidden from view) and truncation (whether part of the object is cut off by the frame boundaries).

Attribute consistency across time and modalities

We ensure consistency across time and multiple sensors. If a vehicle is marked as switching lanes in the point cloud data, the same action must be reflected in the 2D camera data and across the sequence of frames. This consistency is crucial for training reliable models that predict object behavior accurately over time.

Efficient correction of pre-annotated data

A customer can provide pre-annotated data with initial attribute predictions from their own models. The platform allows annotators to review these predictions and correct them where needed. This validation process is streamlined by providing annotators with tools to easily browse large datasets and quickly identify incorrect or missing annotations.

For large-scale projects like this, maintaining a balance between automated pre-annotation and human correction is vital to achieving accurate, high-quality annotations without excessive manual effort.