Writing labeling guidelines for autonomy and multi-sensor use cases: structure and best-practice template

High-quality perception systems of autonomous vehicles & robots often leverage a large corpus of labeled ground truth data. Generating these datasets requires a well-thought-through labeling specifications or guidelines document. Publicly available datasets often share the guideline documents, for example for 2D datasets such as Cityscapes or BDD100K or for 3D datasets such as KITTI and Waymo Open Dataset (see here for more examples).

However, it is often not straightforward to just copy the labeling guidelines from any publicly available dataset for a variety of reasons:

• Difference in sensors: there might be different type of hardware deployed on the vehicle or robot such as different types of LiDAR sensors with different field-of-view (FoV) and detection range or different cameras with different resolution.
• Difference in collected data: there might be differences in the collection time of day or in the geographical area.
• Difference in labeling objective: some datasets are built for model training purposes whereas others are built for validation or mapping purposes. Each different objective requires different labeling guidelines.
• Difference in annotations required: you might be interested in labeling specific type of 3D polylines for different types of category.
• Difference in models: you might be leveraging one of the publicly available models, or customizing your own model architecture. The choice will impact the required ground truth dataset and therefore the labeling guidelines.

To build your own guideline document, you could finetune the guidelines of one of the publicly available datasets. Alternatively, you could build your own from scratch leveraging some best practices.

In this article, I will focus on explaining the structure of a best-practice labeling guideline, providing an exhaustive overview of questions to ask yourself for multi-sensor labeling purposes and sharing an extensive labeling template to build your own labeling guidelines. Although there is some overlap as well with topics such as quality control procedures, feedback & communication mechanisms and training resources, I consider these out of scope of this article.

Let’s structure the article in three main sections:

1. Categories & attributes – What do you want to label?
2. Rules – How do you want to label it?
3. Additional information – What should we know as well?

BONUS: in the shared template, we provide default assumptions and answers based on the best available public datasets.

This is an obvious part of any labeling specification document.

The categories define what types of objects or features need to be labeled in your dataset. Common categories for on-road purposes include vehicles, pedestrians and traffic signs.

For each category, include the following items:

• A clear definition
• A set of descriptive & visual examples
• A set of descriptive & visual counterexamples, indicating what should not be labeled within the mentioned category. Include an explanation and, if applicable, the correct category.
• The sensor(s) it needs to be labeled on. E.g. some objects might only need to be labeled on specific sensors.
• Whether or not it needs to be labeled with individual instances. If so, whether or not individual objects that are part of a crowd should be labeled jointly or individually.

Attributes provide additional information for each object. Category-agnostic examples could include the following:

• Occlusion: the level of occlusion (none, partially, full – or a certain percentage)
• Truncation: whether or not an object is truncated by the image border or sensor field-of-view
• Visibility: for objects that are not occluded, a certain level of visibility (clear, blurry, foggy

Object-level attributes can also be dependent on the type of category of the object, e.g. pedestrians could hold an attribute to indicate whether or not they are standing, walking or running while traffic lights could hold an attribute to indicate the state of the traffic light (green, orange, red) or the type of traffic light.

The most common answer type for an attribute is a predefined list of answers for which just one or multiple can be selected. Alternative types are text, number or boolean.

An additional type of object-level attribute is a linking or relationship attribute. This allows to provide additional information on the relationship between two or more objects. Such a relationship can then entail a set of state attributes by itself as well such as the type of relationship between two objects.

These attributes allow to provide additional information about the scene. These could be for training purposes (e.g. type of weather) but could also be used to indicate data quality (quality of calibration) or general scene information.

Labeling rules or conventions are additional guidelines that are generally not included in the definition of categories and attributes. They generally are answers to questions that arise throughout the labeling project.

One way to illustrate this is to refer to CAPTCHA challenge-response tests asking to point out specific objects out of a 3×3 example grid. One example is to point out thumbnails with sidewalks – you might have received thumbnails with truncated or blurry sidewalks in the past, leaving you wondered if you should select the thumbnail or not.

When I receive guideline documents from prospects and customers, it typically includes the part about categories and attributes, but often lacks this section entirely. Nevertheless, clear rules ensure consistency and accuracy across the dataset. Especially for more advanced sensor types and labeling projects, the number of rules that needs to be covered grows exponentially.

One way to uncover questions is to process the guidelines of similar public datasets and extract the choices that were made. You can also task the engineering team to label some samples themselves and to raise questions whenever there’s doubt. Alternatively you can start from existing templates, such as the one provided below.

One could structure the type of rules in a couple of different topics, ranging from more general rules to sensor-specific, annotation-specific and data-specific rules. Let’s go through each of these and list some examples. Of course, these rules and questions can overlap – that’s not an issue, as long as the answers are consistent.

These are some general rules that apply to each labeling project.

• How precise/tight should annotations be? For 2D segmentation masks, what’s the acceptable number of pixels for border alignment? For 3D cuboids, how much centimeters is allowed between the cuboid edge and the closest object point?
• What should be done with unclear objects/areas? Should they be left unlabeled, or should they be labeled with an additional ‘visibility’ or ‘unclear’ attribute?
• Which parts/areas do not require labeling? Should only the driveable space of the vehicle or robot be labeled, or also other areas such as parking lots? What should be done with less illuminated areas or areas further away?
• Can labels overlap?
• How should one cope with groups of individual instances that are difficult to distinguish from each other?

Rules typically differ between the types of sensors and even characteristics of sensors. For example, for 3D data, following questions are typically addressed upfront:

• What is the horizon of labeling? Should the horizon be different ahead of the vehicle or robot vs. behind the vehicle or robot?
• What should be done with 3D occlusions? If objects are clearly distinguishable in provided auxiliary camera data but the 3D point cloud only contains points from the front of the object, should only the front part of the object be labeled or should there be a best guess made about the true dimension of the object?
• How should be dealt with sensor artifacts?

For 2D data, these could be some sensor-specific questions to tackle:

• What should be the minimum object dimension of objects in the back of the scene?
• How to handle motion blur?
• Are there specific rules with respect to camera distortion?

Of course, the chosen type of annotations also impact how data needs to be labeled.

For example, the majority of point cloud labeling projects on Segments.ai encompass 3D cuboid labeling. Cuboid-specific rules can be structured in a couple of different topics:

• Position: Can there be some overlap between cuboids and the ground plane?
• Rotation & heading: Should all three rotational degrees of freedom be labeled including yaw, pitch & roll or should only the yaw be labeled? Should the main direction/heading of an object point towards the main direction of movement or rather indicate the front face of the object? What should be the yaw direction/heading of a faceless object such as a cone?
• Dimensions: Is there a minimum size of a cuboid? Should cuboids be labeled with default dimensions based on their category and, if so, when can there be deviated from the default dimensions and what should the dimensions of a cuboid be based on? Can the dimensions of a cuboid change throughout a sequence? And how tight should a cuboid be?
• Occlusion: Should an object be labeled if it is only visible on 3D sensors and not on 2D sensors? Vice versa, how should an object be labeled if it is not visible on 3D sensors and only on 2D sensors? Should an object be labeled with the same ID if the object re-enters a scene or becomes unoccluded again?
• Extremities: Should fixed extremities or protruding parts be included in or excluded from the cuboid and, if excluded, should they be labeled with separate cuboids? What about variable or articulating extremities?
• Issues/exceptions: Can cuboids overlap? What should be done with unclear cuboid objects? In case of bad calibration, how should the cuboid be fitted – to the 3D point cloud or rather to the most confident reference image? And what should happen in case of bad ego poses and ego pose-related data drift?

Other types of annotations might address other questions, including

• For 2D bitmap segmentation labeling: How smooth do object edges need to be? Can (bitmap) labels overlap? Should see-through parts (e.g. between fences or wheel spokes) be labeled?
• For 3D polygon & polyline labeling: What is the minimum & maximum number of nodes that should be used to represent polygons & polylines? Can nodes be added/removed between frames? Are polygons or polylines directional? How can interpolation be leveraged for improved labeling throughput?

These rules cater to the specifics of the data being labeled. Some examples include

• How should be dealt with dust & reflections?
• How should be dealt with ego pose-related data drift?
• How should be dealt with weather conditions?

Following items should ideally also be included in your specifications document.

You might want to provide an overview of the sensors deployed with the related information. Try being exhaustive with providing the following information per sensor:

• Which type of sensor is it?
• How many of these type of sensors and how/where are they placed/positioned?
• What’s the file format of the sensor data?
• In case of point cloud data, is it in local or world reference frame? In case of local reference system, are the ego poses available?
• What is the sampling rate?
• Are intrinsics and extrinsics available?
• What is the coordinate system for the camera rotation?
• Are the distortion coefficients known?

Although it’s best to have a first 1.0 version ready before the start of any labeling project, large-scale labeling projects often deal with required changes in the specification document.

It is recommended to keep track of both minor and major changes by including the date of specification update, the date of implementation in the labeling workflow, the specification release version and the person responsible for the change.

A number of platforms, including Segments.ai, allow to ask & answers questions and address issues on the platform itself. Nevertheless, it’s beneficial to list the major questions in the guideline document itself and the corresponding edge cases.

It’s helpful to include the date, the person who raised the question, any assumptions, reference images and links.

We’ve compiled the above in an easy-to-use template.