From Argo AI to Bucket Robotics: Firsthand Lessons from Running Autonomous Robotics

I want to share a story that illustrates the challenges of managing complex systems, especially when those systems span different time zones, continents, and even natural phenomena. My co-founder Steph and I worked together at Argo AI, where we experienced firsthand the intricacies of operating self-driving vehicles across the globe.

In March 2022, I was on my third work trip to Munich as part of Argo AI’s Autonomy Integration team. Our mission: make an experimental Volkswagen Bus fully autonomous.

Argo AI, a Level 4 autonomous vehicle company based in Pittsburgh, partnered with Ford and later Volkswagen to bring autonomy to their platforms. Level 4 autonomy means integrating cameras, radar, LiDAR, and IMUs with powerful computing to build a system capable of driving itself within mapped areas. Our close relationship with the car manufacturers allowed us to integrate our heavy-duty computing with their vehicles without having to reverse-engineer CAN buses, which was a significant advantage.

The intercontinental nature of our work was challenging, and building robots capable of driving on public roads during a global pandemic added a new layer of complexity. We could simulate some elements, like traffic light placement, on a test track in the US, but we wouldn’t know for sure how things would work until we deployed the technology on the actual vehicles in Europe.

When we received our first vehicles from Volkswagen and set up a test track in Europe, we worked through the usual integration issues—ground loops, log ingest pipelines, and making sure we had all the necessary tools for inevitable hardware fixes. After that, we moved on to the “easy” part: solving autonomy on public roads. We operated the vehicles with two safety drivers—one behind the wheel and one in the passenger seat, monitoring the system and annotating vehicle behavior for later triage. The ride quality was rough at first, but we were operating autonomously for stretches! This progress allowed us to offer demo rides to our executive partners.

Operating an autonomous vehicle on public roads requires complex orchestration across many teams. You have to deploy and update software, maintain hardware, calibrate sensors, perceive, plan, and act on-road—all in a naturalistic way that fits the real world. Then, you ingest data, triage logs, analyze performance, write more software, and repeat the cycle. And you’re doing this on multiple vehicles, so a good issue-tracking system is essential. Running a fleet of robots is an absolute blast.

Because the operation was so complex, our team was distributed across the globe. We had LiDAR engineers in New Jersey, software engineers in California, vehicle experts in Detroit, and on-the-ground experts across Germany. As a field liaison, I traveled from operations center to operations center, ensuring successful expansion.

Trips to Munich were among my favorite experiences at Argo. Despite operating during the COVID-19 pandemic and living off Marriott meals, I got to experience autonomy on a new continent and build relationships with some of the best roboticists in the world. During one of these trips, I woke up to a few hundred missed Slack notifications—far more than the usual 50 messages overnight. What was unusual was that I had been added to every data labeling Slack channel, with hundreds of direct messages reporting the same thing: every camera on every vehicle in Germany was broken. Every. Single. Camera.

“OK, this is bad,” I thought. “How the heck is this possible? How can every camera be bad at the same time?!? “

While reading through Slack, I found that the issue was first identified by our internal map generation team. We were expanding our operational area in Munich, so we paid special attention to the maps we were building. However, every image from every camera from every car had an orange/salmon tone. Was it a bug in the demo software? A power problem with the vehicles? A logging drive corruption? Firmware issues during calibration?

The real culprit? A massive Saharan dust storm had turned the skies of Europe an eerie orange, causing our sensors to pick up unexpected data. Unaware of this natural phenomenon, the team initially assumed a technical malfunction.

We had operated all day without noticing an impact, and the morning after the logged data concerns, the skies were clear.

This incident highlights a critical point: while robots can perform tasks with precision, the complexity truly arises from human interaction with these systems. We often expect people to interact with robots in a robotic, error-free manner, but humans bring intuition and context that machines simply don’t possess.

As we refined our autonomous vehicles in Munich, we realized that our perception models needed to handle a wide array of environmental conditions. It wasn’t just about integrating hardware and software; it was about ensuring reliability in any circumstance.

This experience underscores a central lesson at Bucket Robotics: unpredictability is the norm in robotics. Whether it’s a Saharan dust storm turning the skies orange or subtle changes in lighting within a manufacturing facility, robots must be prepared for a wide range of conditions.

Perception in robotics is deeply complex. Modern advances in computer vision and edge computing allow rapid innovation, but in manufacturing, we still encounter “orange dust” issues that require human intuition to interpret unexpected scenarios. These issues come from dust, lighting changes, condensation, or other tricky little things in a factory.

As we look ahead to the next 5-10 years, the convergence of robotics, AI, and synthetic data generation will fundamentally reshape industries across the globe. At Bucket Robotics, we envision a future where automation isn’t just a tool for efficiency but a cornerstone of innovation, enabling industries to achieve levels of precision, quality, and adaptability that were previously unimaginable.

Synthetic data generation is the creation of artificial data that simulates real-world conditions. In our case, we start with CAD models of parts that our customers want to inspect and generate tens of thousands of scenes varying between perfect and defective parts. In a self-driving car environment, you’ve got an infinitely evolving world to account for. In manufacturing, you’re singularly focused on a particular component, and a defect can occur in infinite ways.

The trickiness of synthetic data for robots is that you’re only as good as the scenarios you can think up to simulate. For example, a perception engineer might not think that the autonomous vehicle will encounter a horse on the streets of Pittsburgh (true story) – so horses wouldn’t end up being simulated. Maintaining a feedback loop is critically important – we simulate the world, but also sample the real world to inform the simulations. So far, we haven’t encountered any horses in factories – but we’re ready.

For robotics, there are a few different classes of simulation you’ll run into:

You’ve reviewed logs from your robots and curated a list of interesting scenarios. These logs are used as scenario templates: On June 15th in Austin, Texas, the robot stopped for a red light. A crowd crossed the crosswalk, including someone wearing a bright green hat. The vehicle misclassified the hat as a green traffic light and attempted to move, resulting in a disengagement from autonomy.

Now that you’ve got your reference scenario and behavior defined (we moved when we shouldn’t have), you can feed the data back into your new code as you make changes to measure whether your robot performs better (the hat is correctly tracked as part of a pedestrian) or worse (the robot keeps thinking it’s a traffic light).

One of the hardest and most expensive things to simulate is accurate sensor data: the video game and film industries make simulating photorealistic camera data easier every year, but accurately simulating noisy data from a particular model of IMU or radar is still a research problem robotics companies don’t usually have time to tackle.

Behavior simulation offers a way around this: don’t simulate sensors; instead, simulate detections or tracks. This is a great way to characterize your robot stack’s prediction and planning parts – prediction, motion planning, and motion control. In behavior simulation, you create a highly structured world with highly structured rules for how you expect the robot’s subcomponent under test to behave. As you change your code, you build confidence in your subcomponent changes based on how you pass or fail scenarios.

Motion planning behavior testing could be like this: your vehicle approaches an all-way stop sign, and another vehicle is stopped in the oncoming lane. The oncoming actor has right-of-way but is creeping – stopping and starting. Correct behavior is: do not collide with this indecisive actor. Does your vehicle do the right thing, or does that stopping-and-starting behavior trick your vehicle into doing something weird, like running into the other actor?

The fun part of behavior simulation is that you want to add fuzzing to your simulations – we’re living in an ML world, so how do you know you’re not teaching to the test? Well, let’s create scenarios with the same overall shape as the original – two vehicles stopping for stop signs, then moving, with the goal of not colliding with the other actor – but let’s change up the timing this time. Maybe both vehicles start at the same time. Perhaps now the actor stops, then accelerates at full speed. This kind of fuzzing gives you confidence in your testing.

What everyone thinks autonomous vehicle simulation is like.

Here, the whole world is modeled – actors act with their own behaviors, hopes, dreams, aspirations, and limitations, and the vehicle is crawling around performing actions in a world that decays and is rebuilt like ours. This is … certainly one way to go about simulating vehicle interactions in the world to understand system behavior. But it is…mostly a path of madness. And if you’ve built a great simulator, you’ve also built an insanely good video game engine with applications far beyond robotics.

In Self-Driving (and it varies in other parts of the robot world), log simulation is the most common, followed by behavior simulation. Basically … no one does a full video game-style simulation – for a reason you wouldn’t fully expect: radar.

Full simulation of how radar handles different reflective surfaces encountered in the world has yet to be solved. We have a good handle on what radar returns from vehicles and pedestrians look like, but an accurate simulation of radar returns from buildings, trees, and all of the errata faced in the world has not been tackled yet.

Now, at Bucket Robotics, we do actually focus on full simulation: because we’re fully simulating:

• A single sensor (a camera)
• A relatively constrained environment

We generate a wide range of synthetic images from CAD models, each reflecting different conditions that the part might encounter in a real-world manufacturing environment. This includes variations in lighting, camera angles, part orientations, and even background settings.

For instance, we create images where a part is illuminated from different angles or under various intensities of light to simulate how it might appear under different factory lighting conditions. That might seem like it’s unnecessary – until you think about how big of a change in ambient lighting there is in Michigan where it’s dark at 3:00PM in winter and bright skies for days in summer.

We can also simulate different defects, such as scratches, dents, or deformations, to teach our models how to recognize these issues. You never really think about how many plastic parts there are in the world until one day…it’s like being able to see the Matrix for the first time. We’ve learned a lot about automotive parts – for injection-molded parts that go in the cabin of the vehicle (in places where people would go insane if they saw a weird discoloration or blemish,) you have your “A” side (customer-facing), and “B” side (the part that connects your plastic to the rest of your car.)

Faults on your customer-facing A side – are not the same as faults on the B side. For the customer-facing side, you want essentially perfection. For the B side – you want to make sure there are no structural issues that damage your alignment clips and pins to the point your part doesn’t attach. All faults are bad, but not all are created equally. So – with all that, we generate tens of thousands of high quality defects that bias regions defects are expected to happen – and where customers focus the most on quality. Compared to a human who received a few minutes of training, this is huge. Compared to a more traditional, “wait for a person to detect failed parts, take a picture of failed part, label where the failure is, repeat until I have statistical significance,” model-based approach to defect detection, also a huge saving.

In the world of manufacturing, the role of synthetic data will continue to expand, becoming a critical component in the development of increasingly sophisticated AI models. Today, synthetic data allows us to simulate a wide range of conditions, but in the near future, we expect these simulations to become even more nuanced and lifelike, incorporating elements such as real-time environmental changes, material properties, and even human interactions. This will enable robots to handle an even broader spectrum of tasks, from delicate assembly operations to real-time quality control in highly dynamic environments.

With all these types of sims, the part that really sucks is trying to get metrics out to determine whether your system has gone better or worse – once you’re running a couple of dozen sims or so, it’s just too much to review by hand. In our experience, most companies roll a metrics stack by hand (with varying levels of convenience and pain-in-the-ass factor) and spit out a link to Looker/Superset/Similar. It’s incredibly tempting to think, “our software North Star is we pass every single one of our billion simulations, and because that’s the number of simulations our systems team says we need, and we passed everything, we’re good forever, right?” Wrong. Regression tests seeded into your data set are crucial – you always want to know you’re failing some percentage of your scenarios, because you’ve got a certain percentage of the scenarios that are intentionally out of scope. The Kobayashi Maru of Star Trek is right, there are some tests you can’t pass…and knowing that you’re still failing the same number of tests gives you some confidence in your overall health.

Now, what does work really well when setting up metrics for simulations is making sure your dashboards are built for the end-user in mind. If your team generating new behavior scenarios is the same team who’s building the dashboard for looking at what new scenarios are being generated – you’ve missed the point and your autonomy teams will see divergent behaviors in simulation vs. in the real world.

Example: your behavior simulation team creates new behaviors for how the robot does path planning. They know that path planning is important, and that we have a bunch of path planning around the X and Y axes. “Aha!” Says the behavior simulation team, sensing an opportunity, “let’s create a bunch of scenarios for how we plan paths in the Z axis! A whole new dimension for the robot!” After working hard and spinning up these new scenarios and presenting them to the motion control team, it’s only then that the simulation team is reminded that…if we’re planning motion in the Z axis, our floor scrubbing robot is falling through the sky and we have bigger problems that are out of our control.

The above is an oversimplification of a situation you see all of the time in robotics. We all need to go out and touch grass, and when dealing with simulations for robots, need to remember that we aren’t building simulators for simulation sake, we’re building simulators for robots that are out in the real world and that the real world is weird and wonderful and unexpected.