Robotics research has a reproducibility problem, partly due to the myriad of interacting components in robots. These components are typically complex, only partially observable, and trained using AI techniques that vary greatly in performance across different environments. In order to master some of the specific challenges of the autonomous driving area, researchers from ETH Zurich, the Toyota Technological Institute, Mila in Montreal and NuTonomy developed the so-called Decentralized Urban Collaborative Benchmarking Network (DuckieNet), a setup that was built with the open source Duckietown Platform. DuckieNet provides a framework for developing, testing and deploying perception and navigation algorithms. The researchers claim it is highly scalable but inexpensive to construct.
The Duckietown project, conceived by a 201
DuckieNet builds on Duckietown by adding special components to platform A. challenges server stores machine learning algorithms, benchmarks, and results, and computes leaderboards to submit jobs to a range of rating engines. The evaluation machines, which can be local or cloud-based, carry out autonomous driving simulations. Physical laboratories with DuckieNet installations conduct real-world experiments, and a localization network of “watchtowers” (inexpensive structures that use the same sensing and computation as the Duckiebots) tracks tags attached to the Duckiebots’ bodies.
DuckieNet is somewhat similar to AWS DeepRacer, Amazon’s service that provides developers with a cloud-based simulator for developing autonomous driving models and making them available in a model car. However, DuckieNet users can define benchmarks such as the mean positional deviation (lateral displacement of a Duckiebot from the center of a lane) and the mean misalignment (mean orientation in relation to the lane alignment) in Docker containers that are sent to the challenge server . (Algorithms can also be submitted and observed as Docker containers.) Apart from tasks such as resetting experiments and charging the duckiebots, the platform is also completely autonomous.
The researchers say one of the main uses of DuckieNet is in hosting research competitions. In fact, DuckieNet has been used at the AI Driving Olympics since the beginning of 2019, a biannual competition designed to measure the state of the art in autonomous driving of vehicles. DuckieNet generates visualizations of performance metrics and leaderboards while providing access to the underlying raw data, including open source baselines and documentation.
“We believe that more effort is needed for reproducible research for robotics, and that this requires us to consider evaluation as much as the algorithms themselves,” the researchers wrote in an article describing their work. “In this way, we can achieve reproducibility through design through research and development processes. Achieving this on a large scale will contribute to a more systematic evaluation of robotics research and thus increase development progress. “