Open-source Physical AI
At the end of 2024, I began investigating machine learning applications in robotics (Physical AI), with a focus on vision-language-action (VLA) models. In that context, I discovered LeRobot, Hugging Face’s open-source initiative, and began using the SO-100 robotic arm as an experimental platform.
LeRobot brings together pre-trained models, datasets, and tools to collect demonstrations, train control policies, and operate robotic arms. The platform makes it more accessible to turn human demonstrations into behaviors that a machine can reproduce, though the challenges of interacting with the physical world remain.
The experiments were conducted on an Avell laptop equipped with an NVIDIA RTX 3070 GPU. Data collection proved to be the most critical stage of the process. Each demonstration required consistent movements, well-positioned cameras, and precise task execution. Seemingly small variations in trajectory, framing, or object handling directly affect dataset quality and, consequently, the robot’s learned behavior.
One of the applications emerged from conversations with textile industry professionals: training the arm to fold clothes. Although it is an everyday task for people, it requires the system to handle a flexible object whose position and shape can vary. The experiment was a useful way to understand how a manual activity can be modeled, demonstrated, and refined with data.
Worldwide hackathon
The LeRobot Arm Hackathon was an international event organized by Hugging Face. We registered Fab Lab Joinville to host the Brazilian edition. The initiative brought Joinville’s technology community together for a weekend of hands-on exploration of the platform.
The video below captures the challenges, teams, and demonstrations that shaped the event:
Featured projects
The teams explored different applications for the robotic arm, from playful interaction to teleoperation and object manipulation. Below are some of the projects I enjoyed most, along with their final videos.
João Sem Braço
A playful interaction demonstration in which the robotic arm takes part in a game of UNO.
Sbórshchiki frúktov
An apple-harvesting proposal that explores how the arm can identify, reach, and handle fruit.
L.A.E.L.E
A force-feedback experiment that adds tactile feedback to operating the arm.
LAPESP
Control of the robotic arm directly from a browser, exploring a web interface for remote operation.
The team that organized the event:
