@factory project launched

1. Target of the “@factory” project: Creating a Research and Development Platform for the Practical Application of Physical AI

@factory is an open-source project aimed at developing a platform for applying general-purpose physical AI technology to specific application domains (factories and manufacturing) and examining its effectiveness. Physical AI technology, born as a result of research, is now finally becoming ready to be put to practical use in society. To achieve this, it is not enough to simply develop individual algorithms; it is necessary to apply them to social issues and verify their usefulness.

@factory, as a platform, aims to enable the rapid evaluation of the effectiveness of physical AI by applying it to real-world problems. To achieve this, it takes into consideration the R&D workflow. Rather than individually defining virtual factories and robots on NVIDIA Isaac Sim, it provides an integrated environment with a plug-in structure that enables factory operation and allows virtual factories and robots to be replaced by other implementations with minimal effort. In other words, @factory is an integrated simulation execution environment that streamlines research and development for applying physical AI to specific real-world problems.

2.Code relase of this time

We have released a complete @factory sample system, consisting of three repositories on GitHub.

Generally, the repositories that make up @factory consist of the core “at_factory” and various virtual factory and robot definitions that are created and plugged into at_factory. This time, we are releasing the at_factory repository, the “at_factory_world1” repository that defines sample virtual factories and machine tools, and the “melon_ros2” repository that defines the mobile arm robot Melon.

By launching these, a factory where AI robots can work will be created in a virtual space using NVIDIA Omniverse/Isaac. This robot can be controlled externally via ROS communication.

This makes it possible to build a physical AI based system for controlling AI robots working in factories, without using an actual factory. This environment allows us to clarify the functions that physical AI should achieve and the evaluation criteria, creating an extremely effective foundation for investigating physical AI.

Conversely, it also makes it possible to specifically visualize what AI robots will look like in a factory, allowing both developers and users to properly evaluate robots utilization and effectiveness. For example, VR goggles can be used to inspect the factory. It also makes it possible to develop robot control software in a virtual environment without using an actual factory. This will be an important tool for utilizing AI robots in the manufacturing industry in the future.

So far, we have visited actual factories and built a virtual factory that closely resembles the one actually in operation. We have also created a model of a CNC lathe that can actually load parts.

We will continue to develop the robots that work in this virtual factory and their control mechanisms. For example, we will use autonomous dual-arm robots to build a factory that can operate fully automatically, from the receipt of parts to the shipment of products.

3.The heart of @factory: a pluggable “virtual factory” platform

The core of @factory system is the at-factory-base mechanism, provided from the at_factory repository and executed in Docker format, which serves as a platform for integrating and running virtual factories on NVIDIA Isaac Sim.

This common framework has a plug-in structure that uses a simple programming mechanism on Isaac Sim, allowing developers to build a simulation environment simply by “plugging in” the necessary elements.

Key elements plugged in:

ElementsOverview
USD definitions for factories and machine toolsUSD definition of small factories and machine tools based on actual surveys.
USD definition for worker robotUSD definition of the robots working there, along with physical optimization information such as friction factors.
Sample code for controlling a robotSample applications and control logic that serve as a starting point for developing AI algorithms.

These definitions (assets) are independent of the common framework, and new and improved versions will be released separately over time, ensuring that users can always use the latest and most diverse environment.

4. at-factory-base docker overview

The at_factory repository contains all the files necessary to create at-factory-base, a Docker repository with the functionality to run virtual factory simulations using Isaac Sim. To address initial concerns Isaac Sim developers have, such as where to place files and how to deploy robots, at-factory-base provides a concrete framework for the infrastructure required for development as follows:

  • Unified environment with Docker: The Isaac Sim + ROS 2 execution environment is provided using Docker, ensuring reproducibility of environment setup,
  • Clarified file placement: A clear directory structure is provided for the placement of persistent data (isaac-sim/) such as simulation scripts, logs, and data, as well as plugins (plugin/),
  • Automatic asset installation mechanism finstall: Robot plugins created and published by developers externally are automatically placed in the appropriate file path, facilitating their use within the Docker environment.

By using the at_factory system, which is built on the foundation of at-factory-base, developers can immediately take on the essential challenge of “realizing complex factory applications” rather than “building infrastructure.”

5.Factory definition sample

The virtual factory definition included in this release has been designed to simulate factory operations under as realistic conditions as possible. It reflects the characteristics of actual factories that were visited multiple times.

A CNC lathe has been implemented as a processing machine operated in the factory. The chuck structure has been accurately reproduced, allowing workpiece loading and unloading to be performed.

Development Team:

Nguyen Thanh An:
 design and development of overall architecture
Wataru Yoshimura:
 factory, machine tool, and robot designer
Satoya Sugimoto:
 robot developer
Kazuya Tago:
 project management