Reinforcement learning for neuron-connected robots

About the project

The goal of this project is to build a closed-loop that connects dissociate culture of living neurons with robots, and to realize goal-oriented interactions between living neurons and the real world. Although there is a long history of controlling robots connected to cultured neurons (i.e., neuron-connected robots), achieving goal-oriented behavior is still challenging due to the difficulty of rewarding neurons for suitable behavior. To address this challenge, we have developed a control system with a reward function by utilizing reinforcement learning (RL) by leveraging Bonsai, a Microsoft AI platform.

Living neurons as sensory/motor processors for robots

Several configurations are possible for neuron-connected robots. For example, the robots are driven based on neural activities, and an RL agent decides how to electrically stimulate the neurons. This configuration can be regarded as a motor processor scenario because the neurons can be regarded as the motor cortex to the robots. Another example is the case where the neurons were electrically stimulated based on the state of robots in the environment. In this case, the neural output is sent to an RL agent as an observable parameter. This configuration can be regarded as a sensory processor scenario because the neurons can be regarded as sensory cortex or sensory organ to the robots. Focusing on these scenarios, we have been developing robots, reward functions, and suitable mapping functions between neurons and robots.

Key technologies

Neural simulator for facilitating RL training. As reinforcement learning typically takes tens of thousands of iterations for training, the use of living neurons during training may cause cell damage. Therefore, we prepared a neuron simulator that mimics the I/O property of cultured neurons.

Learning with the plasticity of neurons. The time-variant I/O of cultured neurons causes a sim-to-real gap. For example, it is known that synaptic connectivity and even neuron positions change as the neurons mature in culture. For another example, electrical stimulation can cause neural fatigue and cell damage. To address this issue, we propose a method to keep the neural simulator “fresh” during the training based on the periodical recording of real neurons.

Living neurons connected to the cloud. In collaboration with Prof. Hirokazu Takahashi (opens in new tab) at The University of Tokyo, we have developed a system that connects cultured neurons living in a laboratory with a cloud-based RL system. This setup facilitates the training of an RL agent using a scalable computational platform and the development of ubiquitous services for controlling neuron-connected robots.