Controlling a Hydraulic System using Reinforcement Learning : Implementation and validation of a DQN-agent on a hydraulic Multi-Chamber cylinder system

Abstract: One of the largest energy losses in an excavator is the compensation loss. In a hydraulic load sensing system where one pump supplies multiple actuators, these compensation losses are inevitable. To minimize the compensation losses the use of a multi chamber cylinder can be used, which can control the load pressure by activate its chambers in different combinations and in turn minimize the compensation losses.  For this proposed architecture, the control of the multi chamber cylinder systems is not trivial. The possible states of the system, due to the number of combinations, makes conventional control, like a rule based strategy, unfeasible. Therefore, is the reinforcement learning a promising approach to find an optimal control.  A hydraulic system was modeled and validated against a physical one, as a base for the reinforcement learning to learn in simulation environment. A satisfactory model was achieved, accurately modeled the static behavior of the system but lacks some dynamics.  A Deep Q-Network agent was used which successfully managed to select optimal combinations for given loads when implemented in the physical test rig, even though the simulation model was not perfect.