Musical Abstractions for Multi-Robot Control


Multi-robot implementation of the high-level control based on musical abstractions. In the image, three followers depict a major chord formation around the leader.

How one single operator should influence the performance of a large group of robots is not a simple question in that it depends on various factors, including the size and composition of the team as well as the objective task to be performed. This work introduces music theory as a new approach for multi-robot control that gives the human operator the power of specifying goal locations while controlling the spacial arrangement of the team.

In particular, we are interested in applying harmony rules to the user’s control input provided through a piano in order to modify the control parameters of a swarm. The aim is to abstract the harmonizing process: instead of generating sound combinations — namely chords — that serve as an accompaniment of a melody line, we apply harmony rules to an objective path (the melody) to obtain a geometry to be displayed by the group (the chord).



Transformation of the operator’s input into control commands for the robot team. The melody played by the human user on the keyboard specifies the goal location for the robot leader and serves as an input for generating the formation geometries.

A leader-follower strategy has been chosen to illustrate the musical-based control. The leader is controlled by the piano input, which is directly mapped to the target locations to be visited by it. The followers must adopt a formation geometry according to the characteristics of the chord that harmonizes the melody tone. In general, within a diatonic scale, each tone could be potentially harmonized using three different chords, that is, the three triads containing that chord. This leads to different chord types associated to each note and, therefore, different geometries. However, harmony rules restrict the choice of a chord for a melody tone, depending on the harmonic progression it generates..


C major scale with the triads (chords) formed over each of the notes. The roman numerals designate the degrees of the scale, while the symbols in the second line indicate the type of the chord associated to each degree of the scale.

The harmonization process used in this project, where we create the chords on top of the bass line, also relates to the leader-follower paradigm. The melody given by the human operator and, thus, the leader robot are identified with the bass line of the composition. The accompaniment chords, subordinate to the melody, control the movement of the followers. Given this hierarchy, we have encoded the harmony rules that generate the followers’ chords as a finite state machine where the states correspond to each one of the chord’s note and the transitions are triggered by the changes in scale degree commanded by the leader, that is, the input melody.


Follower’s automaton for the C Major scale created using voice rules for triads in root position. Transitions depicted in maroon indicate a dependency with previous transitions, while blue transitions are independent of previous melody notes.



  • María Santos
  • Magnus Egerstedt

Comments are closed.