Robot Opera

I have been working on a project that involves robots and performance since 2017.

Chris Kiefer, Tim Hopkins, and my colleague Ed Hughes, who's professor of composition in music, we got together and we made two short operas with two very small robots that Ron Chrisley had in his office.

It was a sort of an exploration of what a robot is on stage. Can a robot perform? What would a robot voice be like if we weren't always trying to make them sound like humans?

A couple years later, we did the same thing again, but this time with a Pepper robot, which is a larger robot. Again, we borrowed it from the informatics department.

Hello. What's your name? I want your job. My name's Laurie. I'm an opera singer.
You can't have my job.
No.
I will. Anything you can do, I can do better. I can do anything better than you.
No, you can't.
You don't even know where the audience is.
It's here.
I can do anything better than you. Yes, I can. No, you can't. Yes, I can. No, you can't. Yes, I can. No, you can't. Yes, I can. No, you can't.

We had Laurie Lichtenberg and her collaborator, Anton Lukoszevieze, a wonderful cellist. So a singer and a cellist, a robot, and I made a sort of robot cello with the help of Sam Bilbow, who's a wonderful PhD student at Sussex.

We were exploring robot and human interaction through theatrical performance and singing.

In 2021, we actually hired a robot from Engineered Arts, who are an amazing company in Cornwall, who make very realistic-looking robots with silicone skin and human hair. And so we did another project, again involving Anton and Laurie as human musicians, and exploring this kind of slightly uncanny realm between human performance and robot performance, but the robot looking very human on this occasion.

And using extended vocal techniques and electronic music and electronic manipulations of human voice. So we're really exploring a hybrid territory then.

Not you.
Not you.
Minnie.
Circle.
Silver.
Mirror mirror.
Not still.
Near.
But not still.
Then far.
Still not.
Not still.
Still not.
Still not.
Here.
Present.
Incorrect.

We had this wonderful meeting and collaboration with Elizabeth Jochum. So we actually invited her onto the panel of Robot Opera Mark 3 in 2021, and she gave a wonderful response to our work. She managed to get some funding from her own university to invite us to Aalborg, where she teaches in Denmark, and we spent three days throwing things together with a team of amazing music technologists, a roboticist, a couple of performers.

And that made us think that actually we do have the experience and the desire to make our own robots, however DIY and rough they might be, because then we could really put into them what we need and what we want aesthetically. So we wouldn't be dealing with the aesthetics of a pre-made robot.

We got some internal money together to hire Dexter Shepherd to build us a robot along the lines of the things that we'd explored in Aalborg, in Denmark. And he had about, oh, not long at all, maybe two weeks to make this robot. He then handed it over to our music technology PhD students, Max Morgan and Steve Simons, to work on the vocal side and the movement side. And then I got Elizabeth over from Denmark and Tim Hopkins on board and Chris playing his amazing feedback cello, and we all got in this room on Monday, and here it is Wednesday. So we had basically two and a half days to pull something together, explore what the robot could do, make something, and do a performance.

And so that's what we just did. Keep going left and turn round and round. In theory.

I'm Dexter. I'm a PhD student in my second year at the University of Sussex. So I got involved in this project through my supervisor, Phil, who used to be a musician and has dabbled in a lot of music projects, and mentioned that there was this robot opera going on, and they needed somebody to build a robot. And I very much enjoy working with hardware and the creativity of just designing something and building it and get it working. So this is very up my street.

Yeah. So when we press play, it will send the sequence as we know it.
Oops. Backward.
Stop. There's left and then there's right.
It's got left and right mixed up in the software. But only—

Performing is nerve-wracking, but when you've got a robot performing, it's even harder because you've got hardware that can go wrong, you've got software that can go wrong. So in a way it's very scary, but also, on the other side, that this is what robots should be doing.

And a lot of AI you see it's more just AI has solved this, AI has made that. But you don't really see AI in robots, which has always been sold to us in sci-fi as that is what AI is. And it's very nice to see the robots and also see that they're friendly as well.

So we've got this guy who is friendly looking, especially when he's got his glowing eyes. And the public, we don't want them fearing robots and seeing "Terminator" and all these very scary things. And even hearing about robotics and what it's being used for in the world.

With this robot itself, the eyes are very expressive. So when I build a robot, and I've got another robot back there with some eyes. Eyes are a big expressive feature for me.

But the behavior itself, so this robot is taking an input and performing. So the real personality of the robot will come out when that side is implemented. When I view the servos, it's just moving. But then actually using that as an input in a way, and an instrument itself, I think will be really interesting to see what sort of sounds get out and whether as well there's patterns in the sound and coherent sounds that are coming from the behavior of the robot itself. I think it will be particularly interesting.

One discussion, for example, around this has been about our robot maker, Dexter's, perceptive understanding about the humanizing function of eyes on a structure like this. How the representation of eyes reminds us of our eyes, and how this can have a humanizing effect.

And in the end, you'll notice that on the robot in its performance state, we moved these structures to a different place on the body. This was a conscious move towards announcing the otherness of the object slightly more. So in a way, going against the idea of humanizing it.

We have stepped back from working with robots that have been pre-designed and built, which bring their own set of decisions about how they look, how they sound, what they can do, and a set of assumptions behind those decisions which have led to what they look like, what they can do, et cetera.

We're interested in interrogating those assumptions a little bit and really stripping things back to ask some slightly more primal questions about the nature of robotics and artificial intelligence.

I'm often thinking about what sound looks like.

In this particular project, my role has been as co-lead with Evelyn Ficarra and with Elizabeth Jochum, who is joining us from Denmark. So we're all kind of leading it in more or less equal ways. My role has been to co-host three days of intensive work feeding in different disciplines. One discipline really led by Elizabeth, which has been puppetry, and she's been working in the field of allowing puppetry to inform and consider robot design for many years. Evelyn, who is bringing acoustic artistry, music making, and so on to this discipline. And I suppose I'm thinking about staging and theater and dramaturgy and how these things knit.

I asked everyone to bring an object that related to their research in some way as a way of introducing themselves. And I also asked them to think about sound in relation to their work and sound in relation to the object that they brought with them. I told them that I wanted them to make a sound with the object. We wanted the object to speak with its own kind of voice.

That morning, we had this great experience of hearing them talk about their work, then make a sound with the object.

I brought a piece of piano, one of those pieces, one of the smaller pieces there. Elizabeth used her scarf as an object. And had a relationship to the world and a relationship to me.

I love how we can connect this sort of materiality with the ideas that they were talking about. Bill Bigg brought an amazing light sculpture which he had made himself, which was fantastic. Cécile brought this beautiful round toy called a fidget, which made a lovely noise when it was spinning round. Steve Symons brought a long stick, which is a stick that he uses as a kind of actuator in his joint instrumental music making. You have one person at either end of the stick, and they have to move it, and that creates sound. So we had a wealth of objects in the room.

When we think about what it means to create a machine that's lively or lifelike or feels that it's autonomous and kind of performing itself, a lot of times we think, "Oh, we have to sit down at the computer and write code," or we have to put sensors on it, and we have to start thinking through in very technical ways how we can force that machine to come to life.

A lot of what we're doing with robot performance is really puppetry in a way. Robots don't have much or any autonomy. They have some inbuilt behaviors sometimes. But I wanted people to really get in there and experience what it would be like to try to think about an object that would move and how it would interact with other objects.

I asked Elizabeth to prepare a puppetry workshop for us because there's a lot of connections between puppetry and robotics.

She ran this wonderful workshop where we all made our own individual puppets and made sounds with them and had them interacting.

We can learn from the basic principles of puppetry, so very simple approaches to the act of animation and bringing objects to life.

So take your paper and crumple it up. And drop it on the floor. And watch it move.

It surprises people that that actually boils down to a couple of really simple components.

We explored with just a piece of paper how everybody, regardless of their background, regardless of their experience, through a couple of approaches of thinking about focus, thinking about breath, thinking about the weight of an object, and then thinking about the thought that that object has and how that object relates or desires or wants to move in the world. And with these basic principles, we can bring any object to life. We started with a piece of crumpled paper. It could be a chair. It could be a pencil.

We were all given piles of newspaper and some cardboard, and we were given the instruction to, as a group, make a robot together that was larger scale, and then we had to act out a scenario with the robot.

What kind of material are we trying to animate? And what are the basic signs that we need or basic kind of motions that we need in order to communicate that to an audience and build up from there?

What really struck me about that experience was hearing Elizabeth talk about how it's the attention that the puppeteer is paying to the materials, and the way it's moving the materials, and the way it has the puppet interact with the audience, and the ideas around emotion and attention and thought that go into puppeteering, which made me think about my own practice with sound objects because I love, of course, recording sounds and working with sounds.

What is the bare minimum for sensors that a machine would need to be able to hear a sound that another performer is making and reflect that sound back in some way?

We knew we wanted the robot to move, and we knew we wanted it to be able to move in the space, so that's how it got its base. We thought it would be nice if it could gesture, but we also knew we needed the robot to be able to listen and able to make a sound on its own, but a sound that was based on what it heard. So in that sense, a true improviser, the way a lot of vocal improvisers work and musical improvisers work.

They asked me to make a broadcast system from my laptop over to the robot and broadcast audio and to try and get audio back and process it through a number of AI models which tie into the movement that Max has been working with for the robot. I built the speakers, really, and the voice that comes in from the movement.

So the movement as the robot moves between poses, it moves through a 3D world, but also it moves through an audio scape at the same time. I feel that gives its embodiedness this connection between the movement and its voice and the sound that it's making.

How the robot was going to be used was kind of an unknown until we got here and started thinking about what we wanted to do.

My research field is music and machine learning, but actually for this project, I've just been helping use the robot. I've been writing some software to control the robot and try and sort of predict what his movements are going to be like and sort of make it as flexible as possible.

I think there was a fair amount of sort of firefighting at the beginning because we found, not being a robot expert myself, the technicalities of how to fix things going wrong, servos exploding, things like that, it's not my usual work. So there's just a certain amount of trying to get to grips with what may or may not happen. We ended up simplifying the robot, so his arms became simpler to make them a bit lighter, less prone to blowing up. I guess that affected his possible movement.

We also changed some of the limbs, and it resulted in it being less sort of jerky. So it had this issue where just as it was standing still, it was constantly trying to sort of remain in one place, which they really liked. I know Evelyn and Tim were really keen on that, but I suppose from the robot's maker's point of view, that wasn't an ideal situation. It led to servos exploding. So we simplified things.

So we removed a bit of the jerkiness and the weird inflections of movement. But there's still enough of it for him to sort of seem to have a personality. There's definitely a kind of fragileness, I think, to the robot, not just in the way that it stops working, but it also has been known to pull out its own wires and twist its limbs into positions and get stuck. So you do feel quite caring, protective over the robot, yeah, especially because I'm meant to be its carer, I guess, operator.

Got some loose connections, so I don't know if it's actually working right now.

Well, I'm really interested to see how we use machine learning with robots to give them some sort of interesting presence and how we can interact with robots that have got some sort of AI in their mappings.

In many ways, it's a very similar problem to designing musical instruments in that how do you design mappings between a human and a musical system, whether that's a cello or if it's a musical robot in some way.

Will Dexter have a bit of a life of its own? That's a thing that comes over into these sorts of instruments as well, that feedback instruments have a kind of sense of agency about them. They have their own kind of autonomy, and you don't really know what they're always going to do. And maybe it's quite interesting if Dexter can have some system to do that.

So in the past workshop, we experimented with some feedback systems which try to give the robot some sort of sense of its own emotion and autonomy and something that we couldn't really predict what's happening. So I think feedback's quite a nice technique for doing that sorts of things. Anything that has its own autonomy, it's got feedback in some way. It's just a matter of trying to compose the feedback and trying to find out how that's going to work.

With AI systems, what you're doing is trying to maybe put something that you can get a bit of control over into the middle of this feedback system. So the robot is projecting sound into the world, it's projecting movement, but it's also taking in something through, I think we might be recording a sound of its own motors, but also through microphones and sounds and other sensors, and then going back possibly through some sort of AI model in the middle, and then coming out in the world and back again. So we've got this loop, and I think we can probably use AI to try and do something that feels serendipitous. I think AI's really good at that, at making things that seem sort of surprising and unpredictable, but in a bit of a controllable way, not completely random or like this thing, sometimes unplayable.

It's not making noise. Then I can activate that speaker. So it'll be as if it's somehow learned how to make sound.
Okay. Which speaker do you mean?
In the paper.
Oh, yeah. Okay.

The sounds come from an AI system called a RAVE model, a real-time audio variable encoder that is usually used to transpose one sound to another. So you train this AI model on a body of sound, then you feed one sound into it, and it transposes that into the sounds that you've taught the model.

So we tried several sounds when Evelyn came to visit, and the one which resonated most seems to be a voice, because it ties in with the opera singer and everything like that. So we've got a voice model that is being controlled by the movements of the arms.

The lesser things move up as up, then you can shake.

I think that it's like walking into a workshop, a very kind of busy Heath Robinson-esque workshop, and I really like the combination of cutting-edge tech with all these kind of bits of paper and sticky back plastic, which reminds me of Blue Peter, for those of a certain generation, where it's all kind of things stuck together and, "Well, let's see if this works," and just plug it in and hope it doesn't explode. I really like that atmosphere. It's really good fun.

We've just done an exercise of imitating the robot sounds and imitating the computerized sounds with our human voices and bodies, and to see what artistically that throws up, and it was really nice, actually. I think it was a really good, effective experiment.

Everything in the event has been some kind of image. It's either been an image of imitation and experiencing the process of imitation, both from humans imitating sound and robots imitating humans, and humans imitating robots, and humans imitating landscape, but also machines imitating humans, machines imitating landscape, machines imitating other sounds, and so on.

So there's been a kind of process of imitation, and that's also an image of learning. So in order to understand the world, we are imitating it, and we're trying to represent it ourselves to show our understanding to ourselves.

And in doing that, we've also been coming up against the kind of slightly abrasive relationship between how a sound is made or an experience exists in the body, and how, to some extent, that is unique to that set of conditions. So we come back up against the boundaries of this possibility, the boundaries of our ability to imitate in our bodies a sound made by another body, the inability of the robot to imitate the activity of human form with its physical constraints, the inability of a human to embody the experience of the robot because of its physical reality