We know of course, you know, as people interact, especially when you’re teaching kids, we do a lot to structure the learning experiences and learning environment of the learner because, again, a human environment is incredibly complicated, to help the learner learn. So how do we as the teachers kind of structure the interactions, use our body, use space, in order to help the learner learn what you want to teach them? So the question I want to look at here is how do we use space and our bodies to socially structure a learning episode for people? So we basically looked at two human participants studies. We looked at humans learning from humans as well as people trying to teach the robot. So in some sense, you know, you do an experiment and to verify you can actually, you know, remove a person and replace it with a robot and then see if you get the same kind of behaviour. So, the task that we looked at, it’s a collaborative task in that one person, the learner, knows information for the desired task which is you’re going to construct a figure using at least eight blocks and the teacher actually knows there’s a secret constraint. There’s a secret constraint that the learner has to learn from you in order to succeed at the task. So, we call it collaborative because each person comes to the interaction with partial knowledge and then of course the goal is something like this, where you want to create this figure. So the teacher in this case, the constraint is the figure must be constructed using all of these triangular blocks and none of the square blocks. So how do people do this? How do we teach each other the sort of constraint without using language? So the only constraint we told people is you can’t talk, because if you talk then it’s obvious, but we wanted to see how people use face and their bodies in order to communicate this.
b. Isolating the key cues
So, we didn’t know what cues would be relevant so we captured a lot of data. So we used that Vicon optical motion capture system we talked about that you saw how to track people’s head pose and hand trajectories. We built a special table called a light table that basically has a camera underneath that sees how people are moving the blocks dynamically in time. And then we also had an external video camera to capture more data. Because, again, we didn’t really know what cues were going to matter. We developed a lot of technologies to essentially capture all of this data, which is a mountain of data as you can imagine, and synchronise it in time and even label it and do some basic feature extraction to automate the analysis process to try to understand what are the patterns of activity that are really significant in doing this task. So here you’re seeing the Vicon system if people are moving their hands and gestures, external camera and this is the light table looking up to see who is moving what objects at what time. You can track colour and orientation, block type and so forth.
So, you know, if you look at the data, people do a lot of stuff, you know. So, again, human behaviour is really complicated. They’re doing all these things so there’s various, you know, hand and head gestures, pointing, tapping, nodding, shaking the head, facial expression, ways that you’re moving the blocks, you know, combinations of these cues, you know. So the question is, is there a sweet spot, is there a sub-set of these cues that ideally that could be very simple yet really prevalent and really reliable in helping the teacher and learner, helping the learner learn the right cue?
So, I’m going to let you, you know, think about this for a little bit because there’s a lot of things it could be. But it turns out that the key, there’s a couple, but one of the key signals that’s used is block movement towards and away from the learner. So this is colour coded data of all the block movements happening in a space in one set of interaction and what you see here is the teacher is on one side of the table and the learner is on this side of the table. Teachers tend to move the bad blocks, the irrelevant blocks away from the learner and they tend to push all the good blocks near the learner. And clearly learners are picking up on this because they tend to only use those good blocks. So here’s the very simple but very prevalent cues that we use to physically structure our learning environment for a learner.
c. Data
And it turns out here’s another graph showing that even the amount of movement indicates how good or bad, so the further you move it towards or away from the learner indicates like how completely irrelevant this thing is or how far you push it towards the learner signals how useful that block is to you. So it’s just another way of showing that data. It turns out that for this task, if you add up all of the translations that happen across the table well, the shape is the length of a football field. So people are moving these things very quickly all of the time. There’s a lot of activity going on. And of course you can imagine this is a challenging problem to build a robot that can actually learn from these interactions but if you’re armed with these cues, you can help a robot do these sorts of tasks by having it be savvy to these cues.
d. Puzzle block task
So this is a video that actually combines two learning tasks together. So the first task is, the robot again that has the puzzle blocks is actually learning how to operate this puzzle block in order to reveal two hidden shapes on this table. So I haven’t talked about this kind of learning and I’m just throwing it in here because it’s kind of cool. But, you know, so Leo, the self-motivated learner when you’re not there he explores on his own, he’s kind of internally driven to be curious and to master these things so he’s learning how to operate the box. But of course the point is if a person’s present, Leo’s exploration should be guidable so the person can help suggest actions for Leo to try, to try different combinations, to highlight significant cues like, you can see these lights are changing on the boxes so to signify important states. So Leo, through collaborating with the person, is starting to learn how to operate these two boxes to reveal a blue shape and a yellow shape that he’s actually going to need in the next task.
So, you know, we fast forward these videos because everybody wants robots to learn but learning videos are really boring to watch. So this is obviously kind of time lapse in order to give you a kind of sense of, now there’s practice and rehearsal, the robot’s learning sub-goals, hierarchies of tasks, how to master those higher order of tasks. There’s a lot of sort of task knowledge that’s being built up incrementally over time.
Okay, now we’re going to do the other interaction. So this is a secret constraint task where the person has a preference of wanting Leo to only use the blue and the yellow blocks. So this is a second kind of learning task for now. Leo, he knows the figure, the robot knows the figure to build which is a sailboat in this case. The person knows the secret constraint or the preference in this case and without talking, Jesse in this case, is trying to get the robot to learn how to do this task. So we have actually brought in human participants to teach the robot. We find that the robot can learn from people off the street which is the point; we want the robot to be able to learn from anyone. So when we use these demonstrations we have to use people in our group because we’re not allowed to show video from people who are human participants.
So, Leo’s starting to pick up on the idea that it’s the yellow and the blue blocks that matter. So now you see he’s starting to use the yellow and the blue blocks to build the sailboat figure and he realises he needs a shape and it’s not there and he’s just learned the skill for how to get it. So this is actually really cool because the robot is immediately applying what it learned in a different context in a new context in order to perform the task.
You don’t see robots doing this a whole lot. This is a big thing. And so he’s continuing to build a figure and then he realises that there’s a yellow shape that he needs and it’s not there but he knows how to get it because he just learned that skill from another interaction and he pulls that into the figure. So, yay Leo. So, I mean the reason why this video, I think, is really significant is it’s showing the robot’s integrating multiple forms of learning through multiple kinds of interactions to immediately apply that knowledge to solve a problem. And again, I mean you don’t see machines doing that a whole lot. So this is really, I think, a milestone in these kinds of systems.
a. Achieving Companionable Robots
Okay, so that’s a lot about Leo. I’m going to start talking about some other robots now. In particular I want to talk about again the social cues and how they can impact human perceptions along these sorts of social dimensions. So looking at social influence. So, you know, one challenge of robotics, personal robots in particular, is longterm interaction. All these studies I’ve shown you are very cool, you know, they’re based on these, sort of these classic psychological experiments. They take about five minutes to do. But robots, personal robots are going to be in your house for a long time, right? So this challenge is how do you build robots that can sustain an engaging, rewarding, interesting longterm interaction with you is a real challenge. And I’m sure many of you, you know, remember Clippy the Microsoft paperclip. You know, when this goes wrong it’s not a good thing, right? You don’t want Clippy to be your robot right? So, longterm interaction is a really important issue.
b. Weight Management Dimension
And the question that we asked in this work I’m going to talk about is, you know, can a robot help a person to achieve some song term behavioural change goal that has, say, positive, enduring outcomes along the health dimension? So, the domain we looked at was weight management. And weight management in the United States, it’s a huge issue. 65% of Americans are either overweight or obese. We know that a lot of chronic diseases later in life are tied to being obese when you’re younger. So in terms of burden on the health care system and your quality of life, managing your weight is really, really important.
This is a longterm endeavour where it’s been found that social support by your friends and your family is a very, you know, successful way of helping a person stick to a diet and exercise programme. So it’s not losing the weight, it’s keeping it off is the challenge. Social support is valuable and important. So the question is can you design a robot that can help you maintain a diet and exercise programme by building a longterm rapport with you?
So in this case, this is the Autumn robot. You know, we actually built a number of these robots and put them in people’s homes in the Boston area. They had to be really simple just because they had to actually work so much more simply than Leonardo robot. So, I mean, just, it has just a couple of cues. It can make eye contact with you, there’s a camera in the forehead and so it can maintain eye contact and can basically do sort of shared joint exercises, looking at you when it’s talking to you, looking at information on the screen when it’s talking to something on the screen. It uses speech synthesis to speak to you but there’s, it’s not doing speech recognition. All the sort of language interactions are based on text on the screen and again we do that for robustness issues.
a. Social network
But the idea here again is, robots and social robots are maybe an intriguing technology because not only can they be useful to you in helping you manage your network of technological devices like a Bluetooth scale or a pedometer to help you with just the logistics of keeping track of your weight in terms of how many calories you’re burning off or even perhaps how many calories you’re eating. But it could potentially also play a role in your social network. So a big theme in a lot of our work is, you know, a lot of old work in AI was about robots are going to replace us, you know. And this newer, more enlightened viewpoint is that robots are going to empower us and support our social networks. They have to be integrated into our existing human networks. So in the case of this robot, if it’s interacting with you everyday, like just maybe five minutes, that’s not someone you’re going to hire someone to do, just to do for five minutes. So this is something that makes sense to have some sort of robot do for you.
b. Tracking progress
But it can help you keep track of your progress and share that or have you choose to share that with your doctor and your trainers. So everyone has much more detail and much more accurate information on how you’re doing so they can provide you with better advice as well. So we want these robots to be empowering not only for you to achieve your goals but also to empower the other people who are helping you to achieve those goals as well.
So I’m going to show a quick video then of this robot, it’s called Autumn. Synthesised voice because it’s got to generate the speech on the fly. So you’ll notice, I mean a lot of the dialogue that the robot uses, it’s based on patient-therapist dialogue so there’s a lot of talk about, ‘Help us to help you achieve your goals.’ So there’s a lot of team building, social rapport building dialogue as well as these non-verbal cues. And there’s actually a lot of other cues, the robot actually has a model of the state of the relationship, so there’s a sort of initial phase where you’re getting to know one another and the explanations may be more detailed. As you interact with the robot longer, it understands that you already know all that stuff so the interactions become much shorter. So there’s a lot of modelling in the relationship and the knowledge that the robot is using to incorporate into its dialogue.
So, we did a study where we built, again, a number of these robots and we put them into people’s homes in the Boston area. People ranged in age from roughly 18 to 72 years of age, so a wide demographic of people, looking at a sixweek study. So we told them they had to interact with the robot mandatorily for four weeks and then they could have an optional two weeks if they chose to keep the robot. And that’s one way to assess engagement and liking is that if they choose to keep the robot an additional two weeks and they don’t have to, then clearly they want to keep interacting with it.
And we looked at three conditions. So the three conditions were the Autumn robot that you saw, a computer that ran the exact same software, meaning it gave you the exact same advice as the Autumn robot and had the same touch interface. So basically you’re just removing the social embodiment. The dialogue and the information is identical. And then just pen and paper logs because that’s typically what you’re given in a weightloss clinic. So these are the kind of control interventions that we looked at. And again, what we were trying to understand, the question was longterm interaction and also quality of the working alliance. So, is the embodiment or the social cues a contributing to a positive working alliance between the human and the robot because that would be indicative of longterm interaction and achieving your longterm weight loss goals.
Okay, so I’m going to let you think about that. Do you think it matters that it’s a robot? Because the computer is giving you the exact same advice, same quality advice. And, you know, the reason why I’m presenting this work is because the robot did a lot better, right? So, it turns out that if you just look at, you know, length of time interacting with the robot, people chose to interact with the robot almost twice as long as they did the computer. Same advice, same everything. There’s something about the embodiment and I think the social rapport essentially of the human and the robot that was built that was just different than the computer or of course, using the pen and paper logs. So people chose to stay and interact with the robot significantly longer.
The other thing is, if you look at sort of standard measures like questionnaires that are used to measure quality of working alliance between say patients and therapists you can apply the same questionnaire to assess the working alliance between the human and the robot. And again you see the robot is scoring much higher on the working alliance inventory. So it seems that people also perceive a much stronger working alliance with the robot even though the advice was identical.
Trust is really interesting. People trusted the robot more than the computer and it turns out they also saw the robot to be more credible. And then the emotional relationship between the robot and the other conditions was incredibly different. So, people would clothe the robot, they would name the robot, when we would pick up the robot they’d come out to the car and say goodbye to the robot. They did not do this with the computer. So the emotional engagement of people to this technology was markedly different.
So, you know, when I started this work with robotics, you know, there was a lot of animated agents, you know, screen agents and so on. People would always ask, ‘Why build a robot? Does it really matter?’ And if all that matters was seeing these visual cues and clearly there’s something going on with human psychology that it really matters that it’s in your world, it’s of your world, it’s in your space, it’s not just a sort of thing on a screen. In ways I think we still have to really understand it matters to people that, I think, this thing is physical. So there’s a lot more, I think, to be understood.
So, one other thing that robots can do that animated agents can’t do is touch via physical interaction. So what about the role of, say, of social touch in human-robot interaction? Now, we know, again, from the human-human literature, social touch such as the handshake can be seen as a warm and open gesture, it can contribute to persuasiveness and liking of people. But it’s complicated because depending on who’s shaking your hand and in what context it could have a negative result as well. If you’re a woman and you’re feeling threatened by this in some way, you’re going to have the exact opposite reaction.
And as you can imagine, there’s a lot of really complicated gender stuff between shaking someone’s hand if you’re same gender versus a different gender. So touch is very complicated in humanhuman interaction. Now, you can imagine, one of the intriguing things about robots is you can use them, in essence, like a controlled experimental tool to control these non-verbal interactions like eye contact, interpersonal distance, handshakes and so forth in a way that’s impossible to do with another person. So we know all of these cues matter, we know all of these cues shape our perception of trust and liking but we don’t know exactly in a quantitative way how and by how much. You can’t tell a person, disable your eyes now and interact with this person and then we’ll see how they perceive you in terms of how likable you are. Or disable your, you know, mouth so you can’t do that with people but you can do that with a robot.
a. Bringing the robot to the people
So, we did a study at the Boston Museum of Science. This is our latest robot, Nexi here which is a mobile humanoid robot interactive with over 300 people in the computer, Conners Place, which is kind of the Science/Technology area of the museum. And a lot of our work now is trying to get these robots out into the real world with real people because, as you can imagine, people interacting with a robot in a public space, I mean, people have a completely different mindset than if they’re going to your lab to interact with a robot at MIT versus bringing the robot to people’s turf, so to speak, and having the robot interact with them in their space. So it’s really important, I think, to get these robots out there in human environments to really kind of understand, you know, what the impact is on people.
b. Interaction with participants
So this is just some snapshots for computers, Conners Place. You know, we try to control the interaction as much as while there’s a lot of distraction, so human participants were recruited. They were brought into an enclosed area so there wouldn’t be a lot of visual distraction so they could kind of focus on the interaction with the robot. People would come in groups potentially or alone. You know, you might be with your friends or your family, so in that case we would choose one person to be the participant in the study and the other people were requested to kind of stand, you know, behind a certain line in the background. So people sometimes were alone, sometimes they weren’t.
c. Donation test
Often when you do human participant studies you have to compensate people for their time so we gave the people upfront $5 in $1 bills. And the reason why you did that is because one of the measures you want to look for in terms of persuasiveness, the role of say, touch and gender and persuasiveness was that the robot was going to ask them for a donation. They didn’t know that going in, that it was going to ask them for a donation. And people like their money; they don’t so give up their money. So, how much money they gave the robot might be an interesting measure in terms of, and whether they gave money, of the persuasiveness of the robot as you bury gender or interpersonal distance or touch or so forth.
So the interaction was basically, they should bring people into this space, you know, it’s a museum setting so the robot gives an educational message, it was a sort of mildly interactive, you know, the robot wasn’t, I’d say, a richlysophisticated activator like Leonardo. It was a simple interaction. At the end of the interaction the robot would thank you and shake your hand in the case of the touch condition say. And then it would ask you for the donation. And the question is, like how much money would people give and this is the donation box shown here. After the whole experiment they could go to another area and do a questionnaire and so forth. So that’s basically the study.
Protocol. Now, you know, again, we talked about gender as interesting implications for touch, so we had two gendered robots. Of course robots have no gender but you can manipulate that to some extent by just the voice that you give the robot. So, in the female robot condition [voice plays on video]. So this is a little interaction here. So this is the beginning of the interaction, the robot goes into, you know, a little more technical description and again, it’s an educational message. Now this is you know, the male condition here. So you can see the robot’s making eye contact here so it’s looking up higher here but it’s basically just changing the voice. So it’s exact same animations, exact same everything, you’re just changing the voice. So that’s how you control, in some sense, the gender, vary the gender of the robot.
a. Effect of handshake on donation
What we found were some interesting things. It turns out that being in a group or alone had a significant impact on how much money people gave the robot. So one thing that you see is, you know, in this situation where people didn’t, the robot didn’t shake the person’s hand there’s really not that much difference so to speak in how much money people gave. But if the robot shook your hand and you were with other people that saw you shake the robot’s hand you gave a lot more money. A lot more money. All you sales people, you get a lot more money. But if you were alone people tended to give less so that might say something about kind of social expectations or expectation violence. The robot’s handshake, frankly, wasn’t the most compelling handshake in the world. I bet if we had a better handshake that was warmer or whatever, I bet we could have pushed this up. But the intriguing thing here is the group dynamics had a real significant impact.
b. Effect of gender
Now it also turns out that gender was really interesting. So, if you didn’t have a handshake condition, and this was women interacting with the female robot or a male robot and men interacting with either the female robot or the male robot, you see a crossgender preference. And you see this in human-human interaction as well. Women prefer the male robot or found the male robot more persuasive and men found the female robot to be more persuasive. Okay and this is with no handshake. But if the robot shook your hand it flipped and women found the woman, the female robot to be more persuasive and men found the male robot more persuasive.
So what does this mean? Touch in robots is really complicated and there’s a lot to be understood here. And it’s typical because when you do a study like this it just turns out that nobody has done this exact study yet for humans to know what humans do. So the intriguing question here is, you know, one theme is that, you know, what happens when robots really do mirror what you see in human-human but the other question is what happens when they don’t? And what’s going on when they don’t? So, this, sort of begs the follow on study in terms of what would people do in a situation if it’s a human-human study? There hasn’t been one done yet and maybe we’ll do that.
But again, I think the bottom line here is that touch is complicated in human-human situations and it’s probably going to be interesting and complex in human-robot interactions.
« Back to Lecture