Solving touch: the last mile of robotics

When will we solve the “last-mile” problem of robotics: robotic touch? We’ve made robots that can go to the bottom of the ocean, that can operate in hard radiation in a melted-down reactor, and can go to Mars, fly or drive around, do basic science … and so much more that humans can’t …

But so many of our robots can’t do the simplest things humans take for granted, like pick up objects, handle them, move them, and work on them. Or identify by touch if an object is a flower or a nail.

It’s kind of the last mile of robotic capability.

A company that’s building undersea robots for the Navy is working on exactly this problem. In this TechFirst we chat with Jorgen Pedersen, who built RE2 Robotics in Pittsburgh, sold his company to Sarcos, and now serves as COO of the combined company.

Check out the story on Forbes, or scroll down for video, audio, and a full transcript.

Subscribe to TechFirst

Transcript: Solving robotic touch (and defusing undersea mines, assembling solar power plants, and more …)

(This transcript has been lightly edited for length and clarity.)

John Koetsier: When will we solve the last mile problem of robotics? We’ve made robots that can go to the bottom of the ocean, that can operate in hard radiation in a melted-down reactor, robots that can go to Mars, fly around, drive around, do basic science, and so much more that we can’t. But so many of our robots can’t do the simplest things humans take for granted, like pick up objects, handle them, move them, work on them, or identify by touch if an object is a flower or a nail.

It’s kind of the last mile of robotic capability, and one of the megatrends I follow is robotics and automation. This is right down the strike zone. There’s a company that’s building undersea robots for the Navy that is working on exactly this problem. They also make helper bots for the solar industry and for perhaps the toughest place to put a robot, construction sites.

Here to chat, is Jorgen Pederson who built RE2 Robotics in Pittsburgh, sold his company to Sarcos and now serves as the COO of the combined company. Welcome to TechFirst, Jorgen. 

Jorgen Pederson: It’s great to be here. Thank you very much for having me on the show. 

John Koetsier: Let’s start really big picture. What’s the end goal for tactile robots? 

Jorgen Pederson: Well, what we’re trying to achieve is getting to that true humanlike capability, right? So we have the ability to be dextrous, to be strong, to be precise in terms, of course, movement from going from point A to point B. But, and most of the industrial automation market is really good at repeating the same task over and over again.

But to perform those more challenging tasks, you need to start incorporating more humanlike capabilities and one of those is that ability to feel, right? It’s called haptics. It’s called perception. The ability to feel and to create shape from feel, you know, as a human being, you can close your eyes, grab an object, and with your mind you can recreate what objects you’re touching … in your mind. So the touch is also a very powerful sense that needs to be incorporated to take us to the next level, and that’s exactly what we’re endeavoring to solve. 

John Koetsier: Talk about that next level. What does that enable? If you build a sense of touch into a robot, what does that unlock? 

Jorgen Pederson: Well, if you are in even an industrial automation situation, but even going beyond the factory floor, once you’re grasping objects, knowing if you’re grasping it too hard, too softly, if an object is starting to slip out of … figuring out how to optimally grasp an object to ensure that you have a firm grasp. And if you look at a human hand, it’s a conformable grasper.

So if you were to throw a baseball at a human hand, that is an underactuated system where you hit that ball and it hits, and all your brain is saying, “Just close,” and your digits naturally conform to whatever shape is in your hand. So we’re mimicking that human trait in the way that we’re approaching this. And now we’re adding in also the sensing that’s going to give us that feel to know are we grasping hard enough? Are we, is it starting to slip out of our hands? What’s the texture of the object that we’re grasping?

These key attributes will allow us to perform more humanlike tasks in a safe and more effective manner. 

John Koetsier: The human hand is pretty crazy. It’s incredibly sensitive. I did a little research just exactly for this show, literally this morning. There’s a University of California study in 2017 that found that we can feel the difference between surfaces that differ by just a single layer of molecules. A single layer. That blows my mind. I’m sure there’s different people that are super sensitive and there’s probably me that, you know, oh, is there a baseball or is it a football … but do you think robots will ever get there? 

Jorgen Pederson: Absolutely. It’s just a matter of time. And the perception, the feeling part of it, that may be the easier part. You have these multimodal sensing solutions, but interpreting what you’re sensing, that’s the hard part. And this is where the human brain is amazing that it can take so many forms of… it can take heat, pressure, vibration, and combine them all very effectively to provide a very solid understanding of what’s being interfaced with. 

John Koetsier: That’s interesting, right? Because that goes back to what you mentioned earlier, which is you’re building a mental model of the object which you kind of… I don’t want to get to this level of consciousness or something, but you’ve obviously got some level of AI that you’ve got to put in place if you’re going to, you know, what does this feel like? What are its dimensions? What’s its texture? What’s its weight? How strong does it feel? Does it give a little when I press, right? All of that. That’s pretty complicated. 

Jorgen Pederson: Yeah, absolutely. Both parts are equally important. The sensing part — how do we sense at a similar level as a human — but also the reasoning part, the fusion, and applying AI and other software techniques. And it’s most likely going to be a multi-algorithm approach to ensure that we’re providing the correct reliability that we need in order to perform some of these complex tasks. 

John Koetsier: And there’s even a third component, right? Because what you’re doing is you’re building robots that can go places people can’t and then relaying that information to an operator that might be kilometers/miles distant, who knows where, in some safe place. Talk about what you’re working on for the U.S. Navy. 

Jorgen Pederson: Sure.

So we are developing a conformable underwater hand that has the ability to feel. And once again, it’s that multisensor approach. We’re partnered with UCLA, Dr. Veronica Santos, and University of Washington professor, Jonathan Posner. They’re creating that… they perform that basic research to really solve some of those really hard problems to provide that feeling, provide that sensing. Now, what we’re doing is taking that foundational research and applying it to real-world robotics. We’re going to do that first through this STARFISH hand that we’re developing, we call it… it’s an acronym, don’t ask me to repeat it, but… 

John Koetsier: [Laughing] I saw what it stood for. It was a very long acronym,* but I like the word STARFISH. [*Strong Tactile mARitime hand for Feeling, Inspecting, Sensing and Handling]. 

Jorgen Pederson: Yes.

It’s providing this conformable grasp where you can feel, and this is really important if you think of some of the tasks that Navy divers are asked to do. Often they’re in turbid waters where your visibility is limited. You may be having to perform a task and going back to, you may have to do half of it, and in some cases all of it, based on feel versus what you can see, right? And so having this capability is critical to enable certain tasks that the Navy is looking to achieve.

And one of those might be dealing with a water-borne IED, right? Think about the USS Cole. If we could have known about that threat, they would’ve sent a diver down to assess and render safe that threat. Those are the types of tasks that naval divers do today. No human should be doing that job, right? Or if they are, they should be at least remote and away from the danger. And that’s where we come in.

John Koetsier: Talk about that USS Cole. I mean, when I’m hearing that I’m thinking of the U.S. Navy vessel that was attacked by like a power boat that was loaded with explosives, but I think the USS Cole was different. Was that something attached to the side? 

Jorgen Pederson: That’s correct. Yeah. So, oftentimes devices can be placed alongside of a vessel and usually explosive devices to render that vessel inoperable. And, you know, naval divers are the ones who have to go take care of those threats. So, we need to remove humans from harm’s way, and robotics is a perfect way to do that.

John Koetsier: Absolutely. I mean, I can’t imagine being somebody who defuses bombs on land, nevermind dealing with them underwater. And you mentioned turbid waters, I mean, oftentimes these ships will be attacked perhaps in a harbor. Harbors are not known for having clear water ’cause there’s a lot of ships moving around, a lot of screws churning up the water, the mud, all that stuff.

So, I mentioned that third piece because you talked about the sensing, you talked about the integration of the data, but the third piece is that telling an operator or helping an operator feel what the actual robot is feeling. So you’re working on these remote technologies, the operator is safe — maybe on board another vessel, maybe in the harbor — the robot might be deep down below or maybe even, in another case, maybe in a nuclear reactor. What kind of feedback can the operator get about what the robot is touching? 

Jorgen Pederson: So, the answer is it depends. If you have a good pipe with high bandwidth, low latency, you can provide a very strong feeling back to an operator when you’re dealing with a teleoperated system.

Where we’re moving towards is the concept of supervised autonomy, where you’re pushing more of the thinking, the AI, the software, down onto the platform and letting it close the loop, which can be done… you can have a very tight control loop locally with the robot. And then you’re just providing higher level feedback commands to the operator and seeking guidance on how to move forward with the mission.

So I mentioned water-borne IEDs which is performed by expeditionary explosive ordnance disposal crews, but we also are tackling what’s called the mine countermeasures mission for the Navy. This is where you may have a moored mine in a harbor or in an area and it’s meant to keep ships, whether they’re ships or submarines, from coming into an area. But we want to render these devices safe.

So, with that same system that can be teleoperated, we now have the ability to autonomously swim to a threat, locate it, grab a device that’s on board, arm that device, place that device on the threat, and render it safe without it going high order. 

John Koetsier: Wow. 

Jorgen Pederson: So, in this case, the reason I’m bringing this up, the human’s still involved but they’re there more as a supervisor, a guide that understands all the nuances. And if there’s an edge case, you know, an error, something that hasn’t been seen before by the robot, the human’s still there and they’re providing that input on how to overcome any error or new situation that arose, and doing it by sending information which doesn’t require as much bandwidth or latency. And now you can extend your reach and you can go farther.

And eventually the person will be simply be onshore and these ships and ROVs and underwater vehicles will be out performing tasks, sending small data packets back to shore where the operator is there to provide that high-level guidance to a mostly autonomous system. And really, talking about keeping people out of harm’s way, having people back on shore and you’re still able to complete these tasks very effectively anywhere within the ocean. 

John Koetsier: And, of course, if you happen to have a high bandwidth connection, then you could get perhaps higher levels of tactility and perhaps higher levels of not just telling the robot the strategy or the goal, but actually operating in terms of the tactics and everything as well.

Jorgen Pederson: Right. 

John Koetsier: Where do you see this growing in the next five years or so? What do you think you’ll be shipping in five years? 

Jorgen Pederson: Yeah. I mean, we’re going to be seeing enhanced end effectors that are entering the market. The majority of what you see out there is a two-finger gripper, right? And there’s a good reason… 80% of what you do as a human just requires a thumb and an index finger. Okay, you add in the ring finger and you’re getting more of a secure grasp. The other digits are marginally useful to…

John Koetsier: [Laughing] Marginally? 

Jorgen Pederson: You know, does anyone really need a pinkie? Really, no… 

John Koetsier: [Laughing] I don’t know. Ask the Yakuza. 

Jorgen Pederson: Yes.

But we are going to see these more humanlike hands. Maybe you don’t need five fingers and it doesn’t necessarily have to be completely anthropomorphic initially, you know, where it’s replicating a human design, but it should be conformable. It should be underactuated, which means that it can be affordable, right? Because if you were to actuate every single link of a finger, those costs add up significantly.

So, the idea is that we’re going to have these multi-digit conformable hands that have enhanced sensing out there that will enable more tasks that currently only humans can do. But in the next five years, we’ll be able to see robots helping to augment human capability and allow for supervised autonomy in new markets. 

John Koetsier: Well, that’s pretty amazing because you’re obviously doing stuff that’s undersea. You’re also working in the solar industry. You’re also working in the construction industry, which is a notoriously tough space to put robots… it’s chaotic, it’s always changing. That pile of things you put there yesterday isn’t there today because somebody used it and built something with it, right? And so it’s always changing. 

Jorgen Pederson: Right.

John Koetsier: You could also assume… we’re really opening up space right now. I mean, we’re launching, half the satellites that are in orbit have been launched in the last few years, right? Elon is working really hard at that, right? You know, there’s going to be some opportunities in space as well, one would think.

Jorgen Pederson: Yeah, no, absolutely. Solar is a perfect example of where robotics is needed.

We have a program with the Department of Energy right now, it’s in partnership with some electric power companies as well as EPCs… those are the companies that install solar fields for energy companies. There’s a strong pull for this type of technology, and there’s a couple reasons. One, there’s a labor shortage. Most people do not want to do that job. It’s usually out in the middle of a desert or some remote place where people don’t want to go.

John Koetsier: And that job is what? Cleaning the solar panels so that they’re at high efficiency all the time? 

Jorgen Pederson: No, this would be the installation. 

John Koetsier: Really? Wow. 

Jorgen Pederson: Right. Yeah. So the people pick up these photovoltaic panels or PV panels and move them from a delivery vehicle over to a racking system. Then they have to secure it to the racking system, and that racking system it can rotate and track the sun. But these panels weigh around 50 pounds right now. But the industry’s moving to higher efficiency panels, which means that these panels could weigh up to 100 pounds soon, which would make it even more challenging for humans to be able to perform this task. Right now, it’s very prone to injuries, fatigue, you know, this isn’t a job that people can do into their later part of their career. 

So, this is a perfect opportunity for robotics where you could still have those domain experts, they’re on site, but you’re putting the brawn on the robots, right? And now what was, you know, a crew of six people is now down to three people, right? So we’re addressing the labor shortage, but we’re also extending the careers of those people doing that job because they’re not doing that backbreaking work themselves, they’re just there to help guide these systems.

And then there’s another win that we could do it with higher quality and increased speed. So now we’re getting to green energy that much faster, which is helping society as well. So it really is a win-win situation of a perfect application of applying robotics to an existing yet growing industry. 

John Koetsier: And sensitivity’s pretty important there as well, because maybe you’re lifting that hundred pounds, but you do not want to damage those expensive panels, right? 

Jorgen Pederson: Yeah, that’s right. It’s a common occurrence to create microcracks in these and those microcracks during installation really affect the efficiency of the panel. And then as you go through the install afterwards and you find these, it then becomes even more expensive to have to go back and replace out however many… even 10% of a large solar field is very expensive to swap out those PV panels.

John Koetsier: Sure. So, Jorgen, you built a lot of this technology that we’ve been talking about with your company RE2 Robotics, I believe, in the Pittsburgh area. You recently sold it to Sarcos. Is that just for scale? Is that just for capital? Is that for joining forces with… I mean, Sarcos, I actually interviewed them too, probably something like a year ago, and they’ve got some pretty cool tech. Is one plus one equaling three here? 

Jorgen Pederson: That’s exactly why we did this. Well, first of all, both companies had the same mission. We’re improving worker safety and productivity through robotics, right? And we are providing human augmentation systems, whether it’s an exoskeleton, which Sarcos is mostly known for, but also these mobile manipulation systems, so, putting these robotic arms on vehicles that move through the world to perform tasks outside of the factory floor, and really that’s where we play. And that’s where that was the perfect fit strategically. RE2 has been developing these robotics systems to work, to go where humans go, to do what humans do.

And Sarcos has been designing these systems to augment human capability, as well.

So when you bring those two worlds together, I’m really excited about the cross-pollination that’s going to occur here… and what’s really exciting is that the technology’s complementary, right? So all we’ve done, really, is put together two world-class teams of rare robotics talent that have been looking outside the factory floor. Now, that said, we could still work inside the factory floor, but what’s unique is we can go wherever humans go, right? 

John Koetsier: Mm-hmm 

Jorgen Pederson: And, you know, you bring those two worlds together, now we have this huge portfolio of complementary technologies, our toolbox is that much bigger now. So whenever we’re looking at a new market, or even new applications within an existing market, the breadth of solutions that we have to bring to bear is incredible. And that’s what’s really exciting about these two worlds coming together.

John Koetsier: Is it significant that you’re in the Pittsburgh area? I mean, there’s long been a history of innovation there, and as we move sort of into the internet/web/cloud era, a lot of the locus of innovation just in the States in general, of course, there’s always been a lot in New York and Austin, you know, but a lot has moved to the West Coast. A

nd as we enter kind of new phases, I mean, we kind of built out cloud, we’ve kind of built out social. There’s going to be continual evolution there, but some of the hard technologies that are going to change our world, whether it’s solar, whether it’s green-energy related, whether it’s automation, that’s going to make our lives better, you know, those sorts of things. We’re not seeing, we didn’t see so much of that let’s put on the West Coast.

What’s significant about being in Pittsburgh? 

Jorgen Pederson: So, I’ll state that when I founded RE2 in 2001, there were six robotics companies in town, in Pittsburgh. When I sold RE2, there’s over a hundred robotics organizations in town, and that momentum continues to grow.

And it’s significant, especially because there’s a uniqueness in Pittsburgh, you know, we have the Pittsburgh Robotics Network. And that’s unique about Pittsburgh, it’s very collaborative, and it’s a community where we we’ve machined parts for other robotics companies in town. We’ve lent engineers to other robotics companies in town, and vice versa. We help each other out. That ecosystem and the camaraderie is very strong, and I think that’s part of what Sarcos saw is being able to tap into this rare talent that is in Pittsburgh. There’s now a critical mass here and it’s continuing to grow.

Companies are moving in, you have Carnegie Mellon and other universities still creating this new talent every year. So we’re leveraging that as Sarcos is tapping that in order to continue to innovate, continue to fuel exactly what we need to do…

John Koetsier: It’s really interesting to me because, I think it was three years ago, pre-COVID, I visited Odense in Denmark, and I kind of dubbed it “Robot Island” or “Automation Island” because that’s a place… Denmark is a tiny country, 10 million people, right?

This is a tiny island, maybe a few hundred thousand people, but they had something like a hundred different companies there that were all focused on robotics and different aspects — gripping, moving, all the lifting, all that stuff — and that’s exactly what they told me, is that having that sort of critical mass of companies, of talent, of “we can do this part, you can do that part, we can collaborate,” made it a hotbed globally on this one island in this tiny country in Scandinavia.

And perhaps something similar is happening in Pittsburgh. 

Jorgen Pederson: It absolutely is. I mean, the Pittsburgh Robotics Network is very strong now. And what’s even more powerful for Sarcos is we have the Pittsburgh location, we also have, we’re in Salt Lake. So now we have a kind of, not really East Coast, not really West Coast, but both sides of the country represented. So, in terms of the war on talent, we now have access to both sides of the country, so if someone would rather be on one side of the country or the other, now we’re providing flexibility as we…

John Koetsier: That is another interesting point you mentioned, because the war on talent, right? I mean, that has changed in the last three years as well. Whereas everybody wanted to be… I wanted to be in New York, I wanted to be in San Francisco, I wanted to be wherever, Miami, you know, whatever it was… people are going back to where they came from. People are moving places they never were because I can work from anywhere. Anyways, Jorgen I want to thank you for taking this time. You’re taking this time for this interview on the day you’re actually moving, so really do appreciate that. Thanks for your time and have yourself a wonderful day.

Jorgen Pederson: Yeah. Thank you very much. I appreciate it.

TechFirst is about smart matter … drones, AI, robots, and other cutting-edge tech

Made it all the way down here? Wow!

The TechFirst with John Koetsier podcast is about tech that is changing the world, including wearable tech, and innovators who are shaping the future. Guests include former Apple CEO John Scully. The head of Facebook gaming. Amazon’s head of robotics. GitHub’s CTO. Twitter’s chief information security officer, and much more. Scientists inventing smart contact lenses. Startup entrepreneurs. Google executives. Former Microsoft CTO Nathan Myhrvold. And much, much more.

Consider supporting TechFirst by becoming a $SMRT stakeholder, connect to my YouTube channel, and subscribe on your podcast platform of choice: