AI, drones, & country-sized digital twins: mirroring the real world in code

Imagine building a digital twin of an entire country: all inputs, outputs, activity, infrastructure, issues, challenges …

That hasn’t happened yet, but a client used Bentley Systems’ tools to create a baseline digital twin for Singapore. In this episode of TechFirst with John Koetsier, we chat with Greg Demchak, who leads digital innovation at Bentley.

Increasingly, the company is using digital twin technology, AI, and drones to monitor, protect, and maintain massive infrastructure projects including wind farms and power lines, and Bentley is helping build the ITER nuclear fusion plant in France.

Scroll down to watch our chat, subscribe to the podcast, and get a full transcript of our conversation …

Watch: AI, drones, and digital twins

(Subscribe to my YouTube channel)

Subscribe to the TechFirst podcast

 

Full transcript: using digital twin tech to monitor, protect, and maintain infrastructure

(This transcript has been lightly edited for length and clarity.)

John Koetsier: Hello and welcome to TechFirst. My name is John Koetsier. Today, we’re chatting with the Senior Director of Digital Innovation Lab of Bentley. No, not that Bentley, I’m talking about the multi-billion dollar international software and engineering company. Increasingly, Bentley is using digital twin technology, AI and drones to monitor, maintain, protect, even build massive infrastructure projects including wind farms, power lines, those sorts of things. The company’s also working and helping to build the ITER Nuclear Fusion plant in France. Our guest is an adjunct professor. He’s a virtual design expert. He now leads digital innovation at Bentley. His name is Greg Demchak. Welcome, Greg!

Greg Demchak: Hello, John. Thank you for having me today. 

John Koetsier: Hey, glad you could be here. I know it’s a little late on a Friday. You’re in London which has gotta be something like 5:00 PM, something like that, local. Thank you. It took us three months to get this going. I’m glad we could finally connect. 

Greg Demchak: Absolutely. So, yeah, I don’t know where you want to start on this subject, but…

John Koetsier: You built a digital twin of an entire country. Talk about that. 

Greg Demchak: Yeah. So I think on that point, a couple things. So we, Bentley Systems, we don’t actually build the digital twins. Our end users, customers do. So we build software and we’re creating a platform that empowers engineering firms, architects, contractors to build these digital twins and get value out of them.

So, the Singapore example is a laser scan representation of the entire island of Singapore. And in that case it’s using lidar scanners from on the ground, but also airplanes and basically drone capture to produce a high resolution like point cloud scan of the conditions of the island. And that becomes a baseline capture of reality that’s basically digitizing reality that you can then move through, fly through, interact with that asset through a web browser or a desktop application or even inside something like a VR headset.

And so that really is sort of a starting point of a digital twin is let’s capture some aspect of our existing conditions, ’cause the world already exists in many ways. So the first step is really to do a digital capture. And once you have the digital capture, you can then start overlaying spatial content on top of that capture.

So that could be real time IoT sensors related to a specific asset somewhere in that location. It could be when you’re adding new engineering models, they’re geopositioned, geolocated against that reality capture. So that’s where you start to see the intersection of sort of like maintenance and operations with design and construction, right? Because the design model is always going to have to be located within some context. 

John Koetsier: Yes. Talk about why people are creating digital twins. What are the core uses? 

Greg Demchak: Yeah, so I think, on this point, I think this is worth mentioning. If you talk digital twin broadly, I don’t think it’s necessarily a requirement that you have a 3D model for a digital twin to exist. In fact, a digital twin could just be streams of data about a physical asset that then you can react to, right? And that’s legitimate. And there’s plenty of examples of digital twins like that, sensor data being surfaced by a physical object that produces trends and it can tell you the state of something.

Bentley Systems, we add another layer of value to the digital twin which is the introduction of 3D geometry — either point clouds, photogrammetry meshes, or 3D CAD engineering models. And why is that important? What’s sort of the value of that? One, it allows our end users to contextualize where that information is in geospatial coordinate space. Like, okay, where is that sensor actually? Well, with a three-dimensional digital twin, you could actually be directed towards that specific exact sensor on a machine relative to the 3D model. 

So, two things. One, I think gives more value to the raw data by contextualizing it, where and when is that happening? And two, it also extends the value chain of investment in 3D modeling or scanning, right? Because if you just build a 3D model, that’s really valuable. You can use that to immerse yourself in it. You can visualize projects before they’re built. But now if we’re saying you can continue the life of that 3D model by connecting it with IoT, you suddenly unlock more possibility, more value for that initial investment in 3D.

John Koetsier: Yeah. You can see how that’d be super helpful and super useful if I’m gonna add components, if I’m gonna add infrastructure. How’s that fit? How’s that gonna look? What’s that gonna feel like? How’s that gonna interoperate? 

Greg Demchak: Yeah. 

John Koetsier: Now, the word twin kind of suggests equivalence, equality, exact duplicate.

Greg Demchak: Mm-hmm. 

John Koetsier: That’s, of course, not true in the ultimate sense because you are not capturing every bit of data about a project, a country, a piece of infrastructure. How much do you have to capture to be calling it a digital twin? 

Greg Demchak: Good question. Well, I think, well, that’s digital twin. Again, it could just be... I think the minimum would just be some amount of data coming off an object, which is even like in some ways, maybe just temperature of an object. The frequency in which you’re capturing that information. So I think as soon as you start capturing any information — time series information about an object — that’s the minimum. 

John Koetsier: Yes. 

Greg Demchak: You’re capturing some information on some regular interval about the state of that thing, and then you can just keep expanding from there.

John Koetsier: Yeah. Yeah. 

Greg Demchak: Okay. You could basically just keep adding more and more sensors, for example, or then you start adding more and more 3D models, or you start doing regular scanning of the object. And if you look at something like a neural radiance field, which is coming out like an… almost like consumer-ready version of reality capture you could do with your phone that’s capturing a very, very high resemblance of something on a higher frequency rate. Then it’s like, okay, what’s sort of like one thing? It’s like what’s the frequency of the information being captured to get it closer to reality? Right? So I think sensors are down to like maybe per second updates. Imagine if also the scanning just the frequency gets closer and closer to reality. So you’re gonna have more tighter resemblance, your digital version will be tighter, more tightly cobbled with reality. 

John Koetsier: Yeah. It’s interesting to think about this stuff, right? Like how high res do you need to be and how high res are we right now? And this is a fairly new… I mean it’s not new, but it’s a fairly new field of engineering. There’s lots of standards in engineering. I wonder if there’s standards right now around how high resolution a digital twin is. How high res it should be. You know, you can imagine that you need one level of resolution for a piece of infrastructure that just is there and doesn’t change that much. And I was gonna say something like a bridge or a dam, but you may need higher resolution for that ’cause they’re critical, I mean, people’s lives depend on them. 

Greg Demchak: Yeah. 

John Koetsier: But there might be some things that, you know, just kind of stay there. Maybe it’s power lines or something. But perhaps another level for, well, you could go all the way to the ITER fusion reactor where you probably would need submillisecond, nanosecond type things. Are there standards around this stuff, like how high res something should be or needs to be in different areas? 

Greg Demchak: Well, I don’t think we’re to the point of standards yet. I think we’re still figuring that stuff out. I think standards will come, based on learning. And it varies, but what I can tell you is I’ve seen… Let’s just take the bridge, for example. Let’s say you want to do a drone acquisition of a bridge. So instead of doing a manual inspection with photos and roping yourself down, you could just do a drone. Same thing with like an offshore wind farm.

John Koetsier: Yep.

Greg Demchak: Exact same. You could deploy the drone and you’re shooting photos. The higher res of the photos probably the better that the AI/ML algorithm’s gonna be at picking up things like defects, cracks, rust, you know, exposed rebar. Right? It probably could be trained better. That’s not to say you couldn’t probably get results out of a lower rest photo, but, you know, you go to a higher resolution and you get good acquisition that’s really good content for AI/ML to run defect detections against, and then render those defects back into an interactive 3D model, for example, for the human operator to then review and check out. So… 

John Koetsier: That’s really interesting, actually, because we’ve seen some infrastructure defects, collapses, catastrophes around the world recently, however, in the U.S. as well, bridges collapsing, those sorts of things, and you wonder, do I need to scan this every month? Do I need to scan this every week? Or is it once a year? And as it ages does that timeframe change? And as I build up this 3D model are there ML models right now for a bridge? 

Greg Demchak: Yep. 

John Koetsier: And accepting visual information and saying, “Hey, check that out closer.” 

Greg Demchak: Absolutely. We’ve developed that. We’ve developed AI/ML algorithms inside of Bentley Systems that do that, that can be trained against like cracks and defects. So, the example there is we have customers in the U.S. that are doing regular inspections on key infrastructure bridges, running it through the crack detection, rendering back those cracks onto the surface and then using that to go do a virtual inspection. I think it’s a really interesting use case.

One, you can do that inspection remotely. So this kind of… almost… this is where you almost go into the whole like metaverse type experience of the experience, ’cause you’ve now captured some asset, you’ve applied some machine learning, and then you can immerse yourself as an inspector and go explore that bridge as if you’re like the drone, right?

So, instead of going physically to the site and with machines and shutting down the asset, or if it’s an offshore wind farm example, by using the drone, you don’t have to then bring the entire asset down offline and lose energy capture. You just have to turn it off for a bit, run the drone, and then you do the really like the detailed analysis remotely in a three-dimensional representation, like a virtual reality capture of the asset. 

John Koetsier: Mm-hmm. 

Greg Demchak: So it saves time. It keeps assets uptime is up. For roads, it means cars are able to, you know, transit, you’re not closing down the bridge. For a wind turbine, it means that thing can be up and operational longer. 

John Koetsier: Super interesting to consider the possibilities here, because if I’m a city or a municipality or a state and I’ve got a ton of infrastructure, I want to have something like this going on. I have literally billions of dollars invested in my infrastructure. I want to know how it’s doing, how it’s holding up against weather conditions, other stuff like that. And you can imagine that maybe this is the reality in Singapore or some other places that are very tightly integrated, very wealthy, and have the capability to do this.

But you can imagine, you know, somebody, an engineer sending that drone out, looking at multiple data streams. Maybe it’s 4K video so you can zoom in really, really close. Maybe there’s some kind of penetrating radar or something like crack detection and those sorts of things that you might find in sort of people check for stress fractures in airplane wings or something like that. Layer on those multiple data streams. That could be super interesting. 

Greg Demchak: Yep. And I’m definitely seeing this happen. I think what seemed expensive, out of reach maybe five, six years ago is becoming more and more accessible to engineering firms.

I think you can just look at the adoption of drone technology over the last couple years, and even the fact that drones now can largely be sent on autonomous flight paths. So it’s like, you don’t even have to have this crazy skill to pilot that thing and get the right shots. You can essentially give it a path and it’ll find its way around and even be conscious of like — not conscious, but you know, like — is it capturing the right quantity and quality of data so that you can do a proper inspection on it? 

John Koetsier: Huh.

Greg Demchak: So, you know, that’s incredible. Like I think we’re seeing the use of drones have a big impact there.

John Koetsier: Sure, sure. Now, when you have actual twins in the physical world, one comes out first. There’s always one slightly older. Have you ever, or would you consider building a digital twin first and then creating the physical artifact? I mean, obviously there’s drawings and there’s plans and even fly-through plans and those sorts of things. Can we consider that a digital twin? The beginning of a digital twin? 

Greg Demchak: I think so. I think… This is how I see that happening. If you’re investing in 3D modeling, which a lot of our clients do, that’s the first kind of projection of a future in 3D format. You’re sort of anticipating an ideal state with a CAD or BIM system, what is planned to be built. And these models increasingly can actually be used to drive CNC fabrication machines. So, in fact the specification, the actual dimensions embedded in that CAD 3D BIM model can drive fabrication. So it’s already… At some point you’ve got the design engineering model and it’s used to fabricate its reality part. So, the design model sort of generates the physical. 

John Koetsier: Wow.

Greg Demchak: And now, once the physical is in place and placed in the field, what can happen is you could then compare where that object was placed in reality. And what you could then do is update the as-built 3D rep model to adjust and then accurately reflect what was really done. Because it’s always the case that you have this perfectly designed 3D model, right? But, in fact everything’s developed with tolerances, so when things are installed in the field there’s always a bit of a variation in what was actually installed. Especially with construction where you really have millimeters, inches, you know, you can have significant tolerances. 

John Koetsier: Yeah. Yeah. 

Greg Demchak: So I think there’s a feedback loop: design, manufacture, install, and then update the 3D model with like the actual location. 

John Koetsier: Yeah. Yeah. So, this is interesting because I live near Vancouver out in the Fraser Valley, actually, in British Columbia, Canada, and last year we had a massive flood and tons of water coming over the border from Washington State, United States, from the Sumas River, a few other rivers that were flooding. We had an atmospheric river, it was a big event. And where I live, I live on a mountain, but below there’s farmland and it actually used to be a lake. And there’s a dam… sorry, a pump system that empties this, and basically thousands of people’s lives, thousands of buildings, probably hundreds of thousands if not millions of farm animals’ lives all depended on this one pump station not failing. Much of it flooded already, but it was all going to flood. And if I was a city manager or province/state infrastructure engineer, I would want to have not only regular signals coming from this sort of critical infrastructure, but I’d like to be able to kind of explore it, see it, virtually as well as physically. Really interesting possibilities you can imagine for the future. And maybe even run tests, like, if this happened, what would happen? How’s that entering into digital twin technology? 

Greg Demchak: Yeah, that’s cool. That’s sort of like a predictive kind of future. So it’s sort of like the other side of your question, like what if you have kind of the pre-digital twin and then there’s like the post or projected future that doesn’t exist. So that’s interesting, like to run simulations on a potential future is really interesting. 

But that makes me think of, you know, quantum computing and this type of thing, like how many possibilities exist out there, and could you run simulations that could show you possible outcomes and then maybe start to design around those? So… 

John Koetsier: Absolutely, and…

Greg Demchak: I think that’s hugely powerful. 

John Koetsier: Absolutely. And then you’re actually running simulations against current state infrastructure or future state infrastructure, not as-it-was-built infrastructure. Because the bridge is never as strong as the day after it was built, right? The dam is steadily weakening. That would be super interesting, super powerful. 

Another area I wanted to get into is extensibility. Every system’s part of another system. They’re all connected to things, the power plant, to the power lines, to the houses, buildings, businesses that it powers, all that stuff. How can you interface digital twins and where’s that sort of fit in with smart cities?

Greg Demchak: Okay, let me try and answer that. So I think…

John Koetsier: It’s just a small question. It’s not that compli– [laughing]

Greg Demchak: No, not a big deal. I think our approach, and I think it aligns with other big software vendors in the market, is in terms of interoperability and exchange and connectivity between different systems.

We’ve really moved a bit from a product-centric thinking to a platform-centric thinking. And when you think platform versus product, it means you’re not just having a product where you have to do everything within that product. If you go with a platform, really the whole value of a platform is how open it is to connect with other platforms through APIs.

That’s the only way, I think, to be successful. I think any digital twin is gonna have to be able to integrate, connect, speak to many other different systems, right? Because there’s endless supply. It’s like sensors on the market. There’s different protocols for communicating. There’s even, there’s dozens of even 3D file formats you have to deal with. 

There’s different drones on the market. There’s different types of photos that come into the system. Sometimes it’s a point cloud, sometimes it’s a photo. So, in order to, I think the idea is that you need to be able to work with a wide range of inputs… and outputs, by the way. And the more open you are as a platform, for both input and output… [lights turn off] Whoop, power just went out here… the better. 

John Koetsier: I can still see you. 

Greg Demchak: Oh, you can still see me. Okay. It’s only 4:30 so it’s… 

John Koetsier: Your little [inaudible] digital twin is losing resolution.  

Greg Demchak: Oh my gosh. But yeah, where was I? No, open and interoperable, I think is super key. And I think all the major platforms are gonna be inter-operating together.

That’s our personal stance. We say, “If you are building a 3D model with any major CAD vendor, it can be processed and connected into our platform.” Same thing with IoT information. We’re largely agnostic about the input. 

And then in terms of outputs too, we’ve taken a new approach to output which is say, “Well, yeah, we have our own rendering technology or web viewing technology, but we also recognize that there’s gaming engine platforms out there” — Unity, Unreal Engine, the Omniverse — so we’ve also said, “You know what? This information can flow into those engines as well as an output.” 

John Koetsier: Amazing. Amazing. Very cool. I remember early on in the Covid pandemic people were having business meetings in Fortnite, right? [laughs]

Greg Demchak: Yeah, yeah. 

John Koetsier: Their avatars get together and… The city I live in just kicked off a drone mapping project at high resolution, I think 4K or 8K resolution, of the entire city and everything like that, and how all this data comes together and how frequently it gets updated. Super interesting. 

We talked about the nuclear fusion project, ITER, in France. Is there a digital twin possibility for something like that? You know, you can imagine the data flow off of an effort like that must be gargantuan. We must be talking gigabits per second of sensor data and nanosecond level kind of changes and stuff like that. What are your thoughts there? 

Greg Demchak: Yeah, I think that’s the plan they’re gonna… Right now, ITER is not operational yet., So that’s still a few years down the line, but there’s gonna be a lot of sensors on that Tokamak keeping track of the status of the temperature, the fusion that’s taking place inside of it. We’ve seen that because where we come in at this point, we are working with the contractor in ITER. The team’s building the reactor. So they’re kind of in that early stage design and construction. So, the digital twin at this point is essentially the planning, the design, and then the construction sequencing as they’re building the Tokamak. So we’re not yet to the point of where there’s sensors, you know, rendered into a digital twin. It’s not operational yet. 

But we’ve managed to, you know, enable that team at ITER to take their super detailed, I mean, extremely detailed 3D engineering models and view them inside of Unreal game Engine and get this very smooth high frame rate navigation. So it’s allowing them to get inside the reactor and check the construction, check the design in an immersive way before they actually assemble and construct it. 

So that’s where we’re kind of at and that’s where the project is at today, is still in design and construction stage, not yet operational. I think they want to have first plasma, but… Maybe you can’t quote me on this, but I think in the next like four years is the plan.

It’s a huge project. Very ambitious, and obviously if they really solve fusion, it becomes a huge game changer globally. I mean, it’s an unlimited source of carbon-free energy, you know, where you put one unit of energy in and you… like a tenfold return out. So, the potential there is really huge. That’s why we’re taking an active interest in it. 

John Koetsier: Amazing. Amazing. And I love what you said earlier, which is that the model, the design becomes the digital twin, gets augmented added, sensor data comes in. Makes a ton of sense. Cool. Greg, thank you so much for taking this time. Really do appreciate you stayed so late in the office. They turned off the lights on you. 

Greg Demchak: The power cut in London. Yeah. [laughter]

John Koetsier: Exactly. The digital twin isn’t updated frequently enough so they don’t realize somebody’s there. Maybe you need to run around for a second and then the sensors will capture you. 

Greg Demchak: Yeah. I don’t know what’s happening. Unfortunately, I don’t know. But yeah, there’s the cost of electricity and gas here has just gone through the roof in the last year. So it’s… I don’t think that’s it, but I can tell you it’s crazy to see how the price of energy has gone up. And I think we’re gonna, maybe that even ties into how digital twins could help, like, just more efficient, better management of energy and core utilities. I think it’s something… I think there’s a huge opportunity there, by the way, to just…

John Koetsier: Interesting. 

Greg Demchak: …have the digital twin monitoring and maybe be more efficient in how we use our energy. 

John Koetsier: Mm-hmm. Cool. 

Greg Demchak: So, yeah, thanks for having me. Really enjoyed the conversation. Amazing. 

John Koetsier: Awesome. Thank you.

TechFirst is about smart matter … drones, AI, robots, and other cutting-edge tech

Made it all the way down here? Wow!

The TechFirst with John Koetsier podcast is about tech that is changing the world, including wearable tech, and innovators who are shaping the future. Guests include former Apple CEO John Scully. The head of Facebook gaming. Amazon’s head of robotics. GitHub’s CTO. Twitter’s chief information security officer, and much more. Scientists inventing smart contact lenses. Startup entrepreneurs. Google executives. Former Microsoft CTO Nathan Myhrvold. And much, much more.

Subscribe to my YouTube channel, and connect on your podcast platform of choice: