iPhone 12 Pro’s lidar enables 100X faster 3D scanning than conventional photogrammetry

There’s a brand-new sensor in the iPhone 12 Pro, and it’s a big clue about the future of technology.

Already just 1 month after iPhone 12 launch, the phone accounts for 5% of new uploads to Sketchfab, the largest global platform for immersive and interactive 3D models. It’s much faster, high-quality especially at room-scale, but not the best at small-scale objects.

In this episode of TechFirst with John Koetsier, we chat with Alban Denoyal, the CEO of Sketchfab about why.

And about the implications of 3D scanning: where it’s used, what it can do, and what it’s changing in augmented reality, mixed reality, VR, and more.

Here’s the post on Forbes …

Scroll down for full video, audio, and a transcript of our conversation …

Listen: iPhone 12 Pro’s lidar

Watch: 100X faster 3D scanning with the iPhone 12 Pro

(Subscribe to my YouTube channel so you’ll get notified when I go live with future guests, or see the videos later.)

Read: iPhone 12 speeds up scanning 100X over conventional photogrammetry

(This transcript has been lightly edited for length and clarity.)

 John Koetsier: What do iPhone 12 Pro, Google Poly, Sketchfab, and 3D scanning all have in common? Welcome to TechFirst with John Koetsier.

There’s a brand new sensor in the iPhone 12 Pro, and it’s a big clue about the future of technology, as well as the future of augmented, mixed reality, maybe even VR. It’s also vastly accelerating the 3D capture of almost everything. To get the scoop and get a little deeper, we’re chatting with Alban Denoyal, CEO and co-founder of Sketchfab, which is the largest global platform for immersive and interactive 3D models.

Welcome, Alban! 

Alban Denoyal: Thanks, John.

John Koetsier: Real pleasure to have you here. We’ve been working on this for —  I’m sure we’ve been connected for years, and there’s just been multiple opportunities for us to connect and have a conversation, do a story or whatever — it never worked out for whatever reason, busy-ness all that stuff. But we’re doing it now, and you told me something amazing last week. You said there’s over 3 million 3D models in Sketchfab’s library from hundreds of thousands of creators, but in the month, like one month since the iPhone 12 Pro launched, lidar scans already account for about 5% of them. Why is that?  

Alban Denoyal:  So it accounts for about like 5% of the daily upload, and so that’s a significant portion.

And I think those are two reasons: one is that lidar in the iPhone makes 3D capture significantly easier and faster, and there are a lot of people getting this new phone.

And the second reason is that Sketchfab has partnered with most of the capture apps available for the iPhone so that you can publish directly from those apps to Sketchfab, making us kind of the default platform to share this type of content. 

John Koetsier: So, the lidar sensor in the iPhone 12 Pro gives you good quality, right? It has range up to I think about five meters or something like that, right, about 15 feet or so. And it’s super easy to do, so more people are creating 3D models of just about everything? 

Alban Denoyal: Yes it’s … in terms of quality, it depends what your expectations are. I’ve been testing a lot of those techs, and so [the] last sort of mainstream iteration of it was Tango on the Android platform which was released four or five years ago.

And so what the iPhone is offering is a more, I would say, more robust tech — especially on the texture side. So it’s able to capture images, so textures that make images mappings of the geometry that are much crisper. And now I’m going back to the same places I have scanned like four or five years ago and scanning them again, and comparing … apple to apple … and that’s pretty cool.

It’s not really ideal for small things. Like it’s really, it’s lasers and it’s great for like rooms; it’s great for murals in the streets. Typically, if you try to scan a shoe, it’s going to struggle. I think it’s a matter of the apps improving, sensor improving, and also Apple exposing more raw data from what it captures. 

John Koetsier: So, I want to get into all the implications of this. I want to talk about Google Poly, as well, which shut down yesterday … and a lot more. But first, maybe give us the quick intro to Sketchfab: what you do, and how long you’ve existed, and a little bit of the scale. 

Alban Denoyal: So, Sketchfab is a platform to publish and find 3D models. And we really have those two components. One which is a utility, so we built the first web-based 3D player on the internet back in 2012 as an easy solution for content creators to publish, share, embed, host, display 3D files on the internet.

And as a result of this — well, from this utility, we grew a community around the utility which published a lot of content, resulting in a growing library of content. And so today we’re also kind of the go-to place for people to find 3D files that they can use in a variety of use cases.

And so today we have, we just passed 4 million users on the platform. We have about 10 million unique visitors a month, and we have like 1 million downloads of free assets a month. 

John Koetsier: Very, very good. You need to update your website because it said 3 million just yesterday. So maybe … that’s the latest data.

Talk about where your models are used. So people are creating 3D models of things in their everyday life. I’ve seen you share it on LinkedIn, you know, of an old shoe, a fire hydrant, a room, a building, a setting, all this stuff. People are creating 3D models of all this stuff, uploading it to Sketchfab.

How does it get used? Who uses it? Where do they use it? Why do they use it? 

Alban Denoyal: So it’s really extremely diverse and I’m always blown away by discovering new ways. So some of the content is free to download and some is paid.

And so typically I sell some of my stuff on my Sketchfab store, and whenever I sell a model, I always email the buyer to ask him … like first, how did you find this model? Because I publish some niche stuff, you know — not niche, but like 3D models of sneakers or chocolate croissant — how did you find it? And what are you going to use it for?

And the first model that I sold was a 3D scan of a chocolate croissant, a ‘pain au chocolat’ in french, selling for $3.99 which is essentially the price of the physical thing. I was like, who on earth would buy a 3D scan for [a] chocolate croissant?

And I did a bit of digging and the guy who bought it was actually building an app, a mobile app for nutrition advice. So the concept is that you look at your plate and it tells you what’s on your plate, and what are the calories and stuff. In order to build his app and teach his app to become smarter, he was doing machine learning on plates and food ingredients. And as a result, he was buying all the 3D food on Sketchfab to essentially teach his algorithms to understand what is the object chocolate croissant, so that the camera of his app could understand that.

But other than that, I think the two main use cases are — it’s more like professional use cases — but one is everything around special effects and making movies and videos. To think … like Avatar is a movie, like more and more movies are using 3D assets as backdrops and just to save money on physical stuff, and I think this trend is only going to accelerate. It is not just movies, but ads and, yeah … a long tail of making videos.

And 3D really gives super powers to a video maker and lets you create things that are not possible to create in reality.

And then the other big segment is video games. Video games is a 3D world essentially, and you need 3D assets to populate this 3D world — which can be cars and guns, but also chairs, and trees, and rocks, and hydrants, and sneakers — why not? Like if you’re building a soccer game, your soccer players are going to have shoes and you need to get those shoes somewhere.

And then I mean, yeah, a very cool example on the video side actually was one of the latest music videos of Post Malone — I think it’s called “Circles,” and he’s dressed as a Knight with shiny armor and he has an ancient shield. And the shield is actually coming — is a 3D file from Sketchfab, and so it was mapped digitally, and so for this kind coming from a museum in Europe — I forget the name, but yeah, it’s mind blowing to think that this type of asset sourced by the community is ending in a clip by Post Malone.

It’s pretty cool.

John Koetsier: Really, really interesting and you can totally see how the 3D models work well, right?

Especially for something that’s trying to understand via machine learning what an object is, because when you take a picture of your plate, the croissant is at this angle, or at that angle, or end-on, or whatever … and if he has the full 3D model, then he can train his machine learning algorithm to recognize it from any angle, from any side.

That’s really, really interesting. Now, you mentioned that you sell some yourself and that others sell the objects on the platform. Is there anybody making a living doing this? How much can you make doing this? Is it just a little bit of pocket change? Give us some examples. 

Alban Denoyal: So our store layer component is fairly new. We launched it beta in 2018, out of beta in 2019, so it’s essentially like two years old.

And so it’s, I don’t think we have people fully making a living off of it, but we have a bunch of sellers who make north of $1,000 a month just on selling assets. 

And those guys are often selling on various platforms. So I do know some of them are making a living off selling 3D assets in general, as sometimes they also make custom work. We also get a lot of artists commissioned, like people find them on Sketchfab and commission them.

So typically one of our top car designers on Sketchfab is called Karol, he lives in Poland, got hired by Tesla to commission 3D versions of all the new Tesla cars.

John Koetsier: Wow!

Alban Denoyal: So, yeah, he’s essentially making a living off that, thanks to being found through Sketchfab, because there’s a big follower base on Sketchfab.

John Koetsier: Wow. Post to Sketchfab, get a job with Tesla. Okay, I got it … works every time, guaranteed. No worries. [Laughter] Let’s talk about Google Poly, because there was some interesting news that you shared yesterday. Google Poly was a place where people could upload and share 3D models. Google shut it down. Thoughts on why they did that? What’s going on here? 

 Alban Denoyal: Well, it’s a long story, but I think when they launched it in 2017, four years ago, they launched it under kind of the general Google mission, which is to organize our world’s information. So they announced it as their approach to organizing the world’s 3D information. And I think they shut it down because it was not achieving that vision in terms of scale. And so typically, if you search for Cybertruck on Poly, you’re going to get two or three results. If you search for Cybertruck on Sketchfab, you’re going to get hundreds of results. And so—

John Koetsier: That is impressive then. I mean, you know, it’s not every day that you go up against one of the ten largest corporations on the planet and you out-compete them. That’s pretty impressive. I noticed that you also just released a way for people to migrate their 3D objects over to Sketchfab from Google Poly, is that correct? 

Alban Denoyal: Uh, not — I mean, you can download from Poly and upload to Sketchfab, so we haven’t built an actual way yet. That being said, a lot of uploads on Poly are coming from Tilt Brush. 

One of the [reasons for] building Poly was as a sharing platform directly from Tilt Brush. And we actually partnered with Tilt Brush … it was maybe six months or nine months ago, so we are also integrated as a sharing feature in Tilt Brush. So you can also publish from Tilt Brush to Sketchfab. We’re thinking of ways to facilitate the transfer for people, but generally speaking, I mean, it’s cordless like YouTube, there is an upload button and you can re-upload your content here and…

John Koetsier: Yes. Yes. Well, that’s good. I love Tilt Brush. I’m connected with a couple of people who make amazing art with Tilt Brush, so I’d love to — that’d be really, really cool. I’d love to see that in Sketchfab, even more than it is right now.

Let’s get back to the iPhone 12 Pro, because it has quickly become one of the leading ways to get 3D models into your platform. And you mentioned larger scale, not the smaller scale stuff. So there’s probably better ways for some of the smaller scale stuff right now, but what’s that unlocking? What is the lidar in iPhone 12 unlocking? Is there just a new surge in community interest? 

 Alban Denoyal: Well, it’s bringing 3D capture to the pockets of everyone.

For me, it’s unlocking mainstream 3D content creation.

It’s unlocking — you should look at the evolution of capture. We started with painting in caves, and then we got photography, and then we got video— 

John Koetsier: Right away. Immediately after. [Laughter]

Alban Denoyal: After a few steps, but — so painting in caves, and photography, and then video. And at each of the steps the goal is always the same: replicate the real world and what we see in front of us. We live in a 3D world, and so 3D capture is a normal, natural evolution of capture if you follow this trend. And yeah, I think, so looking at the capability to capture what’s around us in its true form, which is 3D, and making this available to normal people, not just people who’ve learned like 3D programs and so on.

And so, this essentially moves our potential user base from maybe 50 million 3D professionals to … two billion people with smartphones. It’s, for us it’s pretty major. 

John Koetsier: Wow. What’s the quality difference? So you mentioned that the iPhone 12 Pro lidar works well in room-scale. It also works, obviously, for smaller objects, but it wasn’t quite as high quality as some other options. What would you use for the highest quality … small, detail object-type scans? 

Alban Denoyal: So typically what’s mostly used today is photogrammetry, which is the concept of taking a lot of pictures and stitching them together as a 3D model. Typically I scan a lot of shoes, and I like to have very nice scans of my shoes and then I can sell them. And so, for a nice scan I would take maybe 300 photos. So it’s like click, click, click, click—

John Koetsier: Wow.

Alban Denoyal: We go around the shoe at many angles. Some pros on Sketchfab would take 1,000 photos or even 2,000 photos. And so, the time to take all those photos and then you process them in the photogrammetry software — so the more pictures and the higher-res the pictures, the longer processing time. And then you get a point cloud and you turn it into a mesh until you get a 3D file.

And so what’s major with the lidar is that this process is transformed from several hours to just a few seconds. 

John Koetsier: Wow. 

Alban Denoyal: The output is not the same, obviously, but it’s kind of a trade-off: quality versus speed, and it depends what’s your need. And you can also combine both and have a quick view one with lidar and then rework it into something that looks much better. 

John Koetsier: So that’s what, 100X increase in speed? 

Alban Denoyal: Yeah, I would say, yeah typically 100X, I mean … yeah.

Typically scanning a room, like I scanned the office, it took three minutes to go around the room with my iPhone and 60 seconds to process — like 4 minutes total. If I had done this with photogrammetry, it would have been at least 500 pictures. So maybe like 10 minutes to take the pictures and then several hours of processing. So, more than 100X. 

John Koetsier: Wow. Really, really impressive. Do you see that quality coming up? I mean, that’s the natural progression, obviously iPhone 13 or 14 or whatever. And do you see that there’s potentially a time when you can get the same quality with the lidar-based scanning as with the other method?

Alban Denoyal: Yeah, definitely. I think we’ll see more and more mixed methods combining the lidar and more old school approaches. I think — so a 3D model is a mix of geometry and texture, and we need improvement on both ends. I think on the texture side, it’s already doing a pretty fabulous job and so we need to improve the geometry side.

And then the improvements need to happen at several levels in the app.

The Apple level, the hardware level, and the software level, and then the apps being built on top of the iOS capabilities. They already keep improving. Like the first iteration came with the iPad Pro, which already had a lidar. I don’t know when it was released, maybe three months ago. So the first generation of apps came then and those guys typically release a new version every week or every other week, and it’s already constantly improving. It’s fascinating. 

John Koetsier: Really cool. So, I want to get a little bit speculative here. I want to talk a little bit about the future. We know augmented reality is here in a lot of ways. We know that mixed reality is perhaps a little harder to achieve, but is here and is coming in some ways.

What are we laying the foundation for here? We’ve got the phones. We think the smart glasses are coming. What are we laying the foundation for here for VR, AR, and MR?

Alban Denoyal: Well the lidar has two types of implication, because one of them is more on the AR positioning side. So it makes AR positioning easier, better, faster … doing better things like occlusion, so that you can hide things digitally. It makes it easier to position, to lock virtual objects in a specific place, because you know where they sit in the actual world through a digital map of the actual world.

And then on the capture side, it unlocks user-generated 3D content, which I think has two implications.

One is that people will be able to capture things that matter to them. So like, their friends and their home and their food. And so it’s similar trends as you saw on Instagram, like you typically take selfies, and your plate at the restaurant, or your dog. And I think the ability to capture things that matter to you is a key part of the equation for AR, where you need both the content and a way to consume it. And so we’re progressing on the way to consume AR content.

And the key question is: what content are we going to look at? And so, yes, there are games and maps and activities and all that. I think if you can revisit memories, like I have a 4D volumetric video of the first steps of my son— 

John Koetsier: Wow.

Alban Denoyal: And it makes it a much more — makes AR valuable for me much more, because I have this piece of content so I can revisit it in AR, which is kind of mind-blowing. And so that’s one thing, and then the fact that you’ll have a 3D version of everything that’s more on the library side.

I keep thinking about this a bit like Spotify, how you have [a] playlist of music, but I can see a future where you have a similar concept for your virtual world or living room.

And so every Monday you are at home, but your home has a virtual layer on top of it, and you’ll have a virtual fig tree and then virtual cats. And every Monday the fig tree becomes a palm tree and the cat becomes a dog. And you have this concept of … I dunno, and then you are following Tesla on Sketchfab, and Tesla has released a new version of their car. And as you’re following them, you get the new Tesla as an AR object showing up on your desk the day it’s released. And yeah, this is the type of use cases I have in mind. 

John Koetsier: Very interesting. The future of Instagram might be a place where you can actually quote/unquote “visit” where an influencer is showing you an amazing sunset on a beach in Hawaii or something like that. Or … a creator, an inventor is showcasing a piece of art and you could actually participate in those in a much more immersive way. Very interesting. I wonder when that will come. Perhaps Sketchfab will be a part of that future.

I want to ask a couple more questions here, because one of the focus points that I have for TechFirst is that it’s about tech that’s changing the world and innovators who are shaping the future. What do you see as the most important implications of this technology? How will it shape our world?

Alban Denoyal: Well I think the … everything around people is super important.

I have a 3D scan of my grandma … and she passed away two years ago. And I have a digital portrait of her in a form and shape that is much closer to what she was in real life. And there’s a famous example of the Tupac hologram. And so this idea of being able to review, to view people as a hologram, which is this fantasy of movies, is pretty mind blowing.

And not only people of the past, but it’s going to have implications in, yeah, social interactions. Kind of the holy grail of Facebook is how to replicate social interactions virtually. And so right now through VR and tomorrow through AR, I think that that’s a key part. So everything around people.

And then a second key part, I think is really around our shared heritage. That’s one of the biggest segments on Sketchfab, is everything around cultural artifacts. We have over a thousand museums publishing on the platform, plus all the user generated content. And so typically when I go to a museum, I scan stuff and publish it on Sketchfab.

And we’re becoming the largest museum on the internet and we have a digital twin of a lot of pieces, works of art. Some of it is disappearing. One recent example was when the National Museum of Rio in Brazil was destroyed by a fire one year ago, and we added a bunch of its collections on Sketchfab. And this is just like — we have like the Rosetta Stone on Sketchfab and, yeah, this is super important to be able to preserve archives of things in their true shape, not just images or videos. 

John Koetsier: Very, very meaningful. Wow, that’s amazing that you had scans of that museum before it was destroyed in the fire. I remember that, and that was a horrific occasion because of course it had treasures in there, right? Irreplaceable treasures. But at least there’s some way to revisit those.

And it’s interesting what you mentioned about the 3D technology and not just people who have passed away, but people who are still among us. I was recently interviewing one of the co-founders of Reface.ai, which is one of the apps of the year according to Google, and has done some amazing things. And they have some plans that sound like they would mesh very nicely for influencers and stars with what you’re doing, except what you’re doing would be in multiple dimensions. Well, Alban, I want to thank you so much for your time. It’s been a real pleasure. 

Alban Denoyal: Thank you, John.

John Koetsier: Excellent. For everybody else, thank you for joining us on TechFirst.

My name is John Koetsier. I’m glad you were along on the show. You’ll be able to get a full transcript of this podcast in about a week at JohnKoetsier.com. See the story at Forbes after that, and the full video will be available on my YouTube channel. Thank you for joining.

Until next time … this is John Koetsier.

Subscribe for free

Made it all the way down here? Wow. You’re dedicated 🙂

The TechFirst with John Koetsier podcast is about tech that is changing the world, and innovators who are shaping the future. Guests include former Apple CEO John Scully. The head of Facebook gaming. Amazon’s head of robotics. GitHub’s CTO. Scientists inventing smart contact lenses. Startup entrepreneurs. Google executives. Former Microsoft CTO Nathan Myhrvold. And much, much more.

Subscribe on your podcast platform of choice: