AI-generated code: what 4000 developers do

When will AI replace developers? Or is it an if?

In this TechFirst we dive into a survey focused on how 4,000 software developers use AI to generate, test, and check code.

Justice Erolin, the CTO of BairesDev, recently surveyed over 4,000 developers globally. The goal: exploreing how AI tools like ChatGPT and GitHub Copilot are being utilized in software development. We chat about how these tools are employed for code generation, scaffolding, and testing, and we discuss the potential over-reliance on AI and its impact on entry-level engineers.

We also highlight key findings from the survey, including surprising trends in AI tool preferences and the perceived productivity impacts.

(Subscribe to my YouTube channel)

Subscribe to the audio podcast

 

Transcript: 4000 developers on AI generating code

This is AI-generated; it is not perfect.

John Koetsier (00:01.87)
Can well AI replace developers? Or is the right word if? Hello and welcome to Tech First. My name is John Kutzer. We all know that AI is pretty amazing, right? And many of us daily create miracles with it. But can AI replace software developers? It’s a good question. Who better to ask than developers themselves, many of whom, maybe most of whom, are using AI right now already today?

According to a new survey by Bears Dev, which has 4 ,000 plus developers in countries all over the world. We’re gonna learn all about that. To learn more, we’re chatting with the CTO, Justice Erwin. Welcome, Justice.

Justice (00:41.213)
Thank you for having me, John.

John Koetsier (00:43.06)
Hey, kind of pumped to have the discussion. I use AI every day. I’m assuming you do as well. How are developers using AI?

Justice (00:52.873)
Great question. So when we’re looking at how our engineers, specifically the Bears 7 engineers, the top reason why AI is being used internally is for code generation or another term scaffolding. This allows an engineer to start a project very, very quickly.

John Koetsier (01:13.019)
What’s that mean? Does it mean that they’re using it for the entire project? Does it mean that, I’m getting started, here’s some code I can build on that? What’s that process look like once you start?

Justice (01:26.715)
So when we’re looking at how an engineer uses it within their specific software development environment, whether it’s through their IDE or even through another piece of tool, that code generation piece is at the beginning of their process. While they start off with writing just a little bit, tools like Copilot, whether it’s GitHub or Microsoft, will come in and backfill or future fill the rest of their code piece.

From there, they can make corrections, can make edits, or they can move on to the next piece of code that they need to write. But that generation piece can happen anywhere within the software development lifecycle. So for example, if you are an automated tester and you want to write unit tests, you can highlight a piece of code and write a prompt, say, write a unit test for this specific function.

John Koetsier (02:20.75)
Mm -hmm. It’s interesting because I saw the results of your survey and saw the most popular AI tools, right? And I’m just looking at chat GPT is the most popular with 54%. We’ve got 30 % of developers using GitHub Copilot, 17 % Microsoft Copilot, and 12 % using Gemini.

It’s interesting because if you’re using Chat GPT, that’s not necessarily integrated into your IDE, is it?

Justice (02:51.881)
That’s absolutely correct. That’s surprises me the most. And looking into this further, I mean, in my expectation, I would have expected that an engineer would use either GitHub Copilot or Microsoft Copilot as their main tool. But we’re seeing more than half use ChatGPT as their main tool. So what I found out was that it could be a licensing issue. On our side, we just need to communicate a little bit better.

and say, you have this license for free because we’re already paying for it. Please utilize them at your disposal. But for the most part, think many engineers know the open source tool, not call it open source tools, but know the free tools out there that they can utilize. And I think that’s why chat GBT is the main leader

John Koetsier (03:41.876)
It’s also probably a bit of a habit, right? Like when Google came out, we got in the habit of using Google. mean, now that I have ChatGPT on my phone, I’m like, that’s interesting. What is going on there? I’ll snap a picture as I’m passing something. I was in Toronto the other day and I saw these crumbling concrete pillars for an elevated highway or freeway. was like, what causes that? And ChatGPT had all the answers, right? And so maybe you’re just used to it and every everything. And so you use it here as well.

Justice (04:08.999)
Yeah, I think that’s a great point. Habit is, I think the number one reason. It’s almost like how engineers use Stack Overflow. They don’t necessarily go in there and use a search bar. They use Google to search for Stack Overflow answers.

John Koetsier (04:23.554)
Exactly. I did see something from GitHub, I want to say, two, three months ago or something like that, where they did some, I don’t know if it was full -on surveying or research or whatever, but they were saying that a lot of developers are using Copilot, GitHub Copilot, similar to how you might pair program in the past. Does that ring a bell?

Justice (04:46.131)
Yeah. So there was also another way of looking at it, like the rubber duck method. Both ways, I would say, are early ancestors of using AI. Well, instead of paying for a second engineer to help you pair program, we now have presented a tool for that. But I think using what’s called legacy methodologies for development is still being utilized today with more modern technologies.

John Koetsier (05:16.342)
I’m not familiar with the rubber duck method. Sounds like I’m having a bath.

Justice (05:21.006)
The rubber duck method is essentially the idea that you have a rubber duck on your desk and you will explain to that rubber duck what your problem is. And the act of just verbalizing the issues that you’re going through, the problems and the code that you’re reading allows you to solve the problem on your

John Koetsier (05:41.054)
OK, OK, cool. I like it. I like it a lot. Where is AI good at cogeneration?

Justice (05:49.871)
Again, on the scaffolding side, just making a quick framework of developing out the baseline. What we’re seeing is that most of our respondents are coming in and editing it, whether they’re minor issues or a little bit more complicated issues. But giving it a starting point, giving an engineer a starting point allows them to save time in the future. So for example, if a piece of code took a normal engineer two hours to write,

and now all of a sudden it’s within seconds because of GenAI or Co -Pilot, they’re probably spending anywhere between 15 to 30 minutes to edit and correct any potential mistakes or potential, let’s say, integrations that they may need. That allows them to spend the next hour and a half or so to work on more complicated issues of, let’s say, future integrations, of edge case testing, et cetera.

John Koetsier (06:44.744)
It’s really interesting to think about it right because most of the time, if you’re building some kind of application, whether it’s SaaS, whether it’s an app, whether it’s back office software, desktop software, even almost everything that you’re doing, mean, like people are solving problems in code that have been solved, right? I mean, obviously this is why we have functions, we have modules you can bring in and everything like that. But even so,

So much of what you’re actually writing in code has been done before. So this is really an interesting way of using AI to kind of handle maybe the more rudimentary or common aspects, also giving you insights on tough stuff. But then, you know, the stuff that really requires specific knowledge about a company and processes or how this will be implemented, then you can do that. Does that make sense?

Justice (07:36.649)
Right. Like there’s only so many ways you build a CRUD app. And that’s what I would say 95 % of every application is. It’s a level of complexity above a CRUD app. But an engineer is not really there to create functions that do that. They’re there to help translate the business requirements or the business logic to apply CRUD to that specific need.

John Koetsier (07:58.124)
Mm -hmm.

John Koetsier (08:07.598)
Interesting. That just reminded me of the guy in office space who got fired. He says, I talked to developers. They’re not good with people. Exactly. Exactly. There are some who don’t want to use AI. Some developers don’t want to use it. Why are they not using it? Are they just old fuddy duddies? Are they all stars and the AI could never create code as good as they create? What are the reasons?

Justice (08:16.518)
Yeah, as long as you send me what is it the TPS reports and you’re good to

Justice (08:36.873)
Um, I, what if the respondents said, or sorry, about 40 % of engineers say that gen AI does not, has not freed up their time. Um, I think there’s a few reasons for that one. haven’t yet. They haven’t yet learned how to use tool, right? Any tool requires some level of aptitude to, use it accordingly. Um, just because I have a hammer doesn’t make me a carpenter. So we need to make sure that we’re educated enough to,

to utilize the tool correctly. The second thing I think is more of a cultural issue. Engineers like to write new code. They like to build something on their own. Back to the carpenter analogy, just because you’re a carpenter doesn’t mean you love IKEA. But there might be some great carpenters that use IKEA for specific needs. So I think it’s a matter of understanding when GEN .AI

John Koetsier (09:18.658)
Mm -hmm.

Justice (09:34.705)
makes a difference in your specific workflow.

John Koetsier (09:39.704)
Do you foresee it being a problem if some developers say, I’m not using AI, it’s got some errors, it doesn’t work, it’s not the perfect fit for me or something like that? Do you see that being a problem? How much more productive are you if you do use it in code development?

Justice (09:57.289)
So I think when we’re looking at the overall results, whether you’re looking at product managers and engineering managers, CTOs like myself, we want to see the end result. If you have 40 working hours in a week, or let’s call it 40 productive hours in a week, that’s what our expectation is. Now, if you’re not using GenAI, then the question will be, are those 40 hours as productive as they could be?

So we want to make sure that from an engineer’s perspective, they have every tool at their disposal to be as productive as they can. Now, if they’re running at 100 % productivity, regardless of the tools that they use, and they’re a rock star, I probably wouldn’t be the guy that says, no, you have to use this. But let’s be honest, not everyone’s a rock star.

John Koetsier (10:45.869)
Mm -hmm.

John Koetsier (10:52.974)
Not everyone’s a rock star and even you are a rock star. Do you want to rewrite the same function 500 times in your career or do you want to just get it

Justice (11:02.417)
Right. I’ve never met a rock star that looks back at their previous code and says, yeah, that was a great piece of code. They always look back at it and say, you know, that could have been improved. And so many of the rock stars that I know actually want to find ways to improve their craft.

John Koetsier (11:18.074)
It’s pretty interesting because I do a lot of writing and I find generous AI very helpful not to write for me, but a lot of research, sometimes some background, other things like that. I find a time saver right there. In the actual writing, I want my personality to show. I want some humor. I want to bring up the points I think are important. I want to sprinkle in the quotes that really, you know, dress up.

a point or an idea and I find AI doesn’t really do that. Is it somewhat similar in code where maybe it’s less so, maybe there’s less personality that shows up in code.

Justice (12:01.123)
personally, think does show up in code in a certain context, mainly around comments and how and styling. but when you get, if you strip all of that away, there might be, I would say there’s only a handful of times where I can see someone’s personality coming through in a piece of code. that provides a little bit of a danger for, for some of our engineers because it means

to a certain extent, we were commoditized, right? Our outputs are fairly the same. So if that’s all true, then we need to make sure that we have the right tools at our disposal, that we’re utilizing as much as we can to get the right output.

John Koetsier (12:45.578)
It’s pretty obvious, given what we’re seeing in the past few years, frankly, and even in recent history, that it’s critical to write secure code, especially in networked applications, especially in applications that are running in places that are network accessible, right? Does AI help engineers write more secure code?

Justice (13:08.339)
So yes and no. think one, you have to make sure you’re prompting the prompting copilot to give it that specific context. On top of that, you want to make sure that you’re also aware of the potential pitfalls of unsecured code. So for example, I’m pretty sure that if I were to write a prompt right now, I could force copilot to write

a piece of code that’s easily injectable by SQL. It’s a matter of how are we leading these AI tools? Are we providing them the context to write the most secure or the best secure applications?

John Koetsier (14:01.47)
It’s pretty interesting really because if you think about it, mean, it’s amazing. Chatch EPT is as good as it currently is at writing code. I’ve seen that personally. I’ve talked to developers who have used it and been very successful using it. It’s a little more surprising that you can use a generative AI application that’s attached to a development environment.

And it’s not automatically, you know, having some like three laws of AI robotics or something like that, you know, for security, for validating inputs, for checking, for integration, those sorts of things.

Justice (14:41.127)
I think that’s where we’ll get to. I think in the future, or at least in my perspective, organizations can start configuring how their AI, how their within IDE tools behave. So these are the style guides, these are the configurations, this is the context that we want to use. These are the places that you shouldn’t even be touching. If we were to do

and also apply AI to, let’s say, code reviews, automated testing, deployments, and integrate that within the CI -CD pipeline, I think that would be almost a heavenly approach to how development and AI can work together in a larger context.

John Koetsier (15:27.694)
Let’s bring CrowdStrike into this. That’s been obviously a massive thing, right? They’re still getting flights canceled because of that one, Delta. I think it was something like 50 ,000 to 75 ,000 people have had their flights missed. And that’s just the airline industry. There’s many other industries. were doctors’ offices in the UK that couldn’t open and manage their practice software.

And there were global issues here. How can AI help us avoid that sort of scenario?

Justice (16:05.667)
That’s a very heavy question. So part of the problem with CrowdStrike was the launch of a content patch that addressed essentially null memory or memory addresses that were non -existent. Could AI have found it prior to that release? Absolutely. Could a human have done it? Absolutely. So maybe

problem wasn’t a lack of AI, maybe it was a lack of process. I don’t know enough about their internal processes to know what the solve was. But let’s say AI today and CrowdStrike as it happened, could AI solve some of these issues? Not entirely because you still needed a person to manually be at that terminal or at that machine to reboot, to go to safe mode.

to delete that file and then restart. No network application could do that. If so, then Delta would have been up by now and not experience 31 % of cancellations yesterday or Monday for those that are watching the recording. It’s a pretty hard

John Koetsier (17:04.45)
Mm -hmm.

John Koetsier (17:20.698)
Crazy. Yeah, yeah. And yet somehow CrowdStrike was set up to automatically push to millions of servers and computers across the globe. anyways, we’re not talking about that so much, but it is interesting. How do you see this evolving in the future? Code generation, the role of developers, and maybe even how many people can be developers?

Justice (17:47.475)
So there’s probably a problem that I don’t know if it’s being solved yet. The first one is going to be around entry -level engineers. If AI is a great way for senior engineers to write a bunch of code very, very fast, that’s what mostly junior engineers are there for. They’re meant to really come in and start filling in the blanks, and then the senior engineer will come and help massage that for a more production -ready environment.

So if AI is taking over what a junior engineer normally does or an entry level engineer normally does, then what’s the role of an entry level engineer? So I think there’s going to be a problem, one, for our educational system to see how we can up the ante on those entry level engineers. And two, how can we give those entry level engineers the right experience to, one, write prompts to understand

potentially missing pieces that they need to start looking for? And then three, how can they educate themselves better on on potential new tools that are coming

John Koetsier (19:00.064)
It’s an interesting challenge you brought up entry level engineers and there’ll be a temptation among them in particular to just let AI do everything and see how it all kind of works. Does it concern you at all that we might have an over -reliance? As we continue to kind of abstract away

some of the tougher or more menial challenges, not just in coding with generative AI, but in many areas of our society and engineering functions and scientific functions and other things like that. we, we sort of paper over those complexities and we just operate at higher levels, we forget how to do the basic stuff. And will that cause huge issues in the

Justice (19:43.282)
Absolutely.

Absolutely. I think that we ran into a very similar problem with network speeds. When I was starting off as an engineer, the maximum speed that you could utilize was 56k. This is back when modems made a sound when you got online.

John Koetsier (20:03.554)
Wow, you had 56K, how lucky was that? No, I’m just joking.

Justice (20:06.865)
I’m a little bit younger, a little bit younger than you, John. But when we’re looking at how developers behaved back in the day, they wanted to make sure we were optimizing payloads very, very well. But now with broadband and especially 5G on a device, I haven’t really seen engineers talk about their full payloads and the time it takes to download a specific JavaScript file or a

image. It’s almost like it doesn’t really matter. So in the future, I think we’re going to hit that same problem where some of the baseline, some of the baseline scaffolding and understanding of how to set up your development code will be an issue moving later on. And you might have to call in some of the old folks like me to help out.

John Koetsier (20:58.529)
Mm -hmm.

John Koetsier (21:05.742)
Where’s all those mainframe engineers when you need them? Wow. Very cool. Super interesting. Maybe we’ll end with this. Of all the results of the survey you did, I believe about 500 engineers that you surveyed about their use of generative AI, what was the most surprising result?

Justice (21:07.655)
No, exactly.

Justice (21:28.265)
The most pricing result, let me scroll down to where I can see it. It’s roughly, even we mentioned this previously, but it was around the tool that they utilize and that’s going to be chat GPT. So despite it being in despite copilot being integrated within their IDE, chat GPT is still number one. And I think habit, as you said, pushes a lot of that. So I think when we’re looking at

John Koetsier (21:40.216)
Mm. Mm -hmm.

Justice (21:56.041)
what that means for GitHub and for Microsoft, they got to start sort of breaking habits in the future. There’s a lot of mental space that chat GPT is taking over. So we should be cognizant of what that looks like in the future. And then the next thing I was looking at was around the 24 % that said there was no change in their quality of work with the use of Gen .ai.

I wonder if it’s because they’re missing those, they’re not seeing what the difference is or without utilizing the time. I have a personal, obviously a vendetta, but personal mission to find out what their anecdotal research, what their anecdotal experience looks like.

John Koetsier (22:26.456)
Mm -hmm.

John Koetsier (22:45.75)
Yeah, yeah, that is super interesting, right? Like, was your code just perfect before and still is, or are you just not seeing those issues? It is really interesting that ChatGPT is being used there. I mean, the way to solve that and the way to solve, you know, habit and going to other outside products is to make something that’s way better. You got to believe that GitHub and Microsoft, I mean, they know a little bit about development. They have a lot of source code that they can look at. You got to think they could get better.

Justice (22:50.217)
All right.

John Koetsier (23:14.776)
Chachipati has been amazing, but where has it gotten all of its learning and training data from? Probably from GitHub. Exactly. Exactly. Well, anyways, this has been great. Thank you so much for your time. I really do appreciate

Justice (23:23.793)
GitHub public repos exactly.

Justice (23:32.915)
Thank you, John.

TechFirst is about smart matter … drones, AI, robots, and other cutting-edge tech

Made it all the way down here? Wow!

The TechFirst with John Koetsier podcast is about tech that is changing the world, including wearable tech, and innovators who are shaping the future. Guests include former Apple CEO John Scully. The head of Facebook gaming. Amazon’s head of robotics. GitHub’s CTO. Twitter’s chief information security officer, and much more. Scientists inventing smart contact lenses. Startup entrepreneurs. Google executives. Former Microsoft CTO Nathan Myhrvold. And much, much more.