Beware: Real risks of using AI as a software dev team
The words that follow are words of caution learned first-hand. They’re for engineering leaders, managers, and CTOs. They’re for working software engineers who know (or suspect) there are major drawbacks to leaning hard into building software with LLM-powered tools. They’re for me, as a way of admitting my failures as an engineering leader so that others might avoid the same mistakes.
Beware building features the team doesn’t understand. When things go wrong and the chatbot can't figure it out, it’s a bad situation.
Beware LLM-generated writing that's used in human communication. It's always excessively verbose and typically useless. Nothing good comes from AI-generated internal communications.
Beware feature bloat. People want focused, reliable, fast, stable, and functional software.
Beware junior engineers being isolated by replacing collaborative human conversations with the chatbot. Onboarding and mentorship are as important as ever. Chatbots aren’t suitable substitutes.
Beware new hires being isolated and told to ask the chatbot.
Beware junior engineers not actually understanding the code being generated. It's often said that an engineer's job is to know when the AI-generated code is wrong, but if a whole generation of developers don't understand the code, how will they know when it's wrong?
Beware the rising costs as new models get created, usage thresholds changing without notice, and the major providers beginning to think more about profit after years of burning cash at unprecedented rates.
Beware relying too heavily on AI code review for security holes and leaks. No amount of AI code review will replace diligent, experienced developers who understand the entire architecture and moving pieces and what could go wrong.
Beware false-positives from the AI code review. The generated messages will be confident. But they are, at least from my experience, wrong about half of the time.
Beware of code not being reused that should be when LLMs generate code. They have no real memory or satisfaction of extracting a function to clean up some code three months ago and won’t remember to use that new function. They’ll often times duplicate it, introducing new code paths where there should only be one. Crafting the perfect Markdown files won't stop this from happening.
Beware automation blindness, the concept of it being very difficult to spot real issues amongst thousands upon thousands of LLM-generated lines of code. This is not the failing of an individual but a human limitation.
Beware the mental health of your team as they use AI for generating code more and more. Understand that leaning heavily into these tools can cause apathy, especially in those who previously took pride in the craft of software development. Check in with your team and talk with them about how they’re feeling and what they’re experiencing. Ask thoughtful, caring questions.
Beware vibe-coded tools built internally in other departments that your team may now be responsible for. They might be coded in all different kinds of tech stacks and programming languages. They might contain sensitive data or have security holes.
Beware the obsessive intent of coding harnesses hooked up LLMs to read .env files and other important secret keys no matter how many Markdown files and JSON configurations explicitly forbid this.
Beware LLMs taking bizarre shortcuts and not actually building out the full feature.
Beware dependency bloat and pulling in packages that aren't needed.
Beware tracking and caring about false metrics that aren't actually meaningful to the business/organization/the team/the world. Glorifying token maxing is bad management.
Beware thinking about software development as a factory. More code and more features does not always translate to more money. Remember that software development is knowledge work and developers are skilled, creative problem solvers tasked with understanding, building, extending, and fixing non-trivial systems.
Beware replacing actual human collaboration with sending AI-generated prototypes back and forth.
Beware giving LLMs hooked up to coding harnesses access to critical services and systems and infrastructure. They have and will take unpredictable, destructive actions.
Beware non-engineers asking the chatbot without the full picture questions about implementation and it, at times, being wrong.
Beware trusting AI too much at every step of the pipeline.
Beware full-auto, giving the coding harness and LLM full access to running shell commands on your computer. As Cal Newport says, it's like strapping a weed wacker to a dog to mow the lawn—chaos will ensue.
Beware the excessive unit tests LLMs generate and almost always useless e2e tests that look "good" but are, in fact, not exercising the critical paths.
Beware the shifting expectations of output speed that inevitably happens when there’s a chatbot that vomits out lines of code very quickly while being backed up by billions of dollars of marketing hype.
Beware of "one more prompt" and how using LLM-powered coding tools can lead to overwork and burnout.
Beware lines of code expanding rapidly. It all adds up over time, slowing everything down, bit by bit.
Beware of changing budgets from the Finance team. Unlimited spending today may turn into a limited budget tomorrow, with usage limits being hit before the month is over.
Beware vibe coding functionality that you have to maintain for years into the future that could otherwise be solved by an existing tool or integration. There's still a cost to maintaining code.
Beware the codebase and tech stack fracturing, sometimes caused by just one person wanting to go in a different direction than the rest of the team. Less oversight and increased speed can cause havoc.
Beware the desire to stop doing human code review in order to accelerate merging pull requests. Sharing knowledge and context around what's changing and why with the team is valuable and helps people feel connected.
Beware leaning heavily into a technology that is not in its final form, with regularly changing costs, models, and budgets. The way these tools are used now very likely won't be the same in a few months or couple years.
Beware the AI hype from AI companies, boosters, and other unsavory characters. Deeply consider the harms, measure the impact, talk with your team. Make thoughtful, sustainable decisions.
Beware the mental health risks of using chatbots, as their impact and risk is only just beginning to be reckoned with.
Beware abandoning core principals just to go faster.
Beware deskilling engineers who are expected to be responsible for the code that's being generated.
Beware the erosion of team culture and morale.
So you've bewared... well, what now? That's your job as an engineering leader to figure out how to effectively integrate a technology into your team in a way that's useful, sustainable, and non-disruptive as possible. Some of the harms from these risks may be able to be minimized or mitigated. Or maybe the business is okay with these risks. Maybe it means scaling back AI usage, maybe it means only using it for specific tasks, maybe it means not using it at all. You could have a meeting with your team to check in on their experiences. You could draft internal policies on responsible usage and set clear expectations. You can explicitly make it okay for engineers to choose not to use AI so that they're not worried there's a Sword of Damocles hovering overhead because of their rejection of the technology.
I don’t know your goals, business, pressures, etc. But I do know that there are a whole lot of risks associated with using AI for software development, not even touching upon the broader ethical and environmental concerns.
It's possible to not use it, of course. Just like software was coded for the past few decades. It might feel like you have to use it or something bad will happen. But the future is not decided. It's actually quite uncertain. I'm convinced through my own experiences leading a team that the way the industry is going about this is bad for the team, bad for the people using the software, bad for the software, bad for the business, bad for the industry, bad for society, and bad for the planet.
Further Reading
Show your support by buying me a coffee or joining my YouTube channel. Thank you for reading!