Improved productivity with generative AI
We noticed that web developers want to take advantage of generative AI to increase their productivity and interact with chat bots to learn new technologies or seek answers to their web development questions.
The developers we talked to are either already using AI in their daily workflow, for business or personal projects, or know someone who is.
Code generation
We learned from you that code generation tools, such as Gemini and Copilot, are great for standard unit tests, basic autocomplete (where you know what to write, but just need to write it), and simpler functions that don’t require expansive knowledge of the codebase. These tools tend to be less useful when it comes to writing more complex, algorithmic code and functions that require broad context of a specific project.
More senior developers mentioned a concern around the long-term quality of their codebases, including issues around code duplication and long-term maintainability. Some were concerned that less experienced team members may not detect errors or know how to accurately validate the code produced by generative AI tools.
Developers have also shared that use cases which require specific domain knowledge, such as writing accessible components, aren’t yet accommodated correctly by code generation tools they’ve tried.
Learning with LLMs
We’ve seen many developers using ChatGPT and Gemini to learn software development concepts, like asking a large language model (LLM) to explain how a sorting algorithm works, ramping up on different programming languages, or closing knowledge gaps.
You think LLMs provide a great experience due to the quick interactivity of questions and responses, and because AI won’t criticize the questions being asked, and just provide the answer they need.
Once again, there is a concern related to more junior developers, where a minimal level of knowledge is necessary to catch the instances where the model is hallucinating and generating incorrect responses.
IP protection as a business concern
Many of the developers we spoke with said their companies don’t yet have policies around developers using generative AI to increase their productivity. Usage of generative AI tools is often driven by the developers experimenting with it.
“My company generally misunderstands what AI means, so they’re not creating the right policies.”
However, businesses that do have a policy tend to discourage usage, due to concerns around leaking the company’s intellectual properties (IP) to third parties. There are cases where such policies have been changed, after engaging directly with the companies behind these tools to understand how the data is used and potential risks.
With enterprise accounts and partnerships which are dedicated to ensure data protection, businesses are more likely to encourage developer usage.
Generative AI for user facing product features
On the product side, it wasn’t a surprise to us that, when we prompted our conversation with the term “AI / ML,” the response was often focused on generative AI. Developers are curious on how generative AI can be used to improve the experience for their users, but are unsure about what those experiences look like and what are the tools available to deliver them in production.
For those developers that have built or are building generative AI features into their products, using generative AI to answer user questions, with chatbots or one-off interfaces, is the most common use-case.
Output quality is the top concern we heard from you. In particular, developers hope to ensure the responses are accurate and aim to prevent the LLM from generating content unrelated to the intended goal. This is particularly true when the LLM’s output is directly user-facing, such as a chatbot.
“Demos with AI are wild. Everytime I demo my project, the output is completely different.”
You’re putting a lot of effort into creating test suites to validate generative AI outputs for a wide variety of prompts, but there’s no clear and established way to test or monitor the responses. Most of the evaluation work is manual. Many developers are new to handling non-deterministic outputs. As a community, we haven’t yet built systems which work well with them.
The cost of running a generative AI model is also an important concern, and developers are carefully evaluating the cost against the benefits for their users.
Standard versus customized models
Most commonly, the developers we spoke to tended to rely on ready-made models and APIs. This optimizes the time-to-market and the use of engineering time and knowledge, which were limited.
“I want to stay in the web development space. I don’t want to become an ML engineer.”
While developers are aware of and see potential value in advanced techniques, such as Retrieve Augment-Generate (RAG) and fine-tuning, you’d rather focus on the web development aspect of their work. Ultimately, you prefer to use the default tools or rely on other teams to produce optimized models for their use cases.