My company has been trying a new model when product folks cut through the red tape of “engineering” and just describe what they want to a powerful LLM pipeline and review the app in a beta env. Sounds perfect, right?
Dear reader, in the couple months this has been going on, these people have caused a dozen high profile SEVs due to extremely poor app performance, networking / kubernetes configuration bugs, bad scaling, observability oversights, supply chain attacks, leaking sensitive information, and cost overruns (on practically every resource they provision).
Some very well-paid people are scrambling to figure out the value that was generated by this pilot program; I’m heating up popcorn rather than holding my breath.
That’s hilarious, idk i think llm could be useful for helping product folks translate their thoughts into actionable items for the devs, but yeah like beyond insane to tell the product people to hop on claude and do it themselves. That’s like a construction company letting the sales team jump in an excavator and start digging!!
llm could be useful for helping product folks translate their thoughts into actionable items
In my experience it makes them give me an essay instead of 10 lines of bullet points and I have you spend an hour asking questions to whittle it down to 10 lines of a bullet points
I was thinking more along the lines of “let the executive tell an LLM to write the shitty prototype version of what they want (which you then rewrite from scratch to not be shit) so you don’t have to decipher their incoherent desires”
even just using it to translate their thoughts into a bullet list isn’t going to work most of the time. LLMs training is, the majority of the time, out of date and stacks change/move faster than an agents data. Plus LLMs favour things that have been posted about on blogs, linkedin posts, etc rather than stuff that is proven and tested.
Like for example it might suggest using a language or library that is insanely out of date or just won’t work but recommends it based purely on say blog posts that have talked a lot about it or it’s potential to do something that it currently can’t do. or like I said it’ll suggest something that’s out of date simply because there’s more documentation online about that specific version than the most up to date version. Or the LLM will report on something that has a “known issue” from like early 2025 and suggests not using it when said issue has been patched routinely since then because the data the LLM has is out of date. the data is constantly out of date.
I guess shouldn’t product people not give a shit what language you write it in? Like that’s the problem product people need to describe what they want while letting the devs do the job of generating the code.
right the product people won’t care or won’t know so the LLM will just spit out whatever. to the product people it’ll look legit and make sense. once it’s handed to a dev in most cases it might as well just be thrown out of the window. so it’s pointless. it’d be easier/faster to just cut out the middle man, the AI, and go directly to the dev and have the product people tell them exactly what they want/need.
I mean honestly in that case im already having my meetings audio recorded and transcribed and summarized by ai because these people at work are fucking idiots and find a way to spend two hours talking without addressing anything actionable.
My company has been trying a new model when product folks cut through the red tape of “engineering” and just describe what they want to a powerful LLM pipeline and review the app in a beta env. Sounds perfect, right?
Dear reader, in the couple months this has been going on, these people have caused a dozen high profile SEVs due to extremely poor app performance, networking / kubernetes configuration bugs, bad scaling, observability oversights, supply chain attacks, leaking sensitive information, and cost overruns (on practically every resource they provision).
Some very well-paid people are scrambling to figure out the value that was generated by this pilot program; I’m heating up popcorn rather than holding my breath.
That’s hilarious, idk i think llm could be useful for helping product folks translate their thoughts into actionable items for the devs, but yeah like beyond insane to tell the product people to hop on claude and do it themselves. That’s like a construction company letting the sales team jump in an excavator and start digging!!
“it’s fine, he doesn’t need to know how to use the controls. We just installed a voice command system into the excavator.”
The system is wrong 25% of the time and will do the opposite of what you want.
In my experience it makes them give me an essay instead of 10 lines of bullet points and I have you spend an hour asking questions to whittle it down to 10 lines of a bullet points
I was thinking more along the lines of “let the executive tell an LLM to write the shitty prototype version of what they want (which you then rewrite from scratch to not be shit) so you don’t have to decipher their incoherent desires”
even just using it to translate their thoughts into a bullet list isn’t going to work most of the time. LLMs training is, the majority of the time, out of date and stacks change/move faster than an agents data. Plus LLMs favour things that have been posted about on blogs, linkedin posts, etc rather than stuff that is proven and tested.
Like for example it might suggest using a language or library that is insanely out of date or just won’t work but recommends it based purely on say blog posts that have talked a lot about it or it’s potential to do something that it currently can’t do. or like I said it’ll suggest something that’s out of date simply because there’s more documentation online about that specific version than the most up to date version. Or the LLM will report on something that has a “known issue” from like early 2025 and suggests not using it when said issue has been patched routinely since then because the data the LLM has is out of date. the data is constantly out of date.
I guess shouldn’t product people not give a shit what language you write it in? Like that’s the problem product people need to describe what they want while letting the devs do the job of generating the code.
right the product people won’t care or won’t know so the LLM will just spit out whatever. to the product people it’ll look legit and make sense. once it’s handed to a dev in most cases it might as well just be thrown out of the window. so it’s pointless. it’d be easier/faster to just cut out the middle man, the AI, and go directly to the dev and have the product people tell them exactly what they want/need.
I mean honestly in that case im already having my meetings audio recorded and transcribed and summarized by ai because these people at work are fucking idiots and find a way to spend two hours talking without addressing anything actionable.
We’re trying to replace product with LLM. Honestly, I welcome it.