Maybe some devs here can help me, I was recently promoted to “head of AI” at my work despite being very outwardly ambivalent towards it. So I’m struggling to figure out what would actually create value instead of just being an expensive waste of time but still satisfy the higher ups AI lust.
My first idea that I thought would actually be useful was just setting up the architecture for an actual analytics database for us and then let them explore it with metabase (then letting them use Claude for their wow factor of exploring it with AI or whatever).
But now I’m somewhat at a loss, so any insight you all have would be really helpful!
This is wonderful thank you.
I’m starting to get the idea that the LLM is just a small part of this and what’s really important is instead the bunch of architecture and guardrails around it to form how the humans and ai will act with any given system.
Personally I see LLMs as a tool like any other. You can use it to mass produce low quality slop, just as you can use it to help you produce a higher quality output.
You’re perfectly right about architecture and guardrails, that’s how it has always been with any other tool or piece of software. It depends on how you use it. Remember the no-code hype train? It’s literally the same, people have been shoving it into everything, no matter whether it made sense. It worked for some, and it made development costs explode for others.
Guardrails are especially important for LLMs because you do not have deterministic outputs and potentially exploding costs.
So analyze, measure, and think about where and how it makes sense to integrate, and build it incrementally, again, just like with any other piece of software. Start slow, keep humans in the loop, measure and analyze, and improve incrementally. When you achieve confidence, potentially start automating going into an agentic direction, when it makes sense and the risks have been considered, but always keep provenance. You do not want blind decisions by the magical AI box.
And just to repeat, because I’ve seen heads roll because of dumb decisions: keep cost under control and always have limits set, and always consider which data flows into the AI and what happens with it afterwards.
Producing a half a million bill in a month by accident or neglect or suddenly having your customer database queryable on a public model is a surefire way to drive the company or at least your career to the ground in seconds of wrong decisions.
Also, read into all the stuff built around LLMs, protocols like MCP, attacks and defenses on LLMs, get knowledge about the inner workings, experiment and learn. When you’re the head of AI, you’re supposed to be the person who knows. And when you know what it does, how it works, and how to use it, you’ll find actually good and appropriate use-cases naturally.