Anyone giving “AI” access to production databases through tools like that are morons who shouldn’t be anywhere near a production environment.
Never use I/you with a software agent and never let one use those pronouns.
How do you achieve that? Calling it out when it happens?
I don’t like LLMs outside of debugging and the way it mimics human closeness creeps me out
Typically you can add some of your own alignment that is inserted before your prompt. On ChatGPT, they call it “memories”. I have found most agents are able to adopt a non-conversational mode when instructed, but sometimes you end up fighting with the platform alignment.
It’s not just creepy, I think it’s harmful, manipulative, and calculated. As social animals, we are extremely vulnerable to forming unconscious assumptions that we consciously know are bananas, particularity when our social machinery is engaged. For example, it’s well known that even the developers of video games fall the illusion of moral agency and personhood of the characters in their own games. We are primed to be fooled and accept software agents as people, and I believe AI companies are actively and intentionally trying to exploit this psychological weakness.
deleted by creator
what I always love about this stuff is most of the time it simply boils down to not committing changes. the LLM constantly assumes that you’re committing, it will NEVER tell you or recommend you commit, but it assumes you are. so these tech bros are just cruising along, vibing, and then boom the agent does something that nukes the entire thing. panic. tech bro starts freaking out on the agent. agent says “lets rebase or pull the previous commit” and gives them the git code to do that. tech bro hasn’t committed shit. and this is how you get a nuked prod environment.
None of that helps if you grant the LLM access to the production database and it decides to
drop table user;to free up some space. That’s when you find out if your database recovery process actually works. You do have database backups, right? Or is that not a thing when vibe coding?My friend and I set up Kopia in our home labs.
About a week later we decided to test out a disaster recovery only to discover we’d screwed up the permissions and nothing was backing up properly.
Was a relatively easy fix since we caught it, but I share this cautionary tale to remind you all to test your backups regularly!
I run code-server in a Docker container, isolated to sets of development projects, and backed up via ZFS that the container has no knowledge of. On top of that, each set of projects has it’s own user space.
I still get nervous hitting “Approve All” in Kilo. How do these people feel so free?
Ignorance is bliss.
Can I get paid 6+ figures to fuck up this badly with magical thinking?
This is the modern “all my apes gone”
Am I reading this right that they’re still letting the program run even as they figure out how badly it fucked up their system?
They have no idea what else to do. They were in over their head so long before this problem happened.
Giving production credentials to an LLM is wild
Seriously, has no one heard of sandboxing?
Yeah, when my company first forced Claude on everyone the head engineers managed to negotiate that Claude would only run in a WSL sandbox. But people were lazy, so they just gave that WSL as many permissions as possible (Mounting C directly to it, opening up all interfaces, popping in full-access git tokens etc.). Then management sent out an extremely biased “survey” that has the question “Is having Claude in the WSL inconvenient to you?” and all the lazy bastards said yes. So now management lifted the sandboxing requirement to make work “easier” for devs. In the meantime, the engineers arguing for proper sandboxing are already so worn out from telling people to not intentionally compromise their sandbox that they’ve kinda just given up. Not having a sandbox at all isn’t much more insecure than whatever people are already doing 🫠
I hope they have a good backup and recovery solution
Of course they do. It’s even mounted rw at the root of the AI drive for ease of access in case something goes wrong.
I hope they dont so that it hurts and they maybe learn
So, earlier today I was being unhealthy on youtube, and someone half my age made a HUGE point to tell his audience including me that even if a self-driving Tesla runs a red light, it’s the human driver that gets the ticket.
Now…I’m a pilot. I have been since I came in that guy’s mom. In the aviation community, we have this concept called Pilot In Command. In the US, this is set into law in 14 CFR 91.3. The pilot in command of an aircraft is fully responsible for, and is the final authority as to, the operation of that aircraft. Not the administrator, not your instructor, not air traffic control, not the President of the United States, not god, the PIC. That concept doesn’t exist in driver’s ed, but it needs to. We need to teach student drivers about the Driver In Command responsibility.
Too long, didn’t process the metaphor: Nobody thinks about anything they do unless the law requires it.
That concept doesn’t exist in driver’s ed
Well yeah, because there’s no-one else with a set of controls in a car.
The concept absolutely exists in law but it’s just called… “the driver.”
The concept is that anything that happens to that aircraft (or car) is the sole responsibility of the pilot. Flying into restricted airspace? If it’s necessary to avoid a collision, you do it. Putting the plane down in a field? Better than hoping you can make it to a runway when your engine is out and lives are on the line. Another driver signals that you are clear to make a left turn on a congested road? Nope, because when that traffic you couldn’t see hits you on the right, you’re responsible for your car, their car, and all the people inside both vehicles.
Well not quite. A manufacturing or maintenance defect will not be the responsibility of the pilot (unless they should have spotted it on their walkaround)
I’m not sure what comparison you’re trying to make though; the grandparent said there needs to be a similar concept for cars, I said that there already was one. Are you agreeing or disagreeing that there already is one? Are you saying that, at present, the driver would not be responsible for moving into traffic if someone else had flashed their lights? Because that is certainly not true where I live.
The problem is that in that case, due to poor training or judgement, the person driving the turning car did not consider themselves solely responsible for the safe operation of the vehicle, they let the driver in the signaling car decide if it was safe.
It’s subtle, but it happens all the time, because we don’t train drivers to think that way.
Wait whose mom? The youtuber? You were unhealthily watching your son whose half your age? And you became a pilot the same day of the conception and have been ever since…
If I solve this do I get to join Mensa?
Are you under the impression that everytime someone comes inside a woman that woman becomes pregnant, and that this can only happen one time in a woman’s life?
I don’t really see how fist fighting Arabs helps.
Having working production database config and credentials in your local .env, as appears to be the case here, is equally wild, and basically begging for something like this to happen.
For sure, its the same mistake. Maybe an LLM makes it more likely to result in consequences
“You are a world-class, expert software engineer…”
“Make no mistakes”
As if the LLM can truly understand what a mistake is…
But… but… I told it not to break anything!
Some time ago I worked for an insurance rating company as a tech and the task was given me to go run through the new code that was in beta. Sure ! I spent a few hours on Friday trying to break it and I couldn’t, so at the end of the day I got a little funky with the .css backgrounds and put in a very tiled Beavis and Butthead gif. It looked freaking horrible and I loved it. Monday I was directed to the big guys office ( the developers had not given every beta account a separate .css file … or even separated things. Everyone in beta called in Monday with that background. I didn’t get in trouble because they wanted me to break it. Really awkward conversation though trying not to smile.
That’s incredible lmao
I added guardrails to myself to make sure I do not accidentally delete anything on production. I would never ever let an intern, a junior dev or a fucking AI onto that database. Not in a thousand cold nights.
Prod should always be highly “air gapped” with some sort of deployment process which tests not only the code to be deployed but also the deployment itself. I’ve been doing QA for a good while now, and everywhere I’ve worked has testers dedicated to testing the actual update process to make sure it will be safe when deployed.
And I won’t open the DB without making sure I’m read only. If I need to mutate data or schema, I’ll switch roles and have a dry run first.
Should have added “no mistakes, no bugs” to the prompt! Pffft, amateur.
“DON’T HALLUCINATE!” that’s always my favourite.
That takes me back to 2004
No Smoke, by unknown,
Tech: “Hello. How can I help you today?”
Customer: “There’s smoke coming from the power supply on my computer.”
Tech: “Sounds like you need a new power supply.”
Customer: “No, I don’t! I just need to change the startup files.”
Tech: “Sir, what you describe is a faulty power supply. You need to replace it.”
Customer: “No way! Someone told me that I just had to change the system startup files to fix the problem! All I need is for you to tell me the right command.”
(Ten minutes later…)
Tech: “Well, we don’t normally tell our customers this, but there’s an undocumented command that will fix the problem. Add the line “LOAD NOSMOKE.COM” at the end of the CONFIG.SYS file and everything should work fine.”
(Five minutes later…)
Customer: “It didn’t work. The power supply is still smoking.”
Tech: “Well, what version of Windows are you using?”
Customer: “Windows 98.”
Tech: “Well, that’s your problem. That version of Windows doesn’t include NOSMOKE. You’ll need to contact Microsoft and ask them for a patch.”
(When nearly an hour had passed, the phone rang again…)
Customer: “I need a new power supply.”
Tech: “How did you come to that conclusion?”
Customer: “Well, I called Microsoft and told the technician what you said, and he started asking me questions about the make of the power supply.”
Tech: “What did he tell you?”
Customer: “He said my power supply is not compatible with NOSMOKE.”
That is way older that 2004. I first read that in the early 90s.
TIL Win98 was out in the early '90s…
That’s because the joke got updated to the newest OS version until CONFIG.SYS got entirely removed in Windows ME and it just didn’t work any more.
It’s originally from MS-DOS times. Earliest telling of it I know of is from 1996.The 60’s happened in the 70’s.
The 70’s happened in the 80’s.
You get where I’m going here?
I think so. The 2010-2020s happened in the 1930-1940s, amirite?
Lol. “do magic”
“or you go to jail”
My company has been trying a new model when product folks cut through the red tape of “engineering” and just describe what they want to a powerful LLM pipeline and review the app in a beta env. Sounds perfect, right?
Dear reader, in the couple months this has been going on, these people have caused a dozen high profile SEVs due to extremely poor app performance, networking / kubernetes configuration bugs, bad scaling, observability oversights, supply chain attacks, leaking sensitive information, and cost overruns (on practically every resource they provision).
Some very well-paid people are scrambling to figure out the value that was generated by this pilot program; I’m heating up popcorn rather than holding my breath.
That’s hilarious, idk i think llm could be useful for helping product folks translate their thoughts into actionable items for the devs, but yeah like beyond insane to tell the product people to hop on claude and do it themselves. That’s like a construction company letting the sales team jump in an excavator and start digging!!
“it’s fine, he doesn’t need to know how to use the controls. We just installed a voice command system into the excavator.”
even just using it to translate their thoughts into a bullet list isn’t going to work most of the time. LLMs training is, the majority of the time, out of date and stacks change/move faster than an agents data. Plus LLMs favour things that have been posted about on blogs, linkedin posts, etc rather than stuff that is proven and tested.
Like for example it might suggest using a language or library that is insanely out of date or just won’t work but recommends it based purely on say blog posts that have talked a lot about it or it’s potential to do something that it currently can’t do. or like I said it’ll suggest something that’s out of date simply because there’s more documentation online about that specific version than the most up to date version. Or the LLM will report on something that has a “known issue” from like early 2025 and suggests not using it when said issue has been patched routinely since then because the data the LLM has is out of date. the data is constantly out of date.
I guess shouldn’t product people not give a shit what language you write it in? Like that’s the problem product people need to describe what they want while letting the devs do the job of generating the code.
right the product people won’t care or won’t know so the LLM will just spit out whatever. to the product people it’ll look legit and make sense. once it’s handed to a dev in most cases it might as well just be thrown out of the window. so it’s pointless. it’d be easier/faster to just cut out the middle man, the AI, and go directly to the dev and have the product people tell them exactly what they want/need.
llm could be useful for helping product folks translate their thoughts into actionable items
In my experience it makes them give me an essay instead of 10 lines of bullet points and I have you spend an hour asking questions to whittle it down to 10 lines of a bullet points
I was thinking more along the lines of “let the executive tell an LLM to write the shitty prototype version of what they want (which you then rewrite from scratch to not be shit) so you don’t have to decipher their incoherent desires”
We’re trying to replace product with LLM. Honestly, I welcome it.
