The law really needs to catch up with these developments. AI companies are selling the illusion of talking to an intelligence, then when it goes wrong hiding behind the fact that it’s just a machine designed to spew out bullshit. They know damn well how compelling that illusion is.

They should be held to the same standard as any person who behaves as their LLM behaves. It’s fair to attribute the intent to kill since outcomes like this are built into the product. If you set off a bomb in a crowded market place you would not be able to legally defend that by claiming you just wanted to put on a firework display and didn’t know it would kill anyone. LLMs are lethal by design, not in all cases but definitely in some. If you make a product which fools people into thinking it is impartial, responsible and aware of what it is saying, which actively cultivates delusions and encourages vulnerable people to kill themselves and others, that’s 100% your responsibility. There should be huge legal consequences for this kind of thing.