I don’t know how the devs would figure out the error and how to fix it.
This is like the biggest factor that people don’t get when thinking of these models in the context of software. “Oh it got it wrong, but the developers will fix it in an update”. Nope, they can fix traditional software mistakes, LLM output and machine learning things… They can throw more training data at it (which sometimes just changes what it gets wrong) and hope for the best, they can do better job at curating the context window to give the model the best shot at outputting the right stuff (e.g. the guy who got Opus to generate a slow crappy buggy compiler had to traditionally write a filter to find and show only the ‘relevent’ compiler output back to the models), they can try to generate code to do what you want and have you review the code and correct issues. But debugging and fixing the model itself… that’s just not a thing at all.
I was in a meeting where a sales executive was bragging about the ‘AI sales agent’ they were working, but admitting frustration with the developres and a bit confused why the software developers weren’t making progress when those same developers always made decent progress before, and they should be able to do this even faster because they have AI tools to help them… It eternally seemed in a state that almost worked but not quite no matter what model or iteration they went to, no matter how much budget they allocated, when it came down to the specific facts and figures it would always screw up.
I cannot understand how long these executives wade in the LLM pool and still believes in capabilities beyond what anyone has experienced.
I cannot understand how long these executives wade in the LLM pool and still believes in capabilities beyond what anyone has experienced.
They leave the actual work to the boots on the ground so they don’t see how shitty the output is. They listen to marketing about how great it is and mandate everyone use it and then any feedback is filtered through all the brownnosers that report to them.
It eternally seemed in a state that almost worked but not quite no matter what model or iteration they went to, no matter how much budget they allocated, when it came down to the specific facts and figures it would always screw up.
This is probably the biggest misunderstanding since “Project Managers think three developers can produce a baby in three months”: Just throw more time and money at AI model “development” for better results. It supposes predictable, deterministic behaviour that can be corrected, but LLMs aren’t deterministic ny design, since that wouldn’t sound human anymore.
Sure, when you’re a developer dedicated to advancing the underlying technology, you may actually produce better results in time, but if you’re just the consumer, you may get a quick turnaround for an alright result (and for some purposes, “alright” may be enough) but eventually you’ll plateau at the limitations of the model.
Of course, executives universally seem to struggle with the concept of upper limits, such as sustainable growth or productivity.
This is like the biggest factor that people don’t get when thinking of these models in the context of software. “Oh it got it wrong, but the developers will fix it in an update”. Nope, they can fix traditional software mistakes, LLM output and machine learning things… They can throw more training data at it (which sometimes just changes what it gets wrong) and hope for the best, they can do better job at curating the context window to give the model the best shot at outputting the right stuff (e.g. the guy who got Opus to generate a slow crappy buggy compiler had to traditionally write a filter to find and show only the ‘relevent’ compiler output back to the models), they can try to generate code to do what you want and have you review the code and correct issues. But debugging and fixing the model itself… that’s just not a thing at all.
I was in a meeting where a sales executive was bragging about the ‘AI sales agent’ they were working, but admitting frustration with the developres and a bit confused why the software developers weren’t making progress when those same developers always made decent progress before, and they should be able to do this even faster because they have AI tools to help them… It eternally seemed in a state that almost worked but not quite no matter what model or iteration they went to, no matter how much budget they allocated, when it came down to the specific facts and figures it would always screw up.
I cannot understand how long these executives wade in the LLM pool and still believes in capabilities beyond what anyone has experienced.
They leave the actual work to the boots on the ground so they don’t see how shitty the output is. They listen to marketing about how great it is and mandate everyone use it and then any feedback is filtered through all the brownnosers that report to them.
This is probably the biggest misunderstanding since “Project Managers think three developers can produce a baby in three months”: Just throw more time and money at AI model “development” for better results. It supposes predictable, deterministic behaviour that can be corrected, but LLMs aren’t deterministic ny design, since that wouldn’t sound human anymore.
Sure, when you’re a developer dedicated to advancing the underlying technology, you may actually produce better results in time, but if you’re just the consumer, you may get a quick turnaround for an alright result (and for some purposes, “alright” may be enough) but eventually you’ll plateau at the limitations of the model.
Of course, executives universally seem to struggle with the concept of upper limits, such as sustainable growth or productivity.