How reliable is AI lke ChatGPT in giving you code that you request?

  • amio
    link
    fedilink
    21
    edit-2
    1 year ago

    ChatGPT is a language model, it’s not intended for code and you’re using it “off label” at your own risk. It can produce working code, which is impressive in itself, but in order to know if it’s decent code you still need to be competent with that language. I had someone run a few prompts for me a while back, it ignored central parts of the query, and its output was basically like a very junior developer - fair enough, but not great or even that good.

    Potentially useful, but if you expect it to be more than one part of the “process”, you might be setting yourself up for trouble.

    Edit: just like it’s not a coder, it’s not a search engine or knowledge base, either. It just knows language and what seems like it ought to follow a given phrase. Be very aware of this difference, because sometimes it spits out 100% falsehoods with the same level of confidence and authority as the true stuff.

    • Kichae
      link
      fedilink
      8
      edit-2
      1 year ago

      I think it’s important for people to also truly understand that generative machine learning models like ChatGPT also only “know” what they’ve seen before. There’s no interpretation or synthesis. It merely regurgitates what it’s seen, with some sampling from a probability distribution.

      This means, if you’re asking for something niche, and it’s only seen what you’re prompting it for once (or, really, the same text repeatedly across multiple websites) , there’s a very good chance that it will just recreate that artifact wholesale.

      Which means you need to be cognizant of what the license for that material is before you use it in a product!

  • AlteredStateBlob
    link
    fedilink
    161 year ago

    I have built several programs with ChatGPT 4 by now. From very basic Python scripts to Python webscrapers and C# in combination with Unity3D.

    In the beginning it was much better than it is currently. At the moment context is severely hampered no matter the limit and you’ll be bashing your head against circular arguments and it straight up ignoring stuff you just posted two messages ago.

    Trying to troubleshoot code it wrote a few days ago will be a slog and like dragging yourself over nails at times. Here’s what I have found to help and make life better:

    1. Be very, very, very precise in your instructions. And keep them saved, so you can reuse them later (point 4)
    2. From the very start plan to build your project with small functions that interact (good policy anyway) which makes troubleshooting and changing these functions much easier and will prevent you running into message limits.
    3. If it fails to work for you the way you need it, you’ll might have to scrap your entire code and start over with ChatGPT -> Again, the reason for point 2 being very important. Scrapping one functin is much less painful than an entire tool.
    4. Start new chats when you feel the quality degrading. Sometimes it helps and since the context is garbage at the moment anyway, it doesn’t matter much.
    5. Post the code it is supposed to fix every single time. It will inevitably refer to other code, code hallucinations, etc. otherwise. Again why point 2 is important.
    • Avalanched
      link
      fedilink
      11 year ago

      god I hate those circular arguments, it’s like you’re arguing with a todler

  • Utilael
    link
    fedilink
    8
    edit-2
    1 year ago

    I agree with the other comments that ChatGPT isn’t really that good for programming, it hallucinates often and you end up working too hard just to try and figure out what it got wrong. However, I have found a good AI engine, phind.com, that has started to replace my google searches. It’s just a wrapper for ChatGPT, but it cites its sources so you can verify or dig deeper, provides search engine results in a sidebar and has upvote/downvote options to help it improve. So it feels like a personal google “agent” that runs off and googles something for you and comes back with a concise report.

    • Deely
      link
      fedilink
      3
      edit-2
      1 year ago

      Personally I just can’t work with system that lies to me (even for a little) but all the time.
      I tried to use chatGpt and Bing bot and phind.com few times and everytime I got answers that looks like real and looks like correct answer but slightly (and few times completely) wrong.
      Everytime I have to reread documentation, check links, investigate is there a reason why LLM answered this way, maybe I wrong this time and LLM found something that I did not found…

      I agree that phind.com get best results, but every small incorrectness here and there irks me and makes me question myself and answer as whole.

      Upd: in general questions, like when you trying to investigate some new field, technology, tooling suite LLM is very, very good. When you want to get something like overview of topic that you interested in.

  • Otome-chan
    link
    fedilink
    71 year ago

    I use chatgpt a lot when coding. It’s pretty good and the code is typically usable. But sometimes it messes up hard and it can take a while to realize that. Net benefit though and I’m sure the technology will improve over time.

  • PolarBearSequence
    link
    fedilink
    71 year ago

    I quite like GitHub Copilot and use it a lot, but I find ChatGPT not all that useful.
    For actual coding, it feels like describing what I want it to do is more complicated than doing it myself.
    I can see some uses as a search engine, but I’ve had a lot of bad luck where it suggested code that was plain wrong or not working and often did not even compile, so most of the time, I’d rather look on GitHub, Stackoverflow or sites like that.

  • mfz
    link
    fedilink
    61 year ago

    It is a conversational tool that can generate decent code if properly prompted but it lacks for the most part enough context. For it to be really useful it has to be able to be trained on my entire project that I’m working with, not just a single file or function.

    What I miss is the ability to “chat with my project”. I.e. have the whole project in the trained context, and then reason about architectural changes, pros and cons, have suggestions for refactoring, help with complex renaming schemes and moving code, etc.

    It would be super interesting to be able to give instructions like:

    • Organise my files by dependency and the logic they implement.
    • Or something like, create web components from common input elements in my html pages.
    • Where is the user auth code implemented?

    Things like that.

    • experbia
      link
      fedilink
      21 year ago

      This would really be extremely helpful, absolutely agree. A mode with less of a view on the details of the code and more with the architecture of it. I wonder if an extension like Codeium could be extended so that only the method signatures and comments and such could be sent in as context so it can reason more generally about your project…

  • Laage
    link
    fedilink
    51 year ago

    I’ve used it a bit to get the framework or boilerplate I need. It’s not a one-click solution!
    What I will do is ask it to generate code for a purpose and then iterate over the suggested code adding and/or modifying specific areas until i have something usable… Depending on the complexity of the code/feature this can take quite a number of iterations. You need to understand the code it gives you!
    Once I feel satisfied I will copy the code to the editor for tweaking and adjusting to my environment.

    I will admit that I’ve been surprised sometimes by the suggestions I’ve been given. Sometimes in a good way, sometimes bad.

    Remember that whatever you feed it will feed the LLM, so don’t give it anything specific to yourself or your organisation.

    • azura
      link
      fedilink
      21 year ago

      It seems like a small thing you said on the side, but it is really important that you actually understand and can explain the code it gives you that you’re copying into your project. Otherwise you’re taking in an unknown, unmaintained and unexplained dependency, and that can lead to problems once that dependency fails.

      • Laage
        link
        fedilink
        21 year ago

        Agreed that I should have been clearer…
        Both my statements of it not being a “one-click solution” and the aside about understanding the code was meant to emphasize that very point.

  • InduperatorRex
    link
    fedilink
    51 year ago

    ChatGPT 4 is pretty decent at checking code for any mistakes, and it can generate pretty good code if you can describe what you want very well. But sometimes it does give you code with a slight mistake or two, so what I normally do is give it the code in a new chat and get it to check it itself

    • vektor
      link
      fedilink
      31 year ago

      so what I normally do is give it the code in a new chat and get it to check it itself

      Big brain moment

  • Flaky
    link
    fedilink
    5
    edit-2
    1 year ago

    I’ve used ChatGPT to answer questions relating to Python. Notably, I asked it how to use QtNetwork to send and receive requests with authentication, as the application I was using did not use any non-standard modules I was more accustomed to like requests but did have PyQt. Not only did it gave me working code snippets but explained it in a way that I was able to understand. No, it’s not perfect. But man it’s better than hunting Google for that one StackOverflow post.

    I have heard it trips up on certain less-used programming languages like Swift though, so depending on your use case YMMV. I’ve also not used Codex but a friend of mine has. Apparently it really liked to mention this one specific GitHub profile.

    For shits and giggles I asked ChatGPT a while back to represent a Pokemon with a Python class, and it gave me working code. Google Bard would trip up and not use the class when I told it to.

  • @psyonity
    link
    41 year ago

    While it’s pretty decent at coding, it’s often (in my experience) either giving you an overcomplicated way, an outdated way or just completely wrong code.

    What I do really like is that when you ask for a snippet you can easely ask for variations, like: “make it a bit shorter”, “add comments” or “don’t/do use library X”. This allows you to quickly get a few variations that allows you to come up with something of your own.

    On a slightly other note, I’ve been impressed by github Copilot on this subject. While still often wrong, with small things it almost feels like it’s reading my mind while programming.

  • LollerCorleone
    link
    fedilink
    41 year ago

    Its good for basic Python scripts and stuff, but not so good for complex programs.

    • ___hulk
      link
      fedilink
      31 year ago

      Like any other tool, it’s as good as you use it. If you can explain complex problems in smaller bites with clear objectives it helps a lot.

      • bored_pistachio
        link
        fedilink
        31 year ago

        Yes, but no.
        You can be as precise as you want, if ChatGPT didn’t have enough training material you won’t get good answer even if you bend over backwards.
        I can’t complain 95% of time, however prompt is not to blame all the time.

  • rastla
    link
    fedilink
    41 year ago

    It’s certainly helpful when you want to try out something new.
    For example I recently wanted to make a Firefox AddOn, which is something I hadn’t done before.
    So I asked OpenAI how to do it and it talked me through it step by step.

    Basically it allowed me to google less, because I could just ask ChatGPT, as it was faster.
    Some info is outdated or wrong, so you still have to know what you are doing and still have to use Google.

    Also, I wanted it to help me get some data from the DOM, but that was a rather difficult job for OpenAI, it never “understood” what I wanted and just gave me code that didn’t do what it was supposed to do, and even after explaining the problem with the code and telling it what I want, it wouldn’t understand the problem and just give me other bad code that changed nothing about the problem.

    So it’s important to understand, that this is just one more source of information/help you have as a dev. It is not a standalone solution that can do your work for you. It merely can help you. The same way as googling, stackoverflow or reading the documentation helps you

  • i_need_a_vacation
    link
    fedilink
    31 year ago

    For basic things like syntax that I can’t remember it’s actually pretty good, way more faster than google IMO.

    When I ask for something a little more complicated it can go two ways:

    • Actually doing a good job and generating something that I can use (I often have to polish that code, but still, it’s better than expected)

    • Doing whatever that I didn’t ask for, so when I point out that’s not what I meant and explain it further, it enters a bucle where it goes back to the two same solutions.

    “You didn’t like solution A? Here, there’s solution B. Solution B is terrible? Here, there’s A again.”

    Overall it’s a great tool though.

  • asteroidrainfall
    link
    fedilink
    3
    edit-2
    1 year ago

    Outside producing one simple WebPack configuration, I haven’t had good experiences using ChatGPT. It often causes me more trouble than it helps. I’ve tried to use it multiple times to write some BASH script, and every time it gives me know that looks nice but is just broken. It’s not syntactically incorrect, it’s more like functionally incorrect.

    For example, it told me that you could pass arrays as function arguments, which you can’t do. Or, it gave me a script that was using variables within a URL string that would be passed into CURL, which won’t work since the URL won’t be encoded properly.

    When I do it, I spend more time trying to fix the code that it gives me. Which, I guess, does have the benefit that it means I got to learn something afterward (both examples above I didn’t know about until ChatGPT gave me the bad code).

    The thing that ensured me that AI won’t take over the programming side of software engineering was when I asked ChatGPT to help me out with some date-time bugs. It just kept making up native JavaScript API functions, couldn’t understand how to parse UTC to figure out a date-time’s timezone, among other issues. The day that AI is able to solve software issues around date-times or currencies is the day that we’ll all be out of a job.

    Edit:
    I guess you could summarize using ChatGPT is like peer-programming with an overly confident CS grad.

    • experbia
      link
      fedilink
      21 year ago

      peer-programming with an overly confident CS grad
      I love this, and agree. I’ve always said that for all tasks, it’s like you’re working with an ADHD eager-to-please intern.

  • Cyzaine
    link
    fedilink
    31 year ago

    Used chat gpt 3.x to assist in Godot and gdscript logic for our game… Nearly always wrong but often gets me thinking in a different direction than the one I was stuck in. So certainly value there.