• @acec
    link
    2310 months ago

    Compile llama.cpp, download a small GGML LLM model and you will have a quite intelligent assiatant running into your phone.

    • @bassomitron
      link
      English
      910 months ago

      Would that actually be decent? Even 6b models feel way too rudimentary after experiencing 33+b models and/or chatgpt. I haven’t tried those really scaled down and optimized models, though!

      • @[email protected]
        link
        fedilink
        210 months ago

        They’re decent for text completion purposes, e.g. generating some corpspeak for an email, or generating some “wikipedia”-like text. You have to know how to write good prompts, don’t try to treat it like ChatGPT.

        For example if i want to know about the history of Puerto Rico I would put:

        “The history of puerto rico starts in about 480BC when”

      • @acec
        link
        210 months ago

        Decent enough for a model 50 times smaller than ChatGPT. I use orca_mini_3b.

    • arthurpizza
      link
      English
      210 months ago

      I got llama to compile but it crashes every time I tried running it.

      • suoko
        link
        fedilink
        110 months ago

        Do we have any llama apk/deb/flatpack/appimage where you just choose the llm to use?

        • @ddx7
          link
          110 months ago

          lamacpp is kind of this. you give it a path to a model and a prompt. download a smallish model GGUF file by thebloke. lamacpp main exec has an interactive mode too which can function like a chatbot. i never built it on android though maybe you need android NDK