I just found https://www.arliai.com/ who offer LLM inference for quite cheap. Without rate-limits and unlimited token generation. No-logging policy and they have an OpenAI compatible API.

I’ve been using runpod.io previously but that’s a whole different service as they sell compute and the customers have to build their own Docker images and run them in their cloud, by the hour/second.

Should I switch to ArliAI? Does anyone have some experience with them? Or can recommend another nice inference service? I still refuse to pay $1.000 for a GPU and then also pay for electricity when I can use some $5/month cloud service and it’d last me 16 years before I reach the price of buying a decent GPU…

Edit: Saw their $5 tier only includes models up to 12B parameters, so I’m not sure anymore. For larger models I’d need to pay close to what other inference services cost.

Edit2: I discarded the idea. 7B parameter models and one 12B one is a bit small to pay for. I can do that at home thanks to llama.cpp

  • @tpWinthropeIIIEnglish
    3•2 months ago

    I tried Mistral Nemo 12B instruct this morning. It’s actually quite good. I’d say it’s close to dolphin mistral 8x7B which is a monster in size and very smart, about 45 or 50GB. So I’d say Arli is a good deal Mistral Nemo 12B for 4 or $5 per month and privacy so they claim.

    If you don’t mind logging for some questions, you can get access to very good or if not the best models at lmsys.org without monetary cost. Just go to the “Arena”. This is where you contribute with your blind evaluation by voting which of two is better. I often get models like 4o and sonnet 3.5 by Anthropic, google’s best, etc., and at other times many good 70B models. You see two answers at once and vote your favorite between the two. In return, you get “free” access.

    Be careful with AMD GPUs as they are not as well supported for local AI. However, support is gaining ground. Some people are doing it but it takes effort and hassle, from what I’ve read.

    • hendrikOP
      1•2 months ago

      Thanks. I’ll try lmsys, but ultimately I do mind privacy. But I also fool around.

      Yeah, I know about AMD GPUs. Nvidia has quite a monopoly on AI and as everyone uses their hardware and software frameworks, that’s what’s supported best. At least currently. My predicion is: that’s about to change. But their competitors didn’t do a great job. But I’ve been annoyed with Nvidia’s stupid Linux drivers for so long, (I mean that also changed,) but I’d like to give my money to someone else, and swallow that pill. If I decide to do it anyways.

      Thanks for the info. I think I can do something with that. Mistral-Nemo is pretty awesome for its size. Intelligent, can write prose, dialogue or answer questions, it’s completely uncensored out of the box…