On that, what would people recommend for a locally hosted (I have a graphics card) chatgpt-like LLM that is open source and doesn’t require a lot of other things to install.
(Just one CMD line installation! That is, if you have pip, pip3, python, pytorch, CUDA, conda, Jupiter note books, Microsoft visual studio, C++, a Linux partition, and docker. Other than that, it is just one line installation!)
Look into llama.cpp - it’s a single C++ program that run quantified models (basically models with some less precision - don’t need a full 64 bits for a double, really). As for models to run on it, there’s so many but I think WizardLM is pretty good.
I looked into this too and it’s pretty resource heavy. I actually had a really good conversation with Chatgpt about making a separate instance of itself locallly. It’s worth talking to it about that and some of the price options
On that, what would people recommend for a locally hosted (I have a graphics card) chatgpt-like LLM that is open source and doesn’t require a lot of other things to install.
(Just one CMD line installation! That is, if you have pip, pip3, python, pytorch, CUDA, conda, Jupiter note books, Microsoft visual studio, C++, a Linux partition, and docker. Other than that, it is just one line installation!)
Look into llama.cpp - it’s a single C++ program that run quantified models (basically models with some less precision - don’t need a full 64 bits for a double, really). As for models to run on it, there’s so many but I think WizardLM is pretty good.
I looked into this too and it’s pretty resource heavy. I actually had a really good conversation with Chatgpt about making a separate instance of itself locallly. It’s worth talking to it about that and some of the price options