The main reason I ask, is because my current favorite model is a Llama 2 70B Q4_1 GGML model quantized by The Bloke. Here’s the thing though, it was labeled as “Instruct” but it defaults to chat in settings in Oobabooga/Textgen. Every other model I have tried to use for technical help and python/bash snippets has failed to meet my expectations for (skeptically acceptable) accuracy. This 70B is powerful enough that I can prompt it to generate code snippets, and if the code creates an error, by pasting the error into the prompt, it almost always generates a solution in a single correction. Other models I have tried to use this paste-error technique on often crash, ‘dig in their heels’ insisting they are correct, or fail in several different ways like over fitting that forces resetting context tokens.
For whatever reason, the specific 70B model I am using has far exceeded my expectations, but I must use it with very specific conditions in Oobabooga/Textgen. It must be set to: chat, llama.cpp, the “divine intellect” perimeter preset, and the character profile set to the default of “None.”
For whatever reason, deviation from these settings ruins the accuracy of code snippets. Speculatively/intuitively, if I try to use the instruct prompt, or a new persistent character profile, it seems like there is an issue in the way the previous context is handled. In a single session the context seems to drift. In any case, code seems to always have errors and paste corrections fail.
I can’t contextualize this issue with such large models. I have had the same issues with smaller models regardless of settings I have tried. I have written or modified a dozen scripts between bash and python using this 70B in chat mode. It is a bit of a pain because the prompt input/output is not proper markdown for code so I have to correct for whitespace scope and have a reasonable understanding of the code syntax, but for the most part, I don’t need to make corrections to specific lines of output. Is this rare, an issue/quirk with: the model quantization, llama.cpp, Textgen, other? Has anyone else experienced something like this? Am I just super lucky to have found a chance combination that works really well at snippets combined with my prompting/coding skill level? I haven’t had much success with the code specific LLMs either. I’m not sure why this model is doing so well for me.
Many thanks. I got it mostly working today. At least, I got it working with llama_cpp. I haven’t gotten llama_cpp_cuda working. I used the same conda environment and distrobox as Oobabooga/Textgen. I tried (re)installing with pip and conda, but there is always some weird missing dependency (not at comp now where I can say exactly which). I tried searching for the libraries that were called out in the error but got no results, internet search had no good results, and even old trusty 70B had no helpful advice. On anaconda.org it had no results, but said something about not showing private libraries without logging in, whatever that means. With pip list there is an entry for something like llama_python_cpp_cuda. It is probably what I need but I’m not sure yet.
The results I got weren’t great, but it is a starting point. I was surprised how most models are around the same speed as Textgen. I can see the potential, but at the same time, for my needs thus far, just having Textgen and the 70B in chat mode all the time is handy and fast, even if I must filter through the poor formatting.
I should have said before, I was exploring the source code of extensions and overall, looking at how stuff works. Your explanation and guidance was too good of an opportunity to pass up. Getting this working on the command line was definitely on my list to try. I spent most of the day reading about the API and asking the 70B to explain stuff. I still have lots to figure out. Like setting up a simple script with terminal input got some odd ball responses several times. The correct answer was in the output but so was an extra half dozen random sentences; most completely unrelated. I think it had to do with a lack of prompt structure. I’m not sure how to format that in the terminal itself. I can do it in the script. It is the obvious solution, I just didn’t get around to it today.
I only used the part above
# ...or the "manual" wayI’m not clear on what the second part is doing exactly or the I/O. What I tried just seemed to loop with no output. I’m sure I’m doing something stupid that is causing the issue.
prompt_tokens = model.tokenize(prompt.encode('utf-8')) model.reset() model.eval(prompt_tokens) generated_tokens = [] while True: next_token = model.sample(temp=0.8, top_p=0.95, top_k=40, repeat_penalty=1.1, frequency_penalty=0.0, presence_penalty=0.0, tfs_z=1.0, mirostat_mode=0, mirostat_tau=5.0, mirostat_eta=0.1) if next_token != model.token_eos(): generated_tokens.append(next_token) model.eval([next_token]) else: break text = model.detokenize([generated_tokens]).decode('utf-8')Thanks again.