Hello everyone!
I am back! I appreciate everyone who has been posting in the interim. Love to see new content being shared.
This thread is inspired by one submitted by @[email protected], who in this post detailed general LLM resources for everyone to discover and interact with (content which has now been consolidated to the full LLM guide here, thanks cll!).
All that being said, I promised I would get back to them on a few questions they had. I figured why not make a thread dedicated to questions any of you would like to ask me.
AMA
If you are here and subscribed to [email protected] - please feel more than welcome asking your question in the comments below. I will try to get back to you within a day or two. I will do my best to answer all questions within reason.
Note that I am not an expert in Machine Learning, but I do work as a professional close to this category of tech. Take my responses with a grain of salt and cross-reference other sources to make sure you are informed.
Always do your own research on top of any of my advice. I am not responsible for the success or failure of your endeavor. Own up to your builds, own up to your development. I am but a resource and friend along the way. A humble guide on your journey to the singularity.
Questions
from @[email protected]
Q1.) What people do you follow for AI? Such as on YT, Twitter, etc.
A1.) I am not on social media often. Lemmy is my safe haven, alongside very specific YouTube channels who give me the information I seek.
Generally speaking, I try to avoid the sensationalism and stay focused on how I can empower myself with this tech now. So far, the channels below have helped me stay informed with practical insights and information.
Whatever it is you’re doing, be sure to have a goal in mind. You will spend an eternity learning with no direction.
For me, it was learning how to regularly and consistently train, fine-tune, and deploy open-source models on HuggingFace. That goal has coincidentally led to the creation of this community and my newfound passion for Machine Learning & AI.
Matthew Berman
I like Matthew. He’s cool. Always on top of AI news.
AemonAlgiz
Aemon is great, provides both general education and technical tutorials.
Aitrepreneur
Useful info, just wish the channel wasn’t so clickbaity. I get it though.
AI Jason
Great for more technical tutorials that you can follow step-by-step.
Venelin Valkov
Another source for tutorials.
Code Your Own AI
General education.
Q2.) What other social media forums provide great information?
A2.) Like I mentioned in my last response, I’m not on social media often. I’m a year one gen-z’er (1997) who never really got into social media (until now). Weird, I know. That being said, the only other place I frequent for knowledge is Lex Fridman’s Podcasts - which are treasure troves of information if you enjoy the discussions and/or content. Particularly any of the episodes with tech-based content.
Even if you don’t like some of the guests, you can infer a lot of interesting information based on what context they share with Lex. Much of what I have pieced together for our community here at [email protected] is information I have cross referenced with other independent papers, news, and articles I find credible. I think that’s called journalism, but the definition of that has been skewed compared to today’s social media standards.
Aside from those podcasts, I am a big fan of the /r/LocalLLaMA community, but Reddit is hard for me to revisit nowadays. It just doesn’t feel the same after the API change. I feel the best parts have left or are tucked in corners that are hard to find.
In other words, I get all my info from those YouTube channels I shared above, random posts and comments I find in niche corners, and/or published papers or articles from firsthand sources (typically Arxiv or Medium).
That, and reading documentation. Lots and lots of documentation on whatever it is I’m using at the time…
Q3.) What GUI do you use for local LLMs?
A3.) I might be biased because I’m a linux & vim kind of guy, so I am totally fine working in a terminal - but if I need a consistent GUI for testing, I typically default to oobabooga’s text-generation-webui.
When I am using a GUI that’s not oobabooga’s, it’s for R&D. In the past I have found myself testing with SillyTavern, koboldAI’s client, and gpt4all - but none of them stuck with my workflows. I revisit them from time-to-time to see how they’ve developed. They’re definitely improving and worth taking a look at if you haven’t already.
I am finding a lot of new interfaces to chat with LLMs, but not any in particular that stands out above the rest in terms of look, feel, and functionality. I think it’s only a matter of time before we see even more UI/UX’s tailored for LLMs. Until then, I’ll be using ooba and other side projects I tinker with from time to time.
When in doubt, you can always learn to code/design your own GUI, but that’s pretty advanced if you’re just starting to get into all of this. At the end of the day, you should use what’s best for you or what you prefer (or like) using the most.
Q4.) What parameters are “best”?
A4.) This is a tricky question. It could mean a lot of things, I’ll do my best.
a.) If this is in reference to hyper parameters and general parameters, I suggest reading this article for starters.
b.) If this question is in reference to what size parameters I personally find ‘best’ within LLMs (3B, 7B, 13B, etc), well, I’d say the best model is anything you can comfortably run that outputs the results you’re hoping for. That could mean a high parameter Falcon 40B model, or perhaps a low parameter 3B Nous model.
The results are so drastically different between use cases and performance that I say experiment with all and see which is the most consistent for you particularly.
For fellow technologists running LLMs at home, that probably means you’re running this on consumer-grade hardware. For desktop builds with NVIDIA GPUs I’ve found really consistent performance from 7B and 13B parameter models.
With some optimization, you can speed up inference depending on what base model you’re using and what platform you’re running it on. Sometimes even some of the 3B models surprise me with their coherence and perplexity. It’s up to you to find what works for you best.
c.) If you’re asking about specific parameters I apply during training - I don’t have an answer to that yet. I haven’t completed enough deployments to give you a worthy response.
Q5.) Is there a Wiki you use?
A5.) I have not found an open-source wiki that suits my need, so I made FOSAI ▲ XYZ to fill help in the gap and serve this community. It’s still a work in progress. I sort of used this to practice my web dev. I have plans on upgrading it to a format that will better serve the variety of content it presents. It’s a little dry IMO, but here nonetheless. I’ll be sure to let everyone know when the new site is live!
When in doubt, read the native documentation of the software (or hardware) you choose to use. It might be a dense read, but nothing CTRL + F can’t help with.
Q6.) Where do you go to learn about LLMs/AI/Machine Learning?
A6.) Like everything I do, I self teach myself concept by concept with a results-based mindset referencing all of the learning material and channels I shared a few questions ago.
I frequently fall victim to paralysis by analysis, so I combat this by dividing my ambition into actionable projects with the primary intention of getting them up and running (with reconciliation of notes, knowledge, and understanding after the fact).
I think it’s important to adopt a mindset that is brave to jump right into the middle of something without understanding any of it. Almost like exposure therapy.
Over time - concepts will become less alien and things will begin to click. Consistency is key. In my experience, curiosity empowers wonder, which can lead to motivation to learn something you care about. If you care about Machine Learning as much as I do, you’ll find ways to teach yourself through sheer will.
On my quest to better understand Machine Learning, I put together an online syllabus that is supposed to equip you with a core foundation of skills necessary to become a professional in the field. This is pieced together from a few videos I stumbled across and recommendations from colleagues in the field. Granted, I have the advantage of already being in tech, but I hold no degree - just a high school diploma.
Math is hard, but part of the fun once you realize the fact that it’s the language of reality, physics, computers and science. If you want to learn the same material I am, check out the Machine Learning page from FOSAI ▲ XYZ. Pick and choose what you want to learn. It is non-linear, non-sequential, and here to be learned at your own pace.
Q7.) How do you find quality models?
A7.) Every other day I check and see what TheBloke has released on his HuggingFace. As far as I’m concerned, he has become an open-source hero for the field. I trust everything he puts out, often cherry picking the model I want to test based on their origin card or foundation model.
If there’s a model worth running, Tom is fast to optimize a GPTQ or GGML model that he hosts on his page for all of us to interface with (for free). A true gigachad.
Support Tom on Patreon! Some his tiers offer direct technical support if you’re hitting a wall and need some hands-on help: https://www.patreon.com/TheBlokeAI
Q8.) What Awesome github repositories do you know?
A8.) I’ve been waiting for someone to ask me this! Haha. Finally…
I have been collecting all kinds of interesting repo’s for FOSS & FOSAI technologies. I am planning to do a new series soon where I dive into each one in more detail.
If you want a head start on that series, check out all of the cool tech people are developing here: https://github.com/adynblaed?tab=stars
I will be highlighting a few for AI / LLM particular use cases soon. You and everyone else here are welcome to explore some of the stars I’ve collected in the meantime.
Q9.) What do you think would be useful to share?
A9.) For [email protected], I think it’s useful for everyone to share their experiences with using FOSS & FOSAI platforms (anything listed in our sidebar) from performance to discoveries, breakthroughs and celebrations.
To be more specific, it would be awesome to see how everyone is using LLMs on a day-to-day basis (if at all), and how it has impacted their lives productively and/or recreationally. For me, this has manifested in the fact I use Google Search 20% of the time I used to, since local LLMs + ChatGPT have replaced primary ‘search’ for me. This is a big deal because I work in tech and I can’t tell you how much of my job was having decent Google-Fu. That has changed into how well I can prompt and how effectively I can extract usable information from a generative pre-trained transformer.
Now that this tech is here, spreading, and open-sourced - I feel we’re in the calm before the storm - the crest of the rising tsunami of change the world is about to see in the next few years. I am excited to brave this ship and ride the crest with each and every one of you - but knowing where you want this ship to sail gives me insights that helps me adjust the content I provide (however long I can provide it) and course correct as needed.
Everyone posting their experience with how AI / LLMs / free open-source artificial intelligence are impacting them now, today is absolutely worth sharing even if it seems small or insignificant. I use this community as a time capsule as much as I do engaging with it for learning, sharing, and staying in the know.
Looking back at what we’re talking about in 2023 versus what we might be talking about in 2024 will be an interesting context to study, especially as the world begins to adopt AI.
Sharing ideas can lead to discussions, with enough reinforcement these ideas become concepts.
Concepts have opportunity to turn to projects, which develop into realities for all of us to explore together.
Whatever your idea is, please share it. No matter how large or small, silly or ridiculous. If it’s satire, label it and let’s have fun in the absurdity. If it’s legitimate, let’s explore the possibilities and draw collective curiosity towards a common goal. This is how we grow and empower one another.
To innovate is to think, to think is to live.
Share your innovations! [email protected] is a safe space to create, think, express, invent, and explore the mind, the self, and the machine.
I’ll be leaving this up for the duration of August! Feel free to use the comments here to ask away any question not already addressed.
I am a big fan of the /r/LocalLLaMA community, but Reddit is hard for me to revisit nowadays.
Quick info: We have [email protected] over here. But I don’t think we’ve had an coordinated effort (yet) to move away from Reddit, so it’s only a few people. (I bet people also split up and some moved to some random discord communities. Nowadays many discussions happen behind closed doors scattered over small and larger discord communities. And LocalLlama on reddit is also still pretty active.)
I will say, if I’m on Reddit, it’s for /r/LocalLLaMA. For the most part, everyone there has been very helpful and supportive of the tech. Glad to see parts of it on Lemmy! Thanks for sharing, I’m going to add it to the community page on FOSAI ▲ XYZ.
What is your vim setup for python? I need a better dev setup for python. Pycharm and VS Code have too much BS in the background and I am never letting their infinitely long list of network connections through my whitelist firewall. I started down that path once; never again. I know about VS Codium, and tried it, but all documentation and examples I came across only work with the proprietary junk. Geany is better than that nonsense. I used Thony with Micropython at one point, but didn’t get very far with that. I tried the Gnome Builder IDE recently. It has a vim like exit troll. You can’t save files in Builder and the instructions to enable saving calls for modifying a configuration file on the host while giving absolutely no info about where the file is located. I need a solid IDE that isn’t dicking around in network/telemetry or configured by the bridge troll from Monty Python’s Holy Grail.
What is a good starting point for practical CLI LLMs. I need something useful more than some toolchain project. I’ve gone down this rabbit hole far too many times with embedded hardware. I like the idea of trying something that is not an AVR, but by the time I get the toolchains setup and deal with all the proprietary bs, I’m already burned out on the project. In this space, I like to play around in the code with some small objective, but that is the full extent of my capabilities most of the time. Like I just spent most of today confused as hell with how a python tempfile works before I realized the temp file creation has its own scope… Not my brightest moment
What is your vim setup for python? I need a better dev setup for python. Pycharm and VS Code have too much BS in the background and I am never letting their infinitely long list of network connections through my whitelist firewall. I started down that path once; never again. I know about VS Codium, and tried it, but all documentation and examples I came across only work with the proprietary junk. Geany is better than that nonsense. I used Thony with Micropython at one point, but didn’t get very far with that. I tried the Gnome Builder IDE recently. It has a vim like exit troll. You can’t save files in Builder and the instructions to enable saving calls for modifying a configuration file on the host while giving absolutely no info about where the file is located. I need a solid IDE that isn’t dicking around in network/telemetry or configured by the bridge troll from Monty Python’s Holy Grail.
I am usually just running the script I’m working on post-editor in whatever command line interface I find myself in. That could be zsh, bash. or something random I found that week. If I have the time, I like to setup zsh, or ohmyzsh depending on my OS, paired with power10k and custom color schemes.
For Windows, I usually set something like this up,
For Mac or Linux (Ubuntu) I like to use vim and/or tmux + rectangle.
As a practice, I often try as many new editors as I can, week by week or month by month. It helps keep me on my toes, but when I’m looking for a stable experience I typically default to VSCode behind my firewall. I feel your pains with the allow listing, but it’s the choice if I have something I’m working on and want to take my time on it. Otherwise, I’ve hopped between some of these. Check them out. Might not be for you, but they’re fun to try:
What is a good starting point for practical CLI LLMs. I need something useful more than some toolchain project. I’ve gone down this rabbit hole far too many times with embedded hardware. I like the idea of trying something that is not an AVR, but by the time I get the toolchains setup and deal with all the proprietary bs, I’m already burned out on the project. In this space, I like to play around in the code with some small objective, but that is the full extent of my capabilities most of the time. Like I just spent most of today confused as hell with how a python tempfile works before I realized the temp file creation has its own scope… Not my brightest moment
What sort of toolchain project were you exploring? I’m curious to hear about that. In all honesty, the reason I have so many GitHub Stars is a.) I am a curious person in general and b.) I’ve been looking for practical and pragmatic use cases for LLMs within my own life too. This has proven to be more difficult than I initially thought given the rapid developments of the space and the many obstacles you have to overcome between design, infrastructure, and model deployment.
That being said, I recently came across Cohere, who have an SDK & API for calling their ‘command’ models. Unfortunately, the models are proprietary but they have a few projects on GitHub that interesting to explore. In my experience, local LLMs aren’t quite at the level of production-grade deployments people expect out of something with the perplexity of ChatGPT-4 (or 3). The tradeoff is data privacy, but the compromise is performance. What I am liking about Cohere is that they focus on bringing the models to you so that data can remain private, with all of the benefits of the API and hosting capabilities of a cloud-based LLM.
For anyone starting a business in AI, being automation agencies or consulting services and integration engineers - I think this is important to consider. At least for enterprise or commercial sectors.
Home projects? Well, that’s another story entirely. I’ll take the performance tradeoff for running a creative or functional model on my own hardware and network private and 100% local.
A fun project I’ve been exploring is deploying your own locally hosted inference cloud API, which you could call from any CLI you’re developing on if you’re connected to your private network. This way, you get an OpenAI-like API you can tinker with, while hot swapping models on your cloud inference platform to test different capabilities.
At this point, you are only limited by the power you can pump into your inference cloud. A colleague of mine has a server of his that has 1TB RAM, 200+ CPU Cores, and x4 GPUs we’re working on setting up with passthrough, pooling available VRAM. We’re hoping to comfortably run 40B GPTQ or high parameter GGML models using this home server rig.
Assuming you get a private LLM cloud working at home, you can do all sorts of things. You can pass documents through something like Llamaindex or Llangchain, taking personal notes or home information and turning it into semantically available knowledge. This would be available to you on any CLI on your network, maybe through something like LocalAI
These are really big ideas, some that have taken me months to put together and test - but they’ve been really exciting to see actually work in small ways that feel fun and futuristic. The only problem is that so many of these libraries are changing with rapid development that projects frequently break with a simple change of library or lack of documentation due to a compatibility issue with a vague library that is new and not fully built out and supported.
I don’t know if that answers your question(s), but I’m around if you want to ask about anything else!
Thanks for all the references. I don’t bounce around so much. I wish I had the focus and energy to do as much as you. I’m not even 40 yet, but chronic injuries are getting the best of me. No sob story, just is what it is.
What sort of toolchain project were you exploring?
As far as embedded hardware? I’ve tried a bunch. My favorite was Flash Forth. I have it working on a few different PIC18’s, a PIC24 and a couple of Arduino Mini’s. For other stuff, I’ve tried 8051 variants, STM32- F0/F1/F4/F7’s, MSO430, PLD’s, ICE40, K210, ESP32, etc. I honestly never got very deep into programming them. I was much more interested in KiCAD board layout and designing/etching my own stuff. I’ve built a couple dozen projects, but nothing complicated or really worth mentioning. I really struggle with managing complexity in embedded. It was only in the last few months that I started exploring CS on the OS/Kernel side of things, and discovered how a CPU scheduler works while troubleshooting a FreeCAD problem. Optimising lead me to a non voluntary context switching problem that was remedied by using Nice/affinity/CPU isolation. That level of exploration into the scheduler was my lightbulb moment for what I was missing in embedded all along .
My main source for info on the scheduler ended up being some CS classes posted on YT by UC Berkeley. That lead me down the rabbit hole of trying to follow along with all the posted classes for CS. I would go back to school if I could, but I have a bad back that makes holding posture for any length of time difficult.
While following the Berkeley CS class lectures, I hit a few places where some specific detail just didn’t make sense. I came across a reference on Lemmy about offline AI, it piqued my curiosity, I found a ref about privateGPT using langchain to question documents, got a capable machine, and that lead me here.
Anyways, I have a challenge for you…
If you are not familiar with Forth, watch the following link. It is ultra simple and way beneath your skill level, but here’s the deal, the way Forth is setup on hardware here shows its real simplicity. Like, looking at the full ANS Forth implementations it looks complicated (IMO), but at a fundamental level, Forth is the simplest possible loop interpreter. https://www.youtube.com/watch?v=PY01_9dANd8
Maybe I am extremely naïve, but Forth Words seem like they could be integrated as a special token system in a LLM. What happens if a simple state machine is setup in Forth, a LLM is given an objective, and the forth interpreter is able to provide the mechanism to test branch and correct. So far, that could be done in any language easily, but what if the loop can write new words as unique tokens added into the LLM where it can call them. There is no syntax issue like other languages, Forth is a super simple language in that regard. All Forth systems include a word for bookmarking the dictionary too, and they can fall back. So the past is immutable to a certain extent. I haven’t found anyone that has tried to explore this idea yet. At a minimum I picture an easy way for a model to modify context on the fly using a Forth interpreter and its dictionary. More interesting would be a network of models promoted by Forth and collectively able to build their own code to reach an objective.
As far as LLM projects, I must have mentioned this already, but I want a LLM trained with the CS curriculum embedded as an individualized learning assistant that is completely open source. If I can get something working soon, I would like to build the database manually as I go. I have all the CS books, lectures, etc. I’d like to learn from both sides (CS/LLM) and see what is possible. I already know most of the information in general on the CS side, but in bits and pieces. My main issues are advanced math, complex project management, and advanced algorithms. I can read and modify most code, but don’t ask me to write anything from scratch outside of a bash script.
