• 0 Posts
  • 35 Comments
Joined 1 year ago
cake
Cake day: July 20th, 2023

help-circle






  • RedoxOS would likely never become feature complete enough to be a stable, useful and daily-drivable OS. It’s currently a hobbyist OS that is mainly used as a testbed for OS programming in Rust.

    If the RedoxOs devs could port the Cosmic DE, they’d become one of the best Toy OS and maybe become used on some serious projects . This could give them enough funds to become a viable OS used by megacorps on infrastructures where security is critical and it may lead it to develop into a truly daily drivable OS.












  • Completely forgot to tell you to only use quantized models. Your pc can run 4bit quantized versions of the models I mentioned. That’s the key for running llms on at consumer level hardware. You can later read further about the different quantizations and toy with other ones like Q5_K_M and such.

    Just read phi-3 got released and apparently it’s a 4B that reach gpt 3.5 level. Follow the news and wait for it to be add to ollama/llama.ccp

    Thank you so much for taking the time to help me with that! I’m very new to the whole LLM things, and sorta figuring it out as I go

    I became fascinated with llms after the first AI booms but all this knowledge is basically useless where I live, so might as well make it useful by teaching people what i know.



  • Yeah, it’s not a potato but not that powerful eaither. Nonetheless, it should run a 7b/8b/9b and maybe 13b models easily.

    running them in Python with Huggingface’s Transformers library (from local models

    That’s your problem right here. Python is great for making llms but is horrible at running them. With a computer as weak as yours, every bit of performance counts.

    Just try ollama or llama.ccp . Their github is also a goldmine for other projects you could try.

    Llama.ccp can partially run the model on the gpu for way faster inference.

    Piper is a pretty decent very lightweight tts engine that can be directly run on your cpu if you want to add tts capabilities to your setup.

    Good luck and happy tinkering!