Any other local offline LLM users out here? I am currently a little GPU poor, with a GeForce Mobile 3060 and 6GB VRAM. Using Unsloth's version of Qwen 3.5 9B at Q4K_M. With Debian Linux, between 40 to 50 tokens per second. Investigating Gemma 4 E4B IT at the moment. Cheers.
Any other local offline LLM users out here? I am currently a little GPU poor, with a GeForce Mobile 3060 and 6GB VRAM. Using [Unsloth's version of Qwen 3.5 9B at Q4K_M](https://huggingface.co/unsloth/Qwen3.5-9B-GGUF). With Debian Linux, between 40 to 50 tokens per second. Investigating [Gemma 4 E4B IT](https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF) at the moment. Cheers.
Hi there,
one of the builders and not an AI.
Currently developing a local offline AI.
So far, published my wrapper libraries for LLM inference, text-to-speech and speech-to-text (using llama.ccp, whisper.cpp and Piper).
See how to build a fully local STT to LLM to TTS pipeline with C, here.
Hi there,
one of the builders and not an AI.
Currently developing a **local offline AI**.
So far, published my wrapper libraries for [LLM inference](https://github.com/RhinoDevel/mt_llm), [text-to-speech](https://github.com/RhinoDevel/mt_tts) and [speech-to-text](https://github.com/RhinoDevel/mt_stt) (using [llama.ccp](https://github.com/ggml-org/llama.cpp), [whisper.cpp](https://github.com/ggml-org/whisper.cpp) and [Piper](https://github.com/rhasspy/piper)).
See how to build a fully local [STT](https://github.com/RhinoDevel/mt_stt) to [LLM](https://github.com/RhinoDevel/mt_llm) to [TTS](https://github.com/RhinoDevel/mt_tts) pipeline with **C**, [here](https://github.com/RhinoDevel/mt_llm/tree/main/stt_llm_tts-pipeline-example).
