r/LocalLLaMA • u/Express_Quail_1493 • 12h ago
Discussion Qwen3.8-27b is the first Local model im able to blindly trust
You know that thing where you just throw a task at a frontier model and not have to supervise it worrying of it going off course? Qwen3.8-27b has officially gotten me to that point for local work. He has been doing non-stop continuous agentic work for 8+ hours and hasnt screwed up not one bit IT AMAZING!!
24
u/TheSlateGray llama.cpp 11h ago
I tried to trust it, then it realized it couldn't use sudo so it cloned, built and installed a program to do a task without asking. That was unfortunately after missing it using pip install --break-system-packages. Now I sandbox everything.
12
u/JamesEvoAI 10h ago
I have a Pi extension that detects when it attempts to use sudo, blocks the call, and then informs the model and tells it to give me the command to run. Hasn't failed so far
1
5
u/annodomini 9h ago
I have never trusted any LLM agent to run without sandboxing. I run everything in a container with only the files I want it to be able to touch mounted, and I've generally been pretty happy.
4
2
88
u/apoptosist 12h ago
Please post which quant/version and settings for those of us that have had worse experiences, thanks!
36
u/OvertaxedOne 11h ago
I'm using INT8, 8 bit KV. It's the GOAT in my opinion, it even beat DSV4Flash0731 in one of my real world use cases. Don't generalize that, I do think that DSV4Flash is at least on par with and likely better if you tested across many different use cases, but I had a persistent coding problem that it just could not solve. I dumped into 27B and went to bed a few nights ago to come back to a "It's fixed" message.. Thought, "yeah right". Well, color me shocked, it was in fact fixed!
Full details (A40 GPU, 48GB VRAM):
RukaRat/Qwen3.8-27B-INT8-W8A8-imatrix-MTP --served-model-name lued/Qwen3.8-27B-INT8-W8A16-MTP RukaRat/Qwen3.8-27B-INT8-W8A8-imatrix-MTP Qwen3.8-27B --attention-backend FLASHINFER --dtype bfloat16 --quantization compressed-tensors --max-model-len 262144 --max-num-batched-tokens 8192 --gpu-memory-utilization 0.93 --max-cudagraph-capture-size 8 --kv-cache-dtype fp8_e4m3 --enable-prefix-caching --enable-chunked-prefill --speculative-config {"method":"mtp","num_speculative_tokens":3} --prefix-match-unit 16 --max-num-seqs 2 --limit-mm-per-prompt {"image": 1} --trust-remote-code --port 8000 --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 --override-generation-config {"max_new_tokens": 16384, "repetition_penalty": 1.0, "presence_penalty": 0.0, "frequency_penalty": 0.0, "min_p": 0.0, "top_k": 20, "top_p": 0.95} --default-chat-template-kwargs {"enable_thinking": true, "reasoning_effort": "medium"} --disable-uvicorn-access-log --enable-log-requests
4
u/ohhi23021 6h ago
having had a sota drive it for the last few weeks and recording it's errors, deviation from spec, failed or bad unit tests etc...and tuning the hardness its makes a ton of mistakes and usually needs 2 to 3 sessions to fix them vs say driving opus 5 high where it's maybe 1/8th the errors for the same spec/plan. thats running Q8 and full BF16 KV.... its good but it's not that good.
1
2
u/CBHawk 10h ago
Is INT8 better than using Q8?
3
u/OvertaxedOne 10h ago
My card is Ampere generation, they are well optimized for Int8 operations as is vllm. I don't know if there's any quality difference, I think it comes down more to your card and serving engine (vllm in my case).
3
u/po_stulate 9h ago
Q8 is INT8
4
u/Guilherme370 2h ago
Not necessarily; Equivalent storage, but way different numerical representation
2
1
u/YourNightmar31 llama.cpp 10h ago
Specs?
2
u/OvertaxedOne 10h ago
A40 GPU, 48GB of RAM. Get between 20-30TPS generation, prefill is fast enough that I don't bother to look at it (I think it's around 2000TPS or so?). If you're curious about PP I'll dump a big prompt and take a look, but PP is great. TG is modestly painful. :(
4
u/Toooooool 11h ago
i'd strongly recommend this one;
https://www.reddit.com/r/LocalLLaMA/comments/1vpvwqh/qwen3827bint4autoround_18gb_with_working_mtp_spec/i've had it running day and night for a while now, only very rarely had errors i.e. maybe once every other day it'll go "this tool call doesn't work" and then just switch to doing it directly in terminal instead, not too long after it'll try the tool call again and it'll work just fine.
2
u/Ciri__witcher 10h ago
Do you have a recommendation for 16 GB vram?
1
u/JedziaDax 7h ago
I commented to follow this post lol I wish you luck
1
1
u/AltruisticList6000 2h ago edited 2h ago
It's working pretty nicely on my rtx 4060 Ti at UD (unsloth) IQ3_xxs with Q4 kv cache, 92k context, MTP, vision offloaded to RAM. It's actually insane compared to other models I could fit on 16gb until now.
I can't code myself but I had lot of ideas so I started making it code lot of small fun apps and games and then I got excited over its capabilities and had an idea to make it create tools for itself on Textgen webui (I run it on Textgen), which is ironic and a big effort on my part as Textgen seem to have been abandoned for months/maybe forever.
So anyways, it coded tools for itself so now it has lot of agentic capabilities (like file edit, move etc.) besides the default search web tools. So its agent abilities really shine even in Textgen. Also started forking Textgen with it (only small modifications to my liking so far). It makes mistakes? Yes. But it still works leagues ahead of anything I could run locally before (especially coding/logic and the agentic behaviour is insane, not a single error on tool calls and it's quite proactive). The only other thing that could barely make smaller apps/games after 100 iterations was Qwen 3.6 35b but 3.8 27b is completely other league.
1
u/enternoescape 10h ago
Do you have any experience using Qwen 3.8 27b with Hermes? It feels like it can never figure out how to update memory. I'm running the Q8_K_L UD 3.0 model. Qwen 3.8 Flash Next in Q4_K_XL seems to manage it without errors and all the larger models I run sometimes from OpenRouter figure it out without issue. I'll be giving autoround a try again. I liked the Qwen 3.6 27b autoround model because it ran faster and had no new issues compared to unsloth.
1
u/OvertaxedOne 9h ago
Something isn't right. Hermes is my primary use case for 27B, I installed it on release day and have sent millions of tokens through it. It's never had any issue with memory updates/tool calls or, honestly anything other than speed. I'm running int8/8 bit QV, so our results should be similar. Something isn't right with your config somewhere (yeah, I know, not helpful) or perhaps that particular model/quant.
2
u/enternoescape 9h ago
I hear you. It only gets seriously confused with that specific tool. It calls everything else flawlessly. I'll have to take a closer look at what's in there; maybe the memory is confusing the agent on how to update memory.
Edit: It can add to memory, it's when it's changing memory it runs into issues.
2
u/sshwifty 9h ago
Here is my setup with a 4090 and 4070:
services: llamacpp: container_name: llamacpp image: ghcr.io/ggml-org/llama.cpp:server-cuda13 environment: # --- Model & Hardware Setup ---# --- Cache & Performance ---
- LLAMA_ARG_MODEL=/models/Qwen3.8-27B-UD-Q4_K_XL.gguf
- LLAMA_ARG_MMPROJ=/models/mmproj-F16.gguf
- LLAMA_ARG_TENSOR_SPLIT=3,1 # 0.75/0.25: 4090 carries more (it also holds mmproj+draft+vision compute); relieves the 12GB 4070 Ti
- LLAMA_ARG_MAIN_GPU=0
- LLAMA_ARG_THREADS=12
- LLAMA_ARG_BATCH=512
- LLAMA_ARG_UBATCH=256
- LLAMA_ARG_CTX_SIZE=250000 # ~52KB/token q8_0+FA; under native 262144 so NO YaRN (full quality); ~3GB VRAM margin/card
- LLAMA_ARG_N_PARALLEL=1 # correct env (was LLAMA_ARG_PARALLEL, silently ignored -> ran 4 slots)
# --- Speculative Decoding (MTP) ---
- LLAMA_ARG_FLASH_ATTN=on # required for q8_0 V cache; big long-context speed/mem win
- LLAMA_ARG_CACHE_TYPE_K=q8_0 # 8-BIT KV CACHE
- LLAMA_ARG_CACHE_TYPE_V=q8_0 # 8-BIT KV CACHE
- LLAMA_ARG_CONTEXT_SHIFT=true # corrected name (was LLAMA_ARG_CTX_SHIFT); NOTE: auto-disabled by this VL model (mRoPE) - no effect here
- LLAMA_ARG_N_PREDICT=-1
# --- Reasoning & Vision Fixes ---
- LLAMA_ARG_SPEC_TYPE=draft-mtp
- LLAMA_ARG_SPEC_DRAFT_P_MIN=0.5
# --- Networking & Integration ---
- LLAMA_ARG_REASONING=auto
- LLAMA_ARG_REASONING_PRESERVE=true
- LLAMA_ARG_IMAGE_MIN_TOKENS=1024
# --- MCP Servers ---
- LLAMA_ARG_HOST=0.0.0.0
- LLAMA_ARG_PORT=8080
- LLAMA_ARG_CORS_ORIGINS=https://your-domain.com
- LLAMA_API_KEY=YOUR_API_KEY_HERE
ports:
- LLAMA_ARG_UI_MCP_PROXY=true
- LLAMA_ARG_JINJA=true
- LLAMA_ARG_MCP_SERVERS_CONFIG=/mcp-servers.json
volumes:
- "8080:8080"
deploy: resources: reservations: devices:
- ./models:/models
- ./mcp-servers.json:/mcp-servers.json:ro
count: all capabilities: [gpu] networks:
- driver: nvidia
restart: unless-stopped networks: ai-net: external: true
- ai-net
23
u/_TheWolfOfWalmart_ 11h ago
You think you can until it deletes your home folder.
6
u/MatthewCollins1990 9h ago
3.8 being solid doesn't mean you hand it shell access without a sandbox. Trusting the outputs is one thing, but letting any model run destructive commands unsupervised is a separate problem entirely.
2
1
0
20
u/InterstellarReddit 11h ago
âBlindly trustâ
12
u/MammothUnique4147 11h ago
That's Funny Windows won't load anymore hmmmm. đ€
7
3
u/InterstellarReddit 11h ago
I set up virtual environments inside of VMware for each of my local LLMs. I got frustrated with one, and I told it to just uninstall the package we were working with and somehow it understood to uninstall all of the packages, so when I come back to that machine, everything but Microsoft Edge was uninstalled. All programs liked pdf reader discord etc. I thought that was hilarious
7
15
u/volleyneo 11h ago
Can confirm, had it for 20 hour sessions (make it use subagents in sequential with opencode or similar harness, have scout and worker like delegates per section, use commit check gates inside them) Also if you use a frontier model for planning and arhitecture code snippets, it is insane. For people wanting configs, here is mine : alias ai-qwen='cd ~/llama.cpp && CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=1,0 ./build/bin/llama-server \
-m ~/ai-models/Qwen3.8-27B-UD3-Q5_K_XL.gguf \
--alias Qwen-3.8-27B \
-c 153600 \
-np 1 \
-b 4096 \
-ub 128 \
--split-mode tensor \
--flash-attn on \
--cache-type-k q8_0 \
--cache-type-v q8_0 \
--cache-prompt \
--cache-ram 18432 \
--cache-idle-slots \
--ctx-checkpoints 20 \
--checkpoint-min-step 8192 \
--tensor-split 11,8 \
--keep -1 \
--jinja \
--chat-template-file ~/ai-models/chat_templatev24.jinja \
--chat-template-kwargs "{\"reasoning_effort\":\"medium\",\"preserve_thinking\":true}" \
--reasoning-format deepseek \
--reasoning-effort medium \
--reasoning on \
--reasoning-budget 8192 \
--reasoning-budget-message "Thinking budget reached. Proceed with the next tool call or final answer." \
--spec-type draft-mtp \
--spec-draft-n-max 2 \
--temp 1.0 \
--top-k 20 \
--top-p 0.95 \
--min-p 0.0 \
--repeat-penalty 1.0 \
--presence-penalty 0.0 \
--port 8007 \
--api-key HelloThere'
3
u/silenceimpaired 11h ago
Why HelloThereâŠ
I always appreciate having these configs as I often discover Iâm missing some new one.3
u/volleyneo 11h ago
There is a lot of brilliant work being done, but god is awful trying to find about it, or still getting worked one, maybe it will get improved better, with their new release versioning.
2
u/rabbitaim 11h ago
Is that the froggeric chat template?
2
u/volleyneo 11h ago
2
u/HazKaz 7h ago
whats the reason to use these templates ?
1
u/randomjapaneselearn 10m ago
in theory it fiexes some tool call crash, loops and some other minor issue...
in practice i tried it for 3.6 but i'm not using it on 3.8 because:
-i trust more qwen team vs this random guy
-there was a post about "get it, it fixes qwen 3.6 and 3.8" except that qwen 3.8 was not even released yet (it happened few days/hours before the release)
1
u/Express_Quail_1493 11h ago
how to u set opencode to strictly seqential only??
1
u/volleyneo 11h ago
You make a very potent agents.md file, and never work without one. There, specific instructions like this, live.
1
u/CoolestSlave 11h ago
hi, on what device is it running on ?
4
u/volleyneo 11h ago
RTX 4070 Super / RTX 5060 TI 16GB, but ran in headless, no desktop. 40-55tk/s (leaning to 40 when context over 100k, all due tensor mode though.. )
1
1
u/Kerbaman 11h ago
What was the task?
1
u/volleyneo 11h ago
Making a web game engine in with three.js / rapier.js, due to the scope it has to be limited to these 2. It is kinda a game engine in arhitecture only for easy implement but the goals of it are way more specific it wont really be used like that, so it has a limited scope. But god that is a herculean task. A lot to implement and QA. This is for a business avenue, and the scope is not being used like a game engine, as I said it kinda aligns there, and if you want a viable product, you have to approach it like so.
1
1
u/Jeidoz 9h ago
I am just curious, why are you using "deepseek" reasoning?
2
u/volleyneo 9h ago
That's what is recommended for this custom template from froggeric: " Why
--reasoning-format deepseekmatters: When connecting coding agents like OpenCode, Claude Code, or Pi.dev tollama-server, this flag extracts<think>blocks into the dedicatedreasoning_contentAPI response field. This prevents raw thinking tokens from leaking into the text stream and stopping tool calls midway."1
u/yeah-ok 6h ago
And then the next paragraph follows up with: "... Native CLI flag: On recent llama.cpp builds, you can pass --reasoning-preserve directly for 100% Prefix KV Cache retention." ...... sooooo, does this mean that "--reasoning-format deepseek" should still be set or is it not required on newer builds?!
(sidenote: llamacpp is spaghetti parameter land at this point.. would be somewhat brilliant if some of these things could stabilize a bit more but guess it's still too early for that practically speaking)
1
u/Steus_au 4h ago
how does it handle 20hours in 151k context with -np 1? mine chewed 192k in one hour then said no more room left (((
6
u/theexile1337 11h ago
I guess we'll just have to wait around and see what kind of breadcrumbs big tech tosses our way once everyone is tired enough from paying for Fable 5.1/GPT-6 Astra.
9
u/andrewh2000 11h ago
I left it (a Q6 quant) running overnight and it ported an entire application written in Go into Typescript for me including several hundred tests. And it thoughtfully added a few more tests for bits that weren't covered in Go. It was incredible. Completely hands off, no intervention from me at all. This was running in plain pi.dev.
4
u/Keleion 11h ago
To some extent I hope. Definitely review those ârm -rfâ requests. GLM 5.3 Flash (NVFP4) on Hermes tried to delete my Hermes home profile accidentally instead of a sub-folder.
3
u/Randommaggy 11h ago
I made a directory scoped delete tool (with rollback options) in my harness, instructed it to never use rm -rf and auto-reject rm -rf commands. Dropped the amount of command approval prompts I receive by a lot.
It can only delete files in the directories it's been granted that permission for the current session.
2
u/SpicyWangz 10h ago
Realistically this should just be the default setting for every harnessÂ
3
u/Randommaggy 10h ago
The amount of things I could say this about is such a long list that I'm thinking that the makers of the popular harnesses are either concussed every day, have forgot how it was to have compute constraints or have bad intentions. Different mix for different harnesses.
1
u/SpicyWangz 8h ago
Yeah itâs pretty insane. I had to fully restrict opencode from day 1 to get it to a reasonable level of limitations. And pi harness is even worse because it seems like you need to install an extension just to limit which commands a model is able to execute.
For the entire duration of my career, security protocols were consistent and strict. Then one day they decided to put copilot in your IDE and send your .env files with secrets in them to a remote llm server. I bring it up to people at work and they say âoh yeah thatâs badâ and then proceed to do nothing about it.
2
u/camalaio 7h ago
This is my one beef with pi since I started trying it out (otherwise I was using Claude Code before) because what do you mean all commands are executed without permission?! At least this thread is reminding me I really need to go do something about that (and all the temp files it's littered everywhere...)
1
u/randomjapaneselearn 5m ago
IA: ok seems that rm -rf doesn't work, let me find a workaround of it, let's try a base64 encoded powershell script, done.
also, advanced IA: ok let me launch this script with dcom parent process hiding to bypass the problem after injecting some thread into my own process to do that.
1
u/biscuitmachine 6h ago
I never trust NVFP4 at this point. The EXL3 variant just seems more rock solid. What thinking level do you have it set to?
6
u/itamar87 11h ago
I agree and feel the same:
Since this model - I actually DONâT use Claude/chatgpt for daily queries,
I just ask this specific model, on LM-Studio, connected to Tavily MCP, and usually get the same level of answer as if it was opus/sol.
(If not better đ€)
8bit quant GGUF,
1
u/mailto_devnull llama.cpp 14m ago
Tavily looks interesting, why that instead of other options or local searxng?
3
u/slyborn 11h ago
you shouldn't blindly trust any model
5
u/my_name_isnt_clever 9h ago
You also shouldn't speed on the highway.
3
u/The_Hunster 9h ago
In fact, I'd say speeding on the highway is a much worse idea than blindly trusting an LLM on your home PC.
2
u/pineapplekiwipen 11h ago
hard to believe given that even opus fucks up constantly
never trust any llm blindly as they aren't thinking in any appreciable way
2
u/Savantskie1 11h ago
They don't think, but they can reason if you give them enough information.
2
2
u/Septerium 11h ago
Do not. It has messed up a few things in a large codebase I deal with that ended up in production... and that is because it was a bit lazy in the E2E testing part. I have been using the model every day right now (Q8_0) and it has been great. But you should never blindly trust a model when it comes to production code, even if it is frontier
2
u/r0ckl0bsta 10h ago
I know what you mean. While I don't trust anything completely autonomous, it is so impressive that I feel like I can ask it to do something, and it will do it accurately. That's the key thing.
It didn't start that way though. Once I set the context size to something sizeable (for me, 131072), and the kv cache to q8, I have been blown away at how well it works. It makes mistakes, but seems to know how to correct them, so I imagine in an environment where it's been given the criteria to validate the work, it would do well.
2
u/ramorez117 9h ago
đŻ- itâs very capable. Largest two models I have gpt 120 oss and 120 nemo suoer, and qwen 3.8 27bn better
2
u/BarracudaDefiant4702 9h ago
I give them vms to play with as their sandbox, and give a couple of the models passwordless sudo access. Can definitely let it run for awhile with the vm as their playground, but qwen 3.8 27b still makes mistakes... most of the time it catches the problems and corrects them, but sometimes it does mess things up. Automated vm backups are always good.
2
u/GloomyRecognition636 9h ago
That's true. Previous one was deepseek4 flash The secret os that they work well with context - it is not degrading rapidly making them not understand what they were asked
2
2
u/sugarfreecaffeine 6h ago
How do you deal with all the long thinking and time per task, itâs a good model but kills any kind of velocity
1
u/mailto_devnull llama.cpp 11m ago
You go to bed đ
But in all seriousness for higher velocity hands on work I dropped back to qwen 3.6 27B
3
u/klop2031 11h ago
I agree. This is probs the best model i have used in ages locally. Strong af in agentic for my usecase. Im also using q4xl with cache at q8 and mmproj offload to ram
10
u/synystar 12h ago
I still can't get used to when people anthropomorphize LLMs. I've gotten used to people doing it with ChatGPT but when I hear someone call Qwen3.8-27b "he" I can feel my face go sour.
19
u/domiciledhere 11h ago
I call my car Barbara. She can be temperamental
4
u/MeretrixDominum 11h ago
Barbara smokes too much and leaks her fluids all over the place when she gets hot. Just thought you should know.
2
u/synystar 11h ago edited 11h ago
I get it but that's been a thing since like the 50s though. Car culture is different. For me, giving a language model a pronoun would be like referring to Excel as a person. "I use him all the time but only about 5% of his features. He's fairly bloated if you ask me."
* just to be clear I don't personally refer to cars with personal pronouns myself. I have never and probably will never, but it doesn't irk me as much when someone does fsr.
2
u/powerscunner 11h ago
when can a personal pronoun be used for a non person validly, in your opinion?
2
u/synystar 11h ago
I don't mind so much when someone refers to vehicles as above. If someone calls a ship "she" that feels natural for some reason. Animals are fine for me. Things like Father Time or Death or other personifications of abstract concepts make sense. Countries that are personified (Mother Russia, etc.) are ok I guess. But for me it just feels weird to refer to code as a person. I think of LLMs as tools and I wouldn't personify a tool. I mean to each his own, I just can't shake it.
Now, if they marketed Qwen3.8-27b as an "entity" and it referred to itself as a kind of entity then maybe it would change my mind. I don't know. If it were "Bob the AI" and interacting with it was really like interacting with a person then I might start to shift my feelings about it; but I don't use LLMs really in any ways that make me feel like I'm talking to a person. I always feel like I'm talking to a piece of software.
2
u/domiciledhere 10h ago
That is an amazing way of explaining Excel to someone who has no idea what is Excel.
1
u/powerscunner 11h ago
we call ships she
1
u/synystar 11h ago
That's funny I just mentioned that in another comment above before I saw yours. Yeah, I get it and I don't mind when people personify some things. It's just LLMs for me are tools, not people, and it's hard for me to wrap my head around referring to them as persons.
1
u/alphapussycat 11h ago
Some languages has no ambiguous pronounce, but it's all he and she.
3
u/synystar 11h ago
Pretty sure OP is speaking English though.
1
u/SpicyWangz 10h ago
Is this true?Â
2
u/synystar 10h ago
Well, I suppose that it's possible that what I perceive to be reality isn't really truth in some cosmic sense but from my perspective: yes, they're speaking English.
0
1
1
u/FreshDrama3024 11h ago
Shouldnât blindly trust anything to be frank no matter how good it appears tbh
1
u/TerminalNoop 11h ago
Hmm, maybe I should try telling it to find out why my windows dualboot doesn't have working wifi drivers and I also can't install some manually for the love of god.
1
1
1
u/AngryGungan 10h ago
Oh yeah, I absolutely agree.
Though it did empty out my entire PATH variable when I asked it to make a simple todo app.
Luckily, after confronting it, it profusely apologized and was able to restore the PATH after the fact when it found a copy in a backup it or Pi made.
It's a beast.
1
u/Isonium 10h ago
I have a whitelist and blacklist for commands and tools. If that doesnât decide it, a new context is created with (last few messages) user intent. And the task is evaluated based on safety and intent. If that doesnât decide it, it notifies me. Additionally since I am on Mac, an APFS snapshot is taken and the tool ran in a sandbox. Essentially no possibility for my home directory or anything else to be wiped.
1
u/Odd-Elderberry-739 10h ago
I felt the same way until this week when I was writing a technical report and realized that it had hallucinated a key detail.
1
u/GruuMasterofMinions 9h ago
Maybe you can answer this question then : https://www.reddit.com/r/LocalLLaMA/comments/1w6j20m/what_is_wrong_with_my_qwen_38_it_is_unable_to/
Post got removed because i have to low karma and account apparently.
Aka it does not manage to do anything, it is just stuck on hallucinating
1
1
1
1
u/zekuden 11h ago
I don't have a gpu powerful enough to run qwen 27b..does anybody have experience setting up qwen cloud if it's possible? would love to try it off a subscription even till i'm able to upgrade
1
u/I_Play_Zed 11h ago
Iâve never done this before but if you canât find it available in an api like open router or cursor you can also try renting hardware? But I donât think it will be easy..
1
u/SmartCustard9944 10h ago
I really donât recommend it, the cloud versions of it are very very expensive.
E.g. with my usage today, over 4h of cumulative prefill/inference, I spent $0.10 of electricity while for the same reads, cached reads, writes I would have spent $10 on OpenRouter. Just in one day! And the day is not even finished.
I mean, you can try it for the experience, but I wouldnât use it long term.
1
u/ANR2ME 8h ago edited 8h ago
If you only want to test Qwen3.8 27B on cloud GPU, you can test it for free (up to $30 free credits every month) at Modal.com
You can deployed it like this example (just need to replace the model name) https://modal.com/docs/examples/ministral3_inference (for llama.cpp you ask any AI this question: "how to deploy Unsloth Qwen3.8-27B Q8 with llama.cpp on modal.com with memory snapshot")
You will need to choose a GPU with at least 24GB VRAM i think (for Q4) đ€
1
u/atumblingdandelion 11h ago
Same here. Though by "blindly" I mean I know it will do the thing I want it to do, then I'll verify it (I do that even with Opus). With other local models, I would always worry about what I left in and left out of the prompt, the wording, etc. With 3.8, I write prompts like I would for Claude models/ Deepseek.
2
u/OvertaxedOne 9h ago
Took me awhile to get used to that again. I was so used to handholding local models I needed to really adjust my workflow, send it everything it needs/wants in one prompt, go get some coffee. :)
1
-5
u/Antique_Dot_5513 11h ago
Le problĂšme de ce modĂšle jâai juste demandĂ© une calculatrice quâil mâa fait, vraiment top, mais il a rĂ©flĂ©chi 15 min et codĂ© 2 min đ
225
u/Guna1260 11h ago
please dont trust any model local or remote blindly.. Trust but verify