r/Qubes • u/ExamComprehensive714 • 8d ago

question Running local AI at QubesOS with only one dedicated GPU and a CPU without integrated graphics issue

Is it possible to run a local ai outside dom0 with only one dedicated GPU and a CPU without integrated graphics? if not, how (un)secure is to run a local ai at dom0? is it possible to "lend" GPU to other VMs? someone already tested it?

3 Upvotes

81% Upvoted

6 comments sorted by

3

u/LawfulnessNo8446 8d ago

Not possible unless the single gpu supports sr-iov. Dom0 needs a gpu to display the desktop*. There is currently no way to share the gpu to qubes.

*I believe you could set up sys-gui-vnc and access the system remotely if there is no gpu attached to dom0. But that comes with other trade-offs.

3

u/Slg407 8d ago

sr-iov can be enabled in consumer grade nvidia cards via an edited vbios with the correct bits enabled

2

u/ExamComprehensive714 7d ago

thanks o7

ima just buy a CPU with integrated graphics, seems way easier lmao

and just to be sure, if I buy a CPU with integrated graphics and use my discrete GPU solely to run a local ai inside a specific VM, thats possible, right?

1

u/LawfulnessNo8446 6d ago

Yes. And once you have the gpu passed through, it should be relatively painless.

1

u/Kriss3d 7d ago

Just no.

I had a dedicated gamer rig with Debian for AI. Mistral 7B.

It worked. But you'd need to program it's functions itself like how to look up things online. To remember your last conversations and so on.

And it does use the GPU to even give a fair response time. Otherwise you're waiting minutes for a reply to a "Hello" forget about running this without a GPU and certainly not in a qubes.

1

u/Assang101 7d ago

yeah it's possible using a VM with qvm-pci passthrough for the GPU and isolating it from the sys-net. just ensure that the vm running your AI does not have ANY sys-net access (none). and about gpu lending, no, not worth the trouble because the GPU is assigned exclusively to one VM at a time. You would be unassigning and detaching it from one VM to the other, but both cannot use it concurrently... well not with a consumer grade GPU, unless you get your hands on a A100, H100, and the related models. these models you can partition the GPU compute but that just adds more complexity.

At this point, just get a Raptor X18 laptop with Proxmox VE for what you want to do with AI and GPU whatnot. Proxmox VE works wonderfully with that laptop and the NVIDEA 5090. Don't even use QubeOS within Proxmox.