80 tokens/sec on two mismatched GPUs — and a thread asking if self-hosting still makes sense (imil.net)
The trick is llama.cpp with speculative decoding and a tuned tensor split across two architectures Nvidia never meant to pair. The catch a reader measured: about 700W under load, enough to heat the room.
The thread turned into a referendum. One reader pays roughly $3 per million tokens in the cloud, unquantized, for the same model; another mourns that the run-your-own-hardware ethos is 'basically dead.'
Sign in to save