Four M5 Ultra Mac Studios linked over Thunderbolt 5 run an 816B-parameter model at 36 tokens a second (machinelearning.apple.com)
Apple's MLX team tested tensor parallelism over RDMA on Thunderbolt 5: four machines decode at 36.73 tokens/s at 8,192 tokens of context, against 15.74 with two-machine pipelining. A single M5 Ultra is about 1.6x faster at decoding than the M3 Ultra.
Sign in to save