tensor-parallelism

🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading

A scalable, agentic-first, and HuggingFace-native RL framework for research (9k lines).

InternEvo is an open-sourced lightweight training framework aims to support model pre-training without the need for extensive dependencies.

Slicing a PyTorch Tensor Into Parallel Shards

LLM inference engine from scratch — paged KV cache, continuous batching, chunked prefill, prefix caching, speculative decoding, CUDA graph, tensor parallelism, OpenAI-compatible serving

Decentralized LLMs fine-tuning and inference with offloading

Qwen3.8-Flash-Next (MoE) and Qwen3.8-27B on one or two 16 GB RTX 50-series GPUs: experts offloaded to CPU, two-GPU tensor parallel. 64 GB of RAM is enough for Flash-Next. Formerly ninfer-tp2.

A custom LLM inference engine for low-cost GPUs: NVIDIA cards from a 1 GB Quadro K420 to an RTX 3080, the AMD Radeon 780M and the CPU. Runs the Bonsai 2B to 27B models and Nemotron with an OpenAI-compatible server.

Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*

An Efficient and Versatile Inference Engine for Distributed LLM Serving

llama.cpp optimized for NVIDIA Tesla V100 (SM70): Qwen3.8-Flash-Next with MTP (prefill 3402 tok/s at 200K context on 6x V100-16GB; also runs on 2x V100-16GB) and Qwen3.8-27B with DFlash2 (decode up to 220 tok/s); 2-6 GPUs, 262K-512K context, multimodal, concurrent serving, built-in web UI.

Switchless RoCE collectives, NCCL profiles and fabric tooling for DGX Spark inference recipes

Simulate distributed LLM training and inference across GPU clusters in your browser. Memory, throughput, cost, and parallelism strategy planning.

JORA: JAX Tensor-Parallel LoRA Library (ACL 2024)

A distributed training framework for large language models powered by Lightning.

vLLM for NVIDIA CMP 170HX (Ampere sm_80): fast, repeatable model serving on one card or several

Tensor-parallel DeepSeek V4 Flash inference on dual AMD Strix Halo over OdinLink USB4/TB5 RDMA or Mellanox RoCE v2.

A decision library for running local LLMs on multi-GPU machines without NVLink: parallelism, communication, quantization, speculative decoding, KV cache. Every conclusion carries its evidence grade and reproducible conditions.

RDMA fork version of Strix Halo inference engine. Qwen Flash Next Q4_K_XL: >2100pp …

GLM-5.3-Flash BIZ for DGX Spark-class GB10 systems. 3.x serves the XSFP4 checkpoint or NVIDIA's pinned NVFP4 one with TensorFold's native Zig/CUDA engine (TP=2 or TP=3, MTP drafts equal serial, tool calls, structured output, images, video). Apache-2.0 code, weights fetched separately. BIZ = business-use intent, not support or certification.