Flexible model sharding
Assign GGUF model layers based on the RAM and VRAM limits you choose, so multiple devices can contribute within their available capacity.
ShardAi pools idle RAM and VRAM across multiple devices, shards large language models, and runs inference over a peer-to-peer network.
Open source · Tauri desktop client · Python inference engine

ShardAi distributes model workloads across the devices you already have, bringing them together in a collaborative AI inference pipeline.
Assign GGUF model layers based on the RAM and VRAM limits you choose, so multiple devices can contribute within their available capacity.
Discover devices on your local network with mDNS/UDP, or connect over the internet through a signaling tracker.
The Tauri desktop app combines React and Rust, while the Python core engine uses llama.cpp for inference and sharding.
Automatically download and cache the weights for assigned model layers, directly from Hugging Face.
Nodes pass serialized, compressed tensors along the pipeline, processing the model one stage at a time.
ShardAi is MIT-licensed open-source software. Explore the architecture, follow progress, and contribute on GitHub.
Each node contributes its available computing resources. Together, the nodes form a distributed inference pipeline.
Devices register their available resources with a signaling tracker and find other peers on the network.
The engine downloads the required weights, then loads the layers assigned to each device.
Each peer computes its part and passes tensors to the next node in the sequence.
Discover the technology, follow the roadmap, and help build ShardAi in the open.