Skip to content
Get packs

3FS

A shared distributed file system built for AI training and inference data, using SSDs and RDMA networking.

GitHub Download source
Stars
10.4k
Forks
1.1k
Licence
MIT
Language
C++
Last update
6 months ago
Type
Other

What it is

3FS (Fire-Flyer File System) is a distributed file system, meaning files are spread across many storage machines and presented to applications as one shared storage layer. It is designed for AI workloads such as training, data preparation, checkpointing, and LLM inference caching. The README says it uses modern SSDs and RDMA networks for high throughput and exposes a standard file interface, so applications do not need a new storage API.

  • Serve training samples to many compute nodes with random access, avoiding separate prefetching or shuffling steps
  • Write high-throughput parallel checkpoints during large-scale training
  • Store and organize outputs of data analytics pipelines in hierarchical directories
  • Use as a lower-cost, larger-capacity alternative to DRAM-based KV caching for inference

Great for

  • Teams running large AI training clusters that need shared, high-throughput storage
  • Engineers building data pipelines that write many intermediate files across compute nodes
  • Infrastructure teams evaluating storage for LLM inference caching
  • Operators comfortable building and running a distributed system from source

Not ideal for

  • Individuals or small projects wanting a simple file share, since it requires a cluster of SSDs and RDMA networking and is designed for large deployments
  • Users without systems administration experience, because setup involves multiple dependencies and a cluster deployment guide
  • Anyone needing a managed, packaged product, since the README describes only build-from-source and Docker build images
Step-by-step guide: how to use 3FSInstall, first run, examples, tips and FAQs

How to get it

  1. 1

    Clone and build from source

    You'll need: Ubuntu 20.04/22.04, openEuler, or OpenCloudOS/TencentOS (or Docker build images) · CMake and a C++ toolchain (clang-14 or gcc 10/11+ as listed in README) · libfuse 3.16.1 or newer · FoundationDB 7.1 or newer · Rust toolchain, minimum 1.75.0, recommended 1.85.0 or newer

    Clone the repository

    git clone https://github.com/deepseek-ai/3fs

    Check out submodules and apply patches

    cd 3fs && git submodule update --init --recursive && ./patches/apply.sh

    Configure the build

    cmake -S . -B build -DCMAKE_CXX_COMPILER=clang++-14 -DCMAKE_C_COMPILER=clang-14 -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DSHUFFLE_METHOD=<method>

    Build

    cmake --build build -j 32
  2. 2

    Run it

    • Install the system dependencies listed for your OS in the README
    • Install libfuse, FoundationDB, and the Rust toolchain
    • Clone the repository and check out submodules
    • Build with CMake, choosing a consistent SHUFFLE_METHOD value
    • Follow the setup guide in deploy/README.md to run a test cluster
  3. 3

    Get the most out of it

    • Read the design notes in docs/design_notes.md to understand the architecture before deploying
    • Follow deploy/README.md for the test cluster setup
    • Use the fio engine for USRBIO in benchmarks/fio_usrbio to benchmark your own hardware
    • Review the USRBIO API reference in src/lib/api/UsrbIo.md if you need a direct high-performance API

Downloads

Source code (.zip)main branch · straight from GitHub

Strengths

  • Uses the standard file interface, so existing applications need no new storage API
  • The README reports strong consistency via Chain Replication with Apportioned Queries (CRAQ)
  • Published benchmarks cover large clusters, including about 6.6 TiB/s aggregate read throughput on a 180-node cluster

Trade-offs

  • Requires significant hardware (SSDs, RDMA-capable networking) and several dependencies such as FoundationDB and Rust, so setup is complex
  • Build compatibility depends on compiler choice; the README warns binaries from different compilers may be incompatible unless the shuffle method is pinned
  • Benchmark results come from the authors' own large test clusters, which most readers cannot reproduce

Stats synced from GitHub. The assessment above was drafted with AI from the project's README on 2026-10-29 and may miss recent changes — check the README before relying on a command. MIT licence: free to use, modify and distribute, including commercially, as long as you keep the copyright notice.