System components
DFD
Storage nodes that persist data chunks to local disk. Each node exposes a gRPC API for chunk operations and reports health status to FoundationDB.
MDS
Metadata Service managing the filesystem namespace. Handles inode allocation, directory structure, and stripe placement through atomic FoundationDB transactions.
Custodian
Maintenance daemon responsible for data durability and storage efficiency. Handles failure detection, data repair, garbage collection, and rebalancing.
Client library
Internal orchestration layer used by Artifact Store and Platform. Handles node selection, data encoding, and integrity verification.
The client library is an internal component. External services interact with DFS through Artifact Store or Platform APIs.
Data model
DFS organizes data using a POSIX-like filesystem model with inodes, directories, and files. Each file consists of one or more stripes, where each stripe contains up to 1 MiB of data distributed across storage nodes.Write operations
- The client determines the appropriate durability policy based on file size
- Data splits into 1 MiB stripes
- Each stripe writes to multiple DFD nodes according to the durability policy
- MDS records stripe metadata and updates the inode atomically
Read operations
- The client resolves the file path through MDS to obtain stripe locations
- Chunks are fetched from DFD nodes in parallel
- For erasure-coded files, missing shards reconstruct from parity data
- The client verifies checksums and streams data to the caller
Durability policies
DFS provides two durability strategies that balance storage efficiency against fault tolerance.Replication
Files of 3 MiB or smaller use replication by default. Each chunk writes to three separate storage nodes. A read succeeds when any single replica returns valid data.Reed-Solomon erasure coding
Files larger than 3 MiB use Reed-Solomon encoding. Data splits into shards with additional parity shards for reconstruction. This approach reduces storage overhead while maintaining fault tolerance.
The client uses SIMD-accelerated encoding and decoding for Reed-Solomon operations.
DFD
DFD (D for Disk) nodes form the storage layer. Each node manages a local data directory and serves chunk operations over gRPC.Storage layout
Chunks organize in a two-level directory structure using the first four hex characters of the chunk UUID:.chunk file contains the raw data. The .meta file stores the CRC32C checksum computed by the client at write time.
API operations
Health reporting
Each DFD node writes a health record to FoundationDB every 15 seconds containing:- Current node state (healthy or dead)
- Last-seen timestamp in nanoseconds
- Network address for client connections
Configuration
MDS
MDS (Metadata Service) provides the transactional metadata layer for the filesystem. It manages namespace operations through atomic commits against FoundationDB.Data organization
MDS stores metadata in dedicated FoundationDB subspaces:API operations
Commit operations
TheCommit API accepts multiple mutations that execute atomically:
MDS validates durability policies during inode creation:
- Replication factor must be at least 3
- Reed-Solomon requires minimum 5 data shards and 3 parity shards
- Directories cannot specify durability policies
Configuration
Custodian
Custodian is the maintenance daemon responsible for cluster health, data durability, and storage efficiency. It runs continuous background processes that detect failures, repair under-replicated data, reclaim unused storage, and balance data distribution across nodes.Responsibilities
Node failure detection
Custodian scans DFD health records at a configurable interval (default: 5 seconds). A node is marked dead when its last heartbeat exceeds the timeout threshold (default: 45 seconds). When a node failure is detected:- Custodian updates the node state to
Deadin FoundationDB - The node is removed from available read and write indexes
- Repair tasks are queued for all stripes that resided on the failed node
Data repair
The repair scanner periodically examines stripe metadata to identify durability violations:- Replication: Stripes with fewer than the required number of healthy replicas
- Reed-Solomon: Stripes missing data or parity shards beyond the reconstruction threshold
Garbage collection
DFS uses a two-phase garbage collection process: Metadata GC scans the inode subspace for entries with zero reference count. After a configurable grace period, these inodes and their associated stripe metadata are deleted. Chunk GC identifies orphaned chunks on DFD nodes—chunks that exist on disk but are not referenced by any stripe metadata. The process builds a set of valid chunk IDs from MDS, then each DFD node compares its local inventory against this set and deletes orphans. A grace period protects chunks that are newly written but not yet committed.Rebalancing
When storage utilization becomes uneven across nodes, Custodian migrates chunks from nodes approaching capacity limits to nodes with available space. This prevents hotspots and ensures write operations can proceed without capacity-related failures.Task queue
Custodian coordinates all maintenance work through a distributed task queue backed by FoundationDB. The queue provides:- Configurable concurrency (default: 16 parallel tasks)
- Automatic retries with exponential backoff (default: 5 attempts)
- Visibility timeouts to prevent duplicate processing
- Priority levels for failure-triggered repairs
Configuration
Client library
The client library orchestrates distributed reads and writes across DFS components. It is used internally by Artifact Store and Platform.Responsibilities
- Maintains a cache of healthy DFD nodes from
ListWriteServers - Selects target nodes for chunk replication
- Computes Reed-Solomon parity shards using SIMD acceleration
- Reconstructs missing shards during reads
- Verifies CRC32C checksums end-to-end
- Manages directory creation and file metadata updates
Durability path options
The client supports inline durability specification in file paths:Deployment
Minimum requirements
A functional DFS deployment requires:- One MDS instance
- Three DFD nodes (minimum for replication factor 3)
- Access to a FoundationDB cluster
- One Custodian instance for health monitoring