Distributed File System for chunked, replicated artifact storage
DFS is a distributed file system designed for storing large artifacts across multiple storage nodes. It handles data chunking, replication, erasure coding, and integrity verification. All other Mistrive services build on DFS for persistent storage.
Storage nodes that persist data chunks to local disk. Each node exposes a gRPC API for chunk operations and reports health status to FoundationDB.
MDS
Metadata Service managing the filesystem namespace. Handles inode allocation, directory structure, and stripe placement through atomic FoundationDB transactions.
Custodian
Maintenance daemon responsible for data durability and storage efficiency. Handles failure detection, data repair, garbage collection, and rebalancing.
Client library
Internal orchestration layer used by Artifact Store and Platform. Handles node selection, data encoding, and integrity verification.
The client library is an internal component. External services interact with DFS through Artifact Store or Platform APIs.
DFS organizes data using a POSIX-like filesystem model with inodes, directories, and files. Each file consists of one or more stripes, where each stripe contains up to 1 MiB of data distributed across storage nodes.
Files of 3 MiB or smaller use replication by default. Each chunk writes to three separate storage nodes. A read succeeds when any single replica returns valid data.
Files larger than 3 MiB use Reed-Solomon encoding. Data splits into shards with additional parity shards for reconstruction. This approach reduces storage overhead while maintaining fault tolerance.
Property
Value
Default configuration
6 data + 3 parity shards
Storage overhead
~1.5×
Fault tolerance
Survives loss of any 3 nodes
The client uses SIMD-accelerated encoding and decoding for Reed-Solomon operations.
MDS (Metadata Service) provides the transactional metadata layer for the filesystem. It manages namespace operations through atomic commits against FoundationDB.
Custodian is the maintenance daemon responsible for cluster health, data durability, and storage efficiency. It runs continuous background processes that detect failures, repair under-replicated data, reclaim unused storage, and balance data distribution across nodes.
Custodian scans DFD health records at a configurable interval (default: 5 seconds). A node is marked dead when its last heartbeat exceeds the timeout threshold (default: 45 seconds).When a node failure is detected:
Custodian updates the node state to Dead in FoundationDB
The node is removed from available read and write indexes
Repair tasks are queued for all stripes that resided on the failed node
The repair scanner periodically examines stripe metadata to identify durability violations:
Replication: Stripes with fewer than the required number of healthy replicas
Reed-Solomon: Stripes missing data or parity shards beyond the reconstruction threshold
For each under-replicated stripe, Custodian reads the surviving chunks, reconstructs any missing data using Reed-Solomon decoding if applicable, and writes new replicas to healthy nodes. Failed node repairs receive elevated priority to restore durability quickly.
DFS uses a two-phase garbage collection process:Metadata GC scans the inode subspace for entries with zero reference count. After a configurable grace period, these inodes and their associated stripe metadata are deleted.Chunk GC identifies orphaned chunks on DFD nodes—chunks that exist on disk but are not referenced by any stripe metadata. The process builds a set of valid chunk IDs from MDS, then each DFD node compares its local inventory against this set and deletes orphans. A grace period protects chunks that are newly written but not yet committed.
When storage utilization becomes uneven across nodes, Custodian migrates chunks from nodes approaching capacity limits to nodes with available space. This prevents hotspots and ensures write operations can proceed without capacity-related failures.
Storage capacity: Add DFD nodes to increase total storage. Nodes register automatically through health reporting.Metadata throughput: MDS is stateless. Deploy multiple instances behind a load balancer for higher throughput.Fault tolerance: Increase the number of DFD nodes to improve data durability. Reed-Solomon configurations with more parity shards tolerate additional failures.Monitoring: Run Custodian with appropriate scan intervals based on your availability requirements. Shorter intervals detect failures faster but increase FoundationDB load.
