SeaweedFS

SeaweedFS is an open-source distributed file system designed to store billions of files with fast access and horizontal scaling. It operates as a server-side application that can be deployed as a single-node instance for development or scaled across thousands of machines for production environments. The system is available as a binary for various operating systems and as a Docker image, allowing for flexible deployment across different infrastructure setups.

Originally developed as a blob store for small files, SeaweedFS minimizes central master overhead by distributing file metadata across volume servers. This architecture allows for fast file access with minimal disk read operations, typically requiring only one disk read. Users can interact with the system via a web-based admin UI, a command-line interface, or through various network protocols and APIs to manage their data and cluster health.

Key features

• S3 compatible API for object storage access
• FUSE mount for reading and writing files as a local directory
• Erasure coding and rack-aware replication for data availability
• Built-in Iceberg REST Catalog for data lakehouse functionality
• WebDAV support for mapping the system as a network drive
• Transparent integration with cloud storage for tiered data management
• Support for Hadoop Compatible File System (HDFS) interfaces
• Kubernetes CSI Driver for container storage integration
• AES256-GCM encrypted storage for data security

The software includes an optional Filer component that provides directory structures and POSIX attributes. Filer is a stateless server that supports a wide variety of metadata stores, including PostgreSQL, MySQL, Redis, Cassandra, and MongoDB. The system also supports automated table maintenance for Iceberg, including compaction, snapshot expiration, and orphan removal. This allows it to integrate with analytics tools like Apache Spark, Trino, and DuckDB without requiring an external Hive Metastore or Glue service.

SeaweedFS is intended for users and organizations requiring a scalable storage solution for massive amounts of small files, object storage, or on-premises data lakehouses.