Filament by Galaxy
Filament by Galaxy is an open-source data movement engine engineered to transfer records from databases, APIs, streams, and object storage into analytical sinks with CRC32-C verification and pluggable backends.
Filament by Galaxy is an open-source data movement engine engineered to transfer records from databases, APIs, streams, and object storage into analytical sinks with CRC32-C verification and pluggable backends.
What the product does and how it is positioned
Filament by Galaxy is an open-source data movement engine designed for data engineers to move information across diverse infrastructure. The system routes data from sources such as databases, APIs, event streams, and object storage through a single pipeline into target destinations including Snowflake, BigQuery, ClickHouse, and Iceberg.
The engine features a pluggable architecture where backends for state, events, and secrets can be configured and swapped without altering existing pipelines. For data integrity, Filament applies CRC32-C hashing across all records during read and write cycles, failing batches if counts, byte volumes, or checksums do not align.
Source-supported ways to use the product
Moving tabular records from operational databases and object storage files into analytical sinks such as Snowflake, BigQuery, ClickHouse, or Iceberg.
Replicating data from APIs and streaming sources while enforcing batch-level validation via CRC32-C checksums, byte totals, and record counts.
Filament implements a data movement pipeline designed to handle ingestion from databases, APIs, event streams, and object storage into target data warehouses and lakehouses.
To protect data integrity across transfers, the engine hashes every record on read and write using CRC32-C. Batches automatically fail if record counts, total bytes, or calculated checksums do not match between endpoints.
The platform is built with a pluggable design allowing operators to select and swap backends for state management, event coordination, and secret storage without modifying pipeline configurations.
Checks to run with your own material and workflow
What was checked and when
Answers based on the source-checked product record
Filament is an open-source data movement engine created by Galaxy for data engineers to move records across infrastructure pipelines.
Filament supports data ingestion from databases, application programming interfaces, event streams, and files stored in object storage.
Filament supports landing data into analytical sinks including Snowflake, BigQuery, ClickHouse, and Apache Iceberg through its pipelines.
Filament hashes every record on read and write using CRC32-C, automatically failing batches unless record counts, byte totals, and checksums match.
Filament features a pluggable architecture that allows users to choose and swap backends for state, events, and secrets without changing pipelines.