InfluxDB 3 Enterprise Internals

How data flows through InfluxDB 3 Enterprise

When data is written to InfluxDB 3 Enterprise, it progresses through multiple stages to ensure durability, optimize performance, and enable efficient querying. Configuration options at each stage affect system behavior, balancing reliability and resource usage.

Data flow for writes

As written data moves through InfluxDB 3 Enterprise, it follows a structured path to ensure durability, efficient querying, and optimized storage.

Figure: Write request, response, and ingest flow for InfluxDB 3 Core and Enterprise

1. Write validation and memory buffer
2. Write-ahead log (WAL) persistence
3. Query availability
4. Parquet storage
5. In-memory cache

Write validation and memory buffer

• Process: InfluxDB validates incoming data before accepting it into the system.
• Impact: Prevents malformed or unsupported data from entering the database.
• Details: The database validates incoming data and stores it in the write buffer (in memory). If no_sync=true, the server sends a response to acknowledge the write without waiting for persistence.

Write-ahead log (WAL) persistence

• Process: The database flushes the write buffer to the WAL every second (default).
• Impact: Ensures durability by persisting data to object storage.
• Tradeoff: More frequent flushing improves durability but increases I/O overhead.
• Details: Every second (default), the database flushes the write buffer to the Write-Ahead Log (WAL) for persistence in the object store. If no_sync=false (default), the server sends a response to acknowledge the write.

Query availability

• Process: The system moves data to the queryable buffer after WAL persistence.
• Impact: Enables fast queries on recent data.
• Tradeoff: A larger buffer speeds up queries but increases memory usage.
• Details: After WAL persistence completes, data moves to the queryable buffer where it becomes available for queries. By default, the server keeps up to 900 WAL files (15 minutes of data) buffered.

Parquet storage

• Process: Every ten minutes (default), data is persisted to Parquet files in object storage.
• Impact: Provides durable, long-term storage.
• Tradeoff: More frequent persistence reduces reliance on the WAL but increases I/O costs.
• Memory usage: The persistence process uses memory from the configured memory pool (exec-mem-pool-size) when converting data to Parquet format. For write-heavy workloads, ensure adequate memory is allocated.
• Details: Every ten minutes (default), InfluxDB 3 Enterprise persists the oldest data from the queryable buffer to the object store in Parquet format, and keeps the remaining data (the most recent 5 minutes) in memory.

WAL tail

The WAL tail is the most recent data in the WAL that InfluxDB 3 Enterprise has not yet durably persisted beyond the WAL.

Because InfluxDB 3 Enterprise flushes the WAL to object storage every second, the WAL tail is durable, but it remains in the WAL until the next persistence step captures it: the next Parquet persistence on the Parquet engine, or the next snapshot on the upgraded storage engine (the default for new clusters in 3.11+).

In-memory cache

• Process: Recently persisted Parquet files are cached in memory.
• Impact: Reduces query latency by minimizing object storage access.
• Details: InfluxDB 3 Enterprise puts Parquet files into an in-memory cache so that queries against the most recently persisted data don’t have to go to object storage.

Upgraded storage engine compaction

The upgraded storage engine (the default for new clusters in 3.11+) compacts data differently from the Parquet engine described above.

Incoming writes are buffered in the WAL, flushed to snapshots, and merged into Gen0 files. From there, InfluxDB 3 Enterprise uses time-disjoint two-level compaction by default: all leading-edge ingest funnels through a hot-tail L1 run set, and several concurrent L1 compaction jobs can run against the leading edge (or any heavily written range) at once. L1 run sets are allowed to transiently overlap in their assigned time range under load—this is a healthy, query-safe state—and the compactor reconciles back to a disjoint L1 as high-priority background work. Under low load, nothing overlaps and behavior matches the legacy layout.

Clusters that started on 3.10 or earlier and have not yet upgraded keep the legacy four-level (L1 through L4) compaction layout, where compaction serializes on a single hot tail and concurrent leading-edge writes must wait. Newly created window/shards use the time-disjoint layout; existing checkpoints keep their recorded layout until explicitly upgraded.

For per-level and per-engine tuning options, see Storage engine configuration reference.