Overview

QuestDB offers high-speed ingestion and low-latency analytics on time-series data. It stores data in a three-tier architecture (streaming WAL files, local binary storage, and local or remote storage in Parquet format) to improve interoperability and avoid vendor lock-in.

WHERE symbol in ('AAPL', 'NVDA')
LATEST ON timestamp PARTITION BY symbol
CREATE MATERIALIZED VIEW 'trades_OHLC'
min(price) AS low
timestamp IN today()
SELECT spread_bps(bids[1][1], asks[1][1])
FROM read_parquet('trades.parquet')
SAMPLE BY 15m

Tier One: Hot ingest (WAL), durable by default

Tier Two: Real-time SQL on live data

Tier Three: Cold storage, open and queryable

This document explains QuestDB's internal architecture.

Key components​

QuestDB is comprised of several key components:

  • Storage engine: Uses a column-oriented design to ensure high I/O performance and low latency.

  • Memory management and native integration: The system leverages both memory mapping and explicit memory management techniques,integrating native code for performance-critical tasks.

  • Query engine: A custom SQL parser, a just-in-time (JIT) compiler, and a vectorized execution engine process data in table page frames for better CPU use.

  • Time-series optimizations: QuestDB is specifically designed for time-series, and it provides several optimizations, such as a designated timestamp, sequential reads, materialized views, and in-memory processing.

  • Data ingestion engine: Supports both bulk and streaming ingestion. It writes data to a row-based write-ahead log (WAL) and then converts it into a columnar format. In QuestDB Enterprise, the WAL segments are shipped to object storage for replication.

  • Networking layer: The system exposes RESTful APIs and implements ILP and PostgreSQL wire protocols to ensure compatibility with existing tools and drivers. It also offers a health and metrics endpoint.

  • Replication layer: QuestDB Enterprise supports horizontal scalability for reads via read replicas, and for writes via multi-primary configurations.

  • Security: QuestDB implements enterprise-grade security with TLS, single sign-on (SSO), and role-based access control with fine-grained granularity.

  • Observability: QuestDB exposes real-time metrics, health checks, and structured logs to monitor performance and streamline diagnostics.

  • Web console: The engine includes a web console for running SQL statements, bulk loading CSV files, and displaying monitoring dashboards. QuestDB Enterprise supports single sign-on (SSO) in the web console.

Design patterns & best practices throughout the codebase​

  • Immutable data structures: The system favors immutability to avoid concurrency issues and simplify state management.

  • Modular architecture: Each component (eg., storage, query processing, ingestion, etc.) has well-defined interfaces that enhance maintainability and decouple functionality.

  • Factory & builder patterns: These patterns are used to centralize construction logic for complex objects such as SQL execution plans and storage buffers.

  • Lazy initialization: Resource-intensive components initialize only when needed to reduce startup overhead.

  • Rigorous testing & benchmarks: Unit tests, integration tests, and performance benchmarks ensure that new enhancements do not compromise reliability or performance.

Further reading & resources​