GTSDB Logo GTSDB Architecture

GTSDB Architecture

GTSDB (Go Time Series Database) uses a WAL-first design philosophy — fundamentally different from traditional databases that rely on WAL + disk blocks. By making the WAL the primary storage, GTSDB minimizes IO and memory usage while maintaining durability.

System Overview

┌─────────────────────────────────────────────────────────┐
│                      GTSDB Server                        │
│                                                          │
│  ┌──────────────┐              ┌──────────────┐         │
│  │  TCP Server  │              │  HTTP Server │         │
│  │  :5555       │              │  :5556       │         │
│  └──────┬───────┘              └──────┬───────┘         │
│         │                             │                  │
│         └──────────┬──────────────────┘                  │
│                    ▼                                      │
│          ┌─────────────────┐                             │
│          │    Handlers     │  ← auth, routing, validation │
│          └────────┬────────┘                             │
│                   ▼                                       │
│          ┌─────────────────┐                             │
│          │     Buffer      │  ← WAL write, indexing, LRU │
│          └────────┬────────┘                             │
│                   ▼                                       │
│          ┌─────────────────┐                             │
│          │     Fanout      │  ← pub/sub notifications    │
│          └────────┬────────┘                             │
│                   ▼                                       │
│          ┌─────────────────┐                             │
│          │   Concurrent    │  ← Map, Set, LRU, RingBuf   │
│          └─────────────────┘                             │
│                                                          │
│  ┌──────────────────────────────────────────────────┐   │
│  │                    Disk (SSD)                     │   │
│  │  *.aof (data)    *.idx (index)    users.json     │   │
│  └──────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────┘

Core Components

main.go

Entry point. Starts TCP + HTTP servers, background compaction, and graceful shutdown via OS signals.

handlers/

HTTP (REST + SSE) and TCP protocol handlers. Shared operation logic in common.go. Input validation (timestamp range, key safety).

buffer/

Data persistence layer. WAL-first approach: .aof files store 16-byte records (timestamp + value), .idx files provide sparse indexing every 5,000 records. File handle LRU cache prevents OS limit exhaustion. Background compaction removes gaps from deleted data.

fanout/

Lock-free publisher-subscriber system for real-time streaming. Consumers subscribe to key updates via SSE (HTTP) or TCP push. Uses atomic pointer swap for zero-allocation pending state.

concurrent/

Custom thread-safe data structures: generic Map[K,V] with RWMutex, generic Set[T], and LRU cache with eviction callback (used to close evicted file handles).

auth/

Token-based authentication with user-key folder isolation. Root user has admin privileges. no_auth_user config option for trusted environments.

Write & Read Path

Write Path

  1. Client sends POST / or TCP with JSON payload
  2. Handlers validate key (no path traversal), timestamp (2000–2100 range), and auth
  3. Buffer writes 16-byte record (int64 timestamp + float64 value) to .aof file
  4. Every 5,000th record triggers an index write to .idx file
  5. File handles are managed by LRU cache (capacity: 700) — evicted handles are automatically closed
  6. Fanout notifies all subscribers of the new data point
  7. fsync is called after each batch write for durability

Read Path

  1. Client requests data with time range, last-X, or downsampling parameters
  2. Handlers validate parameters and route to buffer layer
  3. Buffer first checks in-memory ring buffer for recent data (hot path)
  4. On cache miss, binary-searches .idx file to find the nearest index entry
  5. Seeks to the offset in .aof file and scans forward linearly
  6. If downsampling requested, aggregates data per interval (avg, sum, min, max, first, last, count, median, p95, p99)
  7. Results returned as JSON array

Storage Format

AOF File (Append-Only File)

[int64 timestamp][float64 value][int64 timestamp][float64 value]...
 └── 8 bytes ──┘└── 8 bytes ──┘
 └────────── 16 bytes per record ──────────┘

Fixed-size 16-byte records enable O(1) offset calculation and fast seeking.

IDX File (Sparse Index)

[int64 timestamp][int64 offset][int64 timestamp][int64 offset]...
 └── 8 bytes ──┘└── 8 bytes ─┘
 └────────── 16 bytes per entry ───────────┘

One index entry per 5,000 data records. Maps timestamp → byte offset in AOF file for fast seeking.

File Naming

  • data/root/<key>.aof — raw data records (user-folder isolated)
  • data/root/<key>.idx — sparse index (timestamp → byte offset)
  • data/root/<key>.aof.gor — Gorilla-compressed data (optional)
  • data/users.json — auth user database

Concurrency Model

concurrent.Map[K, V]

Generic concurrent map with RWMutex per operation. Used for idToCountMap, lastValue, lastTimestamp, dataPatchLocks.

concurrent.Set[T]

Thread-safe generic set. Used for allIds (key registry) and fanout consumers.

concurrent.LRU[K, V]

LRU cache with eviction callback. Used for file handle management (dataFileHandles + indexFileHandles, capacity 700 each). Evicted handles are closed via callback.

synchronous.RingBuffer[T]

Fixed-size ring buffer for in-memory caching of recent data points. Provides hot-path reads without disk access.

Compaction

Why Compaction Is Needed

GTSDB stores data in append-only .aof files. When data points are deleted (via deleteDataPoint or deletekey), the records are not physically removed from the AOF file — instead, the entire file is rewritten without the deleted records. However, partial overwrites (single-point timestamp overwrites) leave the old record in place and write a new one elsewhere, creating "holes" of dead space. Over time, these gaps accumulate and waste disk.

Compaction solves this by reading all live data points and rewriting them into a fresh, gap-free file — effectively defragmenting the storage.

Compaction Algorithm (Step by Step)

1
Acquire Key-Level Lock

Uses the same per-key mutex as data-patch to prevent concurrent modifications during compaction.

2
Read All Live Data Points

Scans the entire .aof file from start to end, collecting every valid record. This is the source of truth — only data that still exists gets rewritten.

3
Write to Temporary Files

Writes all live data points to key.aof.tmp and rebuilds the sparse index in key.idx.tmp. Old file handles are closed and removed from the LRU cache. Existing files remain untouched until the rename — crash-safe by design.

4
Atomic Rename (Two-Phase)

Phase 1: Rename .idx.tmp → .idx (smaller file, less risk).
Phase 2: Rename .aof.tmp → .aof.
Rollback: If Phase 2 fails, Phase 1 is reversed — old index restored, temp files cleaned up. This guarantees the on-disk state never becomes inconsistent.

5
Re-open Handles & Update Caches

New file handles are re-opened via prepareFileHandles and placed in the LRU. lastValue, lastTimestamp, and idToCountMap are updated to reflect the new state. The global totalDataPoints counter is adjusted for the difference.

Background Compaction

  • Runs automatically every 1 hour
  • Scans all keys via GetAllIds()
  • Compacts files exceeding 100 MB
  • Respects shutdown signals (clean exit)

Manual Compaction

  • Triggered via API: {"operation":"compact","key":"sensor1"}
  • Compacts a single key on demand
  • Useful after bulk deletes or patching
  • Same atomic algorithm as background

Gorilla Compression

Time-Series Optimized Compression

During compaction, GTSDB optionally compresses AOF files using the Facebook Gorilla algorithm — a bit-level encoder designed specifically for time-series data. Unlike generic compression (zstd/gzip), Gorilla exploits the predictable structure of (timestamp, value) pairs.

Enable via gtsdb.ini:

[buffer]
compaction_compression = true

Timestamp: Delta-of-Delta

For regular-interval data (e.g., 1 reading/second), 90%+ of timestamps encode in 1 bit.

T1=1000, T2=1001, T3=1002  → deltas: +1,+1  → DoD: 0,0
Encoded: 0 0  (2 bits for 3 timestamps)

Value: XOR

Adjacent float64 values share most bits. Only the differing bits are stored. Constant values take 1 bit (XOR=0).

V1=42.5, V2=42.6 → XOR has leading/trailing zeros
Encoded: only middle meaningful bits

Benchmark Results (i7-13700KF, 5,000 points)

MetricRaw AOF (16B/record)Gorilla CompressedImprovement
Write (5K pts)6.5 ms0.12 ms56x faster
Decode (5K pts)—85 µsnegligible vs disk I/O
Disk space80 KB10 KB7.98x smaller
Allocations5,00012417x fewer

Background Processes

TCP Ping

Server sends periodic ping messages to TCP clients to detect disconnection. Uses sync.Once for safe cleanup.

Data Migration

On startup, migrates existing unprefixed keys into user folders (e.g., "sensor1" → "root/sensor1") for multi-tenant isolation.

Graceful Shutdown

HTTP server uses Shutdown() with 10-second timeout. TCP listener closes cleanly. All file handles are synced via FlushRemainingDataPoints().

Observability

GET /health

No-auth health check: status, version, key count.

GET /metrics

Prometheus-format metrics: gtsdb_key_count, gtsdb_data_points_total, gtsdb_uptime_seconds, gtsdb_goroutines, go_memstats_alloc_bytes, go_memstats_heap_inuse_bytes, go_gc_duration_seconds_sum, go_cpu_count.

POST / (serverinfo)

Detailed server info: version, uptime, goroutines, memory, listen addresses, data directory, file handle LRU capacity.

Key Design Decisions

DecisionRationale
WAL as primary storageAvoids double-write (WAL + data files) of traditional DBs. Reduces IO by 50%.
Fixed 16-byte recordsEnables O(1) random access via byte offset calculation. No need for record delimiters.
Sparse index (every 5K)Balances seek speed vs index file size. At most ~80 records to scan linearly after index seek.
LRU file handle cachePrevents OS "too many open files" errors. 700 capacity supports hundreds of active keys.
Dual protocol (HTTP + TCP)HTTP for web/admin integration. TCP for low-latency IoT/edge device communication.
User-folder key isolationMulti-tenant support without complex ACLs. Simple prefix-based access control.
Lock-free fanoutAtomic pointer swap for pub/sub pending state. Zero heap allocation per publish.