PulseStream

Overview

PulseStream is a real-time analytics pipeline in Go that ingests the public GitHub Events API, aggregates activity metrics in near-real-time, persists them to PostgreSQL, and serves them through a REST API and a Grafana dashboard. It's built to demonstrate the unglamorous 80% of a data platform — the layer before “load a CSV into pandas”: getting real, continuously-arriving external data into a queryable, aggregated form, reliably. Around 1,600 lines of Go with a single runtime dependency (pgx).

Ingestion under a live rate limit

The ingest layer polls /events with conditional requests (If-None-Match, so a 304 costs nothing against the rate limit), honors X-Poll-Interval and X-RateLimit headers, and deduplicates overlapping pages with an in-memory ring set. It stays deliberately dumb — acquisition only, no business logic.

Backpressure & worker pool

A buffered Go channel sits between poller and processor; a full channel blocks the poller, which is the backpressure mechanism. A worker pool (default NumCPU) normalizes each event and, in a single transaction, inserts the raw row and increments two rollup tables — aggregation happens at write time, so the dashboard is near-real-time with no scheduler.

Idempotent storage & crash recovery

Storage is an append-only events table plus incrementally-maintained rollups, so dashboard queries never scan raw events. Writes are idempotent with a measured mid-batch crash-recovery guarantee — the pipeline restarts without double-counting.

Serving layer

A read-only REST API over Postgres built on the Go 1.22 stdlib http.ServeMux (no router dependency), plus a provisioned four-panel Grafana dashboard for live visualization.

Built with

• Go
• PostgreSQL
• Grafana
• pgx
• Docker