Implement ingest-service: MQTT -> ClickHouse (batched) + RabbitMQ event

Subscribes to devices/+/telemetry, validates payloads, buffers writes into
ClickHouse (flush on size or interval), and emits a per-reading event on
RabbitMQ for rule-engine-service to consume later. Adds the initial Postgres
and ClickHouse schema migrations, a Dockerfile, and wires the service into
docker-compose. Verified end-to-end via docker compose + mosquitto_pub: row
lands in ClickHouse and the event reaches the telemetry.new_reading queue.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-26 16:52:45 +05:00
co-authored by Claude Sonnet 5
parent 6ce487735f
commit 938700cf86
19 changed files with 990 additions and 2 deletions
+31 -1
View File
@@ -55,6 +55,11 @@ services:
- ./migrations/clickhouse:/docker-entrypoint-initdb.d:ro
networks:
- home-automation
healthcheck:
test: ["CMD", "clickhouse-client", "--query", "SELECT 1"]
interval: 5s
timeout: 5s
retries: 10
restart: unless-stopped
redis:
@@ -86,6 +91,31 @@ services:
retries: 10
restart: unless-stopped
# App services (ingest-service, device-control-service, rule-engine-service,
ingest-service:
build: ./services/ingest-service
environment:
MQTT_HOST: mosquitto
MQTT_PORT: 1883
CLICKHOUSE_HOST: clickhouse
CLICKHOUSE_NATIVE_PORT: 9000
CLICKHOUSE_DB: ${CLICKHOUSE_DB}
CLICKHOUSE_USER: ${CLICKHOUSE_USER}
CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD}
RABBITMQ_HOST: rabbitmq
RABBITMQ_PORT: 5672
RABBITMQ_USER: ${RABBITMQ_USER}
RABBITMQ_PASSWORD: ${RABBITMQ_PASSWORD}
depends_on:
mosquitto:
condition: service_started
clickhouse:
condition: service_healthy
rabbitmq:
condition: service_healthy
networks:
- home-automation
restart: unless-stopped
# Remaining app services (device-control-service, rule-engine-service,
# health-check-service, esp32-emulator, laravel-app) are added here as they
# get implemented — see services/*/README.md for the plan for each one.