cactoandClaude Sonnet 5
938700cf86
Subscribes to devices/+/telemetry, validates payloads, buffers writes into ClickHouse (flush on size or interval), and emits a per-reading event on RabbitMQ for rule-engine-service to consume later. Adds the initial Postgres and ClickHouse schema migrations, a Dockerfile, and wires the service into docker-compose. Verified end-to-end via docker compose + mosquitto_pub: row lands in ClickHouse and the event reaches the telemetry.new_reading queue. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
1.4 KiB
ingest-service (Go)
Status: implemented (MVP).
Responsibility:
- Subscribe to MQTT topics
devices/{device_id}/telemetry. - Validate/parse payload (device_id, sensor_type, value, timestamp).
- Batch-write raw readings into ClickHouse
telemetrytable. - Publish a "new reading" event to RabbitMQ for rule-engine-service.
Not this service's job: interpreting what a sensor reading means (thresholds, actions) — that belongs to rule-engine-service. This service only ingests and stores.
Note on zone_id: the telemetry payload carries zone_id directly (the
publishing device/emulator is configured with its own zone), rather than
ingest-service looking it up from PostgreSQL. This keeps ingest-service's
dependencies limited to MQTT/ClickHouse/RabbitMQ, matching the architecture
diagram in the root README. A real fleet where devices don't know their own
zone would need a device-registry lookup (cached, like rule-engine caches
rules) — that's a natural follow-up, not implemented yet.
Running
cd services/ingest-service
go test ./...
go build ./cmd/ingest-service
Configuration is env-var driven (see the ingest-service section of the
repo-root .env.example). Via docker compose up ingest-service it talks to
the mosquitto/clickhouse/rabbitmq containers directly; connect from a
local go run by exporting MQTT_HOST=localhost, CLICKHOUSE_HOST=localhost, etc.