Implement ingest-service: MQTT -> ClickHouse (batched) + RabbitMQ event
Subscribes to devices/+/telemetry, validates payloads, buffers writes into ClickHouse (flush on size or interval), and emits a per-reading event on RabbitMQ for rule-engine-service to consume later. Adds the initial Postgres and ClickHouse schema migrations, a Dockerfile, and wires the service into docker-compose. Verified end-to-end via docker compose + mosquitto_pub: row lands in ClickHouse and the event reaches the telemetry.new_reading queue. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
# ingest-service (Go)
|
||||
|
||||
Status: not implemented yet.
|
||||
Status: implemented (MVP).
|
||||
|
||||
Responsibility:
|
||||
- Subscribe to MQTT topics `devices/{device_id}/telemetry`.
|
||||
@@ -11,3 +11,24 @@ Responsibility:
|
||||
Not this service's job: interpreting what a sensor reading *means* (thresholds,
|
||||
actions) — that belongs to rule-engine-service. This service only ingests and
|
||||
stores.
|
||||
|
||||
Note on `zone_id`: the telemetry payload carries `zone_id` directly (the
|
||||
publishing device/emulator is configured with its own zone), rather than
|
||||
ingest-service looking it up from PostgreSQL. This keeps ingest-service's
|
||||
dependencies limited to MQTT/ClickHouse/RabbitMQ, matching the architecture
|
||||
diagram in the root README. A real fleet where devices don't know their own
|
||||
zone would need a device-registry lookup (cached, like rule-engine caches
|
||||
rules) — that's a natural follow-up, not implemented yet.
|
||||
|
||||
## Running
|
||||
|
||||
```bash
|
||||
cd services/ingest-service
|
||||
go test ./...
|
||||
go build ./cmd/ingest-service
|
||||
```
|
||||
|
||||
Configuration is env-var driven (see the ingest-service section of the
|
||||
repo-root `.env.example`). Via `docker compose up ingest-service` it talks to
|
||||
the `mosquitto`/`clickhouse`/`rabbitmq` containers directly; connect from a
|
||||
local `go run` by exporting `MQTT_HOST=localhost`, `CLICKHOUSE_HOST=localhost`, etc.
|
||||
|
||||
Reference in New Issue
Block a user