Skip to content
ENNO
← Back to blog
·4 min read·

OpenTelemetry Into a Stack You Own

Four machines, one self-hosted SigNoz on a 7.5 GB box. The collector setup and naming scheme that keep it usable, and the two defaults that quietly deleted my logs and buried the box in its own profiling data.

OpenTelemetry Into a Stack You Own

Observability vendors charge by the gigabyte, so the moment a system gets interesting it also gets expensive. The small production platform I run has four machines, and all of their telemetry goes to a self-hosted SigNoz on one of them, a box with 7.5 GB of RAM. It takes in about 5,000 spans, 800,000 metric samples and a megabyte of logs an hour, and these days it's quiet. Getting it there took a few mistakes, and those are most of this post.

OpenTelemetry is what makes this sane. Traces, metrics and logs use one set of libraries and one wire protocol, the app sends to an endpoint, and what sits at the other end is just config. That's worth having even if you pay for a hosted backend, because switching provider stops being a rewrite.

One collector per machine

I don't point applications straight at the backend. Every machine runs its own collector, and everything on that machine sends to it locally. It gathers host metrics and per-container stats, tails the container log files, probes the public URLs every 30 seconds, and passes on whatever the apps send.

Diagram of four machines (prod, staging, ops, staff), each with its own agent on port 4327. The agent stamps deployment.environment (staging on the staging box, production elsewhere), a short service.name with no environment in it, a tier (app, infra, tools or observability) and service.version as a 12-character git sha. Everything goes over OTLP to the backend, which keeps metrics 30 days, traces 15 days and logs 15 days, the last only once set; unset meant 0.
Apps only ever talk to the box they run on.

The main reason for the extra hop is naming. The collector is the one place that stamps environment, tier and service name on everything that leaves the machine:

yaml
receivers: otlp: protocols: grpc: { endpoint: 0.0.0.0:4327 } http: { endpoint: 0.0.0.0:4328 } processors: transform/naming: trace_statements: - context: resource statements: - set(attributes["deployment.environment"], "staging") where attributes["host.name"] == "staging" - set(attributes["deployment.environment"], "production") where attributes["host.name"] != "staging" - set(attributes["tier"], "app") where IsMatch(attributes["service.name"], "^(api|web|migrate|postgres)$") - set(attributes["tier"], "infra") where IsMatch(attributes["service.name"], "^(traefik|hostmetrics)$") # the same statements again for metric_statements and log_statements

The apps never need to know where the backend lives, and when I want to rename something I edit one file instead of twenty service configs.

Settle the names before there's data

This is the one I'd most like to save people from. If a service name has the environment baked into it (api-staging), or the same service is called one thing in metrics and another in traces, you can't build a dashboard that groups by anything. So I decided up front. service.name is short and never contains the environment, deployment.environment is either staging or production, and every series, span and log line carries a tier. That holds for the infrastructure too, so Traefik and Postgres show up on the same overview as the apps. Fixing names after the fact is possible, just miserable.

Set retention for every signal, and check it

Traces, metrics and logs go stale at different speeds. I keep metrics for 30 days and traces and logs for 15. Set all three explicitly.

I learned that the hard way. Metrics and traces had retention set, logs didn't, and in the version I was running an unset log retention didn't mean "keep forever". Every log row was stamped with a retention of zero days, and the background merge deleted it minutes after it arrived. The UI looked healthy. The logs table sat at around 15 rows while thousands were going in. One API call fixed it:

bash
curl -X POST "$SIGNOZ/api/v2/settings/ttl" \ -H "SIGNOZ-API-KEY: $KEY" -H "Content-Type: application/json" \ -d '{"type":"logs","defaultTTLDays":15,"ttlConditions":[]}'

If you're not sure yours is set, write something today and go look for it tomorrow.

Turn off the database's own profiling

Two numbers from one day on a 7.5 GB box: 177 million rows a day into system.trace_log, the ClickHouse profiler's own table, against 44 MB of actual telemetry from every machine. Symptom: three cores busy merging all day and the swap alert flapping at 510 of 512 MiB. Fix: profilers set to 0, only query_log, part_log and text_log kept, the existing trace_log truncated by hand.
ClickHouse was mostly busy watching itself.

SigNoz keeps telemetry in ClickHouse, and the ClickHouse config it ships with has the query and memory profilers switched on. On my small box that meant 177 million rows a day into ClickHouse's own trace_log, while the telemetry I actually cared about came to 44 MB. Merging all that kept three cores busy and pushed the machine into swap, and the swap alert flapped at 510 of 512 MiB. Now the profilers are off:

yaml
query_profiler_real_time_period_ns: 0 query_profiler_cpu_time_period_ns: 0 memory_profiler_step: 0 log_processors_profiles: 0 log_query_views: 0

and only query_log, part_log and text_log are kept, each with a one-day TTL. One catch: removing a log table from the config stops it being created, but a table that already exists stays and keeps merging until you truncate it yourself.

Dashboards in the repo, alerts on symptoms

Dashboards are generated from a Python script that sits in the repo, and the same script then runs every panel's query and prints whether it got data, came back empty or errored. Clicking one together in the UI is quicker the first time. After that, it lives in one place with no history.

Alerts go to one channel and cover symptoms. A machine stopped reporting, swap, a disk filling, 5xx at the front door, availability under 99%, certificates within 30 days of expiry, a burst of errors in the app logs. Two of them were wrong at first. The disk alert filtered on a mountpoint the collector never reports, so it could never have fired. And the availability alert paged at night over one failed request on a quiet route. It now needs at least 100 requests and at least three server errors in ten minutes before it says anything. I found the disk one by reading the rules again, not because anything told me.

#OpenTelemetry#observability#self-hosting#monitoring#SigNoz