OpenTelemetry Into a Stack You Own
Four machines, one self-hosted SigNoz on a 7.5 GB box. The collector setup and naming scheme that keep it usable, and the two defaults that quietly deleted my logs and buried the box in its own profiling data.

Observability vendors charge by the gigabyte, so the moment a system gets interesting it also gets expensive. The small production platform I run has four machines, and all of their telemetry goes to a self-hosted SigNoz on one of them, a box with 7.5 GB of RAM. It takes in about 5,000 spans, 800,000 metric samples and a megabyte of logs an hour, and these days it's quiet. Getting it there took a few mistakes, and those are most of this post.
OpenTelemetry is what makes this sane. Traces, metrics and logs use one set of libraries and one wire protocol, the app sends to an endpoint, and what sits at the other end is just config. That's worth having even if you pay for a hosted backend, because switching provider stops being a rewrite.
One collector per machine
I don't point applications straight at the backend. Every machine runs its own collector, and everything on that machine sends to it locally. It gathers host metrics and per-container stats, tails the container log files, probes the public URLs every 30 seconds, and passes on whatever the apps send.

The main reason for the extra hop is naming. The collector is the one place that stamps environment, tier and service name on everything that leaves the machine:
yamlreceivers: otlp: protocols: grpc: { endpoint: 0.0.0.0:4327 } http: { endpoint: 0.0.0.0:4328 } processors: transform/naming: trace_statements: - context: resource statements: - set(attributes["deployment.environment"], "staging") where attributes["host.name"] == "staging" - set(attributes["deployment.environment"], "production") where attributes["host.name"] != "staging" - set(attributes["tier"], "app") where IsMatch(attributes["service.name"], "^(api|web|migrate|postgres)$") - set(attributes["tier"], "infra") where IsMatch(attributes["service.name"], "^(traefik|hostmetrics)$") # the same statements again for metric_statements and log_statements
The apps never need to know where the backend lives, and when I want to rename something I edit one file instead of twenty service configs.
Settle the names before there's data
This is the one I'd most like to save people from. If a service name has the environment baked into it (api-staging), or the same service is called one thing in metrics and another in traces, you can't build a dashboard that groups by anything. So I decided up front. service.name is short and never contains the environment, deployment.environment is either staging or production, and every series, span and log line carries a tier. That holds for the infrastructure too, so Traefik and Postgres show up on the same overview as the apps. Fixing names after the fact is possible, just miserable.
Set retention for every signal, and check it
Traces, metrics and logs go stale at different speeds. I keep metrics for 30 days and traces and logs for 15. Set all three explicitly.
I learned that the hard way. Metrics and traces had retention set, logs didn't, and in the version I was running an unset log retention didn't mean "keep forever". Every log row was stamped with a retention of zero days, and the background merge deleted it minutes after it arrived. The UI looked healthy. The logs table sat at around 15 rows while thousands were going in. One API call fixed it:
bashcurl -X POST "$SIGNOZ/api/v2/settings/ttl" \ -H "SIGNOZ-API-KEY: $KEY" -H "Content-Type: application/json" \ -d '{"type":"logs","defaultTTLDays":15,"ttlConditions":[]}'
If you're not sure yours is set, write something today and go look for it tomorrow.
Turn off the database's own profiling

SigNoz keeps telemetry in ClickHouse, and the ClickHouse config it ships with has the query and memory profilers switched on. On my small box that meant 177 million rows a day into ClickHouse's own trace_log, while the telemetry I actually cared about came to 44 MB. Merging all that kept three cores busy and pushed the machine into swap, and the swap alert flapped at 510 of 512 MiB. Now the profilers are off:
yamlquery_profiler_real_time_period_ns: 0 query_profiler_cpu_time_period_ns: 0 memory_profiler_step: 0 log_processors_profiles: 0 log_query_views: 0
and only query_log, part_log and text_log are kept, each with a one-day TTL. One catch: removing a log table from the config stops it being created, but a table that already exists stays and keeps merging until you truncate it yourself.
Dashboards in the repo, alerts on symptoms
Dashboards are generated from a Python script that sits in the repo, and the same script then runs every panel's query and prints whether it got data, came back empty or errored. Clicking one together in the UI is quicker the first time. After that, it lives in one place with no history.
Alerts go to one channel and cover symptoms. A machine stopped reporting, swap, a disk filling, 5xx at the front door, availability under 99%, certificates within 30 days of expiry, a burst of errors in the app logs. Two of them were wrong at first. The disk alert filtered on a mountpoint the collector never reports, so it could never have fired. And the availability alert paged at night over one failed request on a quiet route. It now needs at least 100 requests and at least three server errors in ten minutes before it says anything. I found the disk one by reading the rules again, not because anything told me.