OpenTelemetry Into a Stack You Own

Observability vendors charge by the gigabyte, which means the moment a system gets interesting it also gets expensive. You can run the whole thing yourself on one modest machine, and the standard is now good enough that you are not locked in either way.

One standard, three signals

OpenTelemetry gives you traces, metrics and logs through one set of libraries and one wire protocol. Your application emits to an endpoint, and what sits on the other end is a configuration value. That is the part worth having even if you use a hosted backend: swapping providers stops being a rewrite.

Put a collector on every machine

Do not point applications straight at your backend. Run a collector on each host and let everything send to it locally. It gathers host metrics and container stats, picks up the container logs, and acts as the gateway for anything else on that box.

It is also the one place to attach the tags that make the data usable, which saves a great deal of pain later. Environment, machine, service name, tier, version. Set them once in the collector rather than in twenty separate service configurations, because the day you want to rename something you will want one file to edit rather than twenty.

Agree the tag scheme before you have data

This is the mistake worth avoiding. Service names that carry the environment in them, or that differ between metrics and traces, make it impossible to build a dashboard that groups by anything. Decide what a service is called, decide what environment means, and use those names everywhere, including for infrastructure components. Retrofitting is possible and unpleasant.

Retention is per signal, and the default may be nothing

Traces, metrics and logs have different value over time and should have different retention. Metrics are cheap and worth keeping longest. Traces are bulky and you rarely want one from six weeks ago. Logs sit in between.

Set all three explicitly. I have seen a backend stamp an unset retention as zero and quietly drop every log row minutes after it arrived, while the interface looked perfectly healthy and the row count stayed near zero. If you are not certain yours is set, insert something and go looking for it tomorrow.

Do not let the database profile itself

Self-hosted backends usually store telemetry in a column store, and those ship with their own internal logging and query profiling switched on. On a small machine that can generate far more data about itself than the actual telemetry it is holding, which shows up as constant background CPU and a disk filling for no visible reason. Turn off the internal profilers, keep the two or three system tables you would genuinely read, and remember that removing a table from configuration does not delete the one already there.

Dashboards belong in the repository

Clicking a dashboard together in a UI is quick, and then it exists in exactly one place with no history and no way to reproduce it. Generate them from code, keep them next to the service, and review changes to them like anything else. The first version takes longer. Every version after that is free.

Alert on symptoms

A small number of alerts people trust beats a wall of alerts people mute. Errors at the front door, availability below target, a machine that has stopped reporting, disk about to fill, and certificates about to expire will cover most of what actually wakes you. Add to it when something surprises you in production, and only then.