Observability is a superset of tools and techniques to understand how a system works, exploring “knowns” and “unknowns” in the systems. Monitoring involves tracking and measuring structured data.
Log Aggregation
- log forwarder - software to collect logs from different data sources, transform logs if required and send those to a log aggregator
- log aggregator - software to aggregate, store and present logs
Process of log aggregation
- collect logs from multiple sources in different formats. Recommended to have a standardised log format.
- standardise - if logs have different formats, at this step you need to align logs to one format
- verify - check, that logs are valid, parsable, and contain all data.
- consolidate - gather logs to one place, where you can analyse logs.
Distributed Tracing
Tracking multiple services calls for the same business operation in the distributed system. Usually implemented by adding a correlation id into the message envelope.
Metrics Aggregation
Similar to logs aggregation. The difference is, that metrics are usually numbers giving the insights about system characteristics/resources/performance.
Types of context in logs
- Temporal context - context of how an object changes in time.
- Relative context - context of how an object changed in relation to other objects.
- Relation context - context about dependencies and defendants in the system of a logged action.
- Proportion context - gives insights about the scope of action in the logs.
OpenTelemetry
Open source observability framework. It provides libraries, conventions, APIs and SDKs to have observability on your system. It is neither an observability backend nor an observability database.
Microservices, 2nd edition by Sam Newman
