Observability: the present and future, with Charity Majors

Last time, most of you said you enjoyed having a summary, to read along. For the next few issues I’m including this: a bullet-point summary for reading or skimming, followed by brief takeaways for things that stood out to me the most.

Stream the Latest Episode

Available now on YouTube, Apple and Spotify. See the episode transcript at the top of this page.

Brought to You By

• Sonar — Trust your developers – verify your AI-generated code.

• Vanta —Automate compliance and simplify security with Vanta.

—

In This Episode

In today's episode of The Pragmatic Engineer, I'm joined by Charity Majors, a well-known observability expert – as well as someone with strong and grounded opinions. Charity is the co-author of "Observability Engineering" and brings extensive experience as an operations and database engineer and an engineering manager. She is the cofounder and CTO of observability scaleup Honeycomb.

Our conversation explores the ever-changing world of observability, covering these topics:

• What is observability? Charity’s take

• What is “Observability 2.0?”

• Why Charity is a fan of platform teams

• Why DevOps is an overloaded term: and probably no longer relevant

• What is cardinality? And why does it impact the cost of observability so much?

• How OpenTelemetry solves for vendor lock-in

• Why Honeycomb wrote its own database

• Why having good observability should be a prerequisite to adding AI code or using AI agents

• And more!

A summary of the conversation

For those of you more interested in reading a summary of the conversation — or skimming over it — see it here. Takeaways follow after the summary.

Observability (o11y) basics

• Observability is about understanding software, specifically the intersection of code, systems, and users.

• It is not just about errors, bugs and outages; it is also about understanding the impact of code.

• Observability is a tool that is critical for development feedback loops, and is not just an operational tool.

• The goal of good o11y is to help engineers understand their software in the language of the business.

• Engineers should be able to tie their work back to top-level goals, and explain how their work translates to the business.

• Sampling is an important lever, contrary to the idea that every log is sacred.

• ‘metrics’ vs ‘Metrics’

  • We need to distinguish between metrics (small 'm') as a generic term for telemetry and Metric (capital 'M') as a specific data type, a number with appended tags.

  • The Metric data type is limited because it doesn't sort any contextual relationship data.

• The Three Pillars Model

  • The three pillars model of observability is this: metrics, logs and traces.

  • Many vendors sell products for each of these pillars – as well as for all of them

  • The problem with the Three Pillars Model is that every request that enters a system is stored multiple times, in different tools (metrics, logs, traces, profiling, analytics).

  • There is little to connect the data points; engineers are left to manually correlate the data.

  • The cost of following this model is high: it’s because storing the same data in multiple tools and databases is very high!

What is Observability 2.0?

• Observability 2.0 m