Prometheus prototype

PromQL NHCB-as-classic conversion demo

Queries written for classic histograms keep working on native histograms with custom buckets (NHCB), as this Prometheus converts NHCB to classic series at query time. Each view below opens a classic histogram query next to its native histogram equivalent.

Setup

This Prometheus runs with the flags below and scrapes itself every 5s with 5 jobs, see its configuration and targets. It exposes prometheus_http_request_duration_seconds with both classic buckets, from 0.1s to 120s, and exponential native buckets, and each job stores it in a different representation.

--enable-feature=promql-nhcb-as-classic
JobScrape optionsStored as
classicClassic histograms, as before a migration.Defaults
classic
nhcbMigrated to native histograms with custom buckets, converted on scrape.
convert_classic_histograms_to_nhcb: true
nhcb
classic-and-nhcbIn the middle of a migration to NHCB, storing both.
convert_classic_histograms_to_nhcb: truealways_scrape_classic_histograms: true
classic nhcb
nativeMigrated to native histograms with exponential buckets.
scrape_native_histograms: trueconvert_classic_histograms_to_nhcb: true
nhe
classic-and-nativeIn the middle of a migration to exponential buckets, scraping both.
scrape_native_histograms: truealways_scrape_classic_histograms: true
classic nhe
  • classic classic histogram series: _bucket, _count and _sum
  • nhcb native histogram with custom buckets (NHCB)
  • nhe native histogram with exponential buckets

The native job also converts histograms exposed with classic buckets only, like prometheus_http_response_size_bytes, to NHCB, so it stores no classic histograms at all.

How queries read histograms

SelectorReturns
prometheus_http_request_duration_seconds_bucket, _count, _sumClassic histogram queriesStored classic, plus converted from nhcb
prometheus_http_request_duration_secondsNative histogram queriesStored nhcb nhe

Where a histogram is both stored as classic and converted from NHCB, stored classic data wins, so nothing is counted twice. Selectors matching the classic suffixes (_bucket, _count, _sum) by exact metric name are converted.

Control matchers

__nhcb_as_classic__="debug"
Enables conversion (also =~"true|debug") and adds a __from_nhcb__ label ("true" for series converted from NHCB, "false" for stored classic series), e.g. response size rates by provenance.
__nhcb_as_classic__="false"
Turns the conversion off for the selector (also ="" or !="true"), which then returns the stored classic series as if the feature were disabled.
__nhcb_as_classic__="true"
Explicitly enables NHCB-to-classic conversion for the selector (also !="false").

1What is stored

Stored histograms

table

__nhcb_as_classic__="false" turns the conversion off, so the classic query only finds classic and classic-and-nhcb, as without the feature. Native histogram queries are not converted and return the stored NHCB histograms of nhcb and classic-and-nhcb, drawn as bucket charts.

Classictop panel
sum by (job) (rate(prometheus_http_request_duration_seconds_count{job=~"classic|nhcb|classic-and-nhcb", __nhcb_as_classic__="false"}[5m]))
Nativebottom panel
sum by (job) (prometheus_http_request_duration_seconds{job=~"classic|nhcb|classic-and-nhcb"})
Open in Prometheus

2Counts and quantiles work in both forms

Request rates

table

With the conversion on, __nhcb_as_classic__="debug" adds the __from_nhcb__ label ("true" for series converted from NHCB, "false" for stored classic series). The classic query returns all three jobs with the same rates as the native query, and classic-and-nhcb is not counted twice because stored classic data shadows converted NHCB.

Classictop panel
sum by (job, __from_nhcb__) (rate(prometheus_http_request_duration_seconds_count{job=~"classic|nhcb|classic-and-nhcb", __nhcb_as_classic__="debug"}[5m]))
Nativebottom panel
histogram_count(sum by (job) (rate(prometheus_http_request_duration_seconds{job=~"classic|nhcb|classic-and-nhcb"}[5m])))
Open in Prometheus

99th percentile

graph

Classic quantile queries work for nhcb alongside classic and classic-and-nhcb, and match the native query on nhcb and classic-and-nhcb because NHCB keeps the exact same custom bucket boundaries.

Classictop panel
histogram_quantile(0.99, sum by (job, le) (rate(prometheus_http_request_duration_seconds_bucket{job=~"classic|nhcb|classic-and-nhcb"}[5m])))
Nativebottom panel
histogram_quantile(0.99, sum by (job) (rate(prometheus_http_request_duration_seconds{job=~"classic|nhcb|classic-and-nhcb"}[5m])))
Open in Prometheus

3Buckets and le

Bucket series

table

Classic _bucket queries return le series for classic, classic-and-nhcb, and nhcb (converted from its custom bucket boundaries), while the native query returns a histogram per job (nhcb and classic-and-nhcb), drawn as a bucket chart.

Classictop panel
sum by (job, le) (rate(prometheus_http_request_duration_seconds_bucket{job=~"classic|nhcb|classic-and-nhcb"}[5m]))
Nativebottom panel
sum by (job) (rate(prometheus_http_request_duration_seconds{job=~"classic|nhcb|classic-and-nhcb"}[5m]))
Open in Prometheus

Filtering by le

table

The share of requests faster than 100ms, a common SLO query. Because NHCB preserves the classic bucket boundaries, le="0.1" works for nhcb as well as classic and classic-and-nhcb, matching histogram_fraction.

Classictop panel
sum by (job) (rate(prometheus_http_request_duration_seconds_bucket{job=~"classic|nhcb|classic-and-nhcb", le="0.1"}[5m])) / sum by (job) (rate(prometheus_http_request_duration_seconds_count{job=~"classic|nhcb|classic-and-nhcb"}[5m]))
Nativebottom panel
histogram_fraction(0, 0.1, sum by (job) (rate(prometheus_http_request_duration_seconds{job=~"classic|nhcb|classic-and-nhcb"}[5m])))
Open in Prometheus