Graduate native histogram ingestion and close the v1-guarantees gaps (#7789)

* Graduate native histogram ingestion and close the v1-guarantees gaps

-blocks-storage.tsdb.enable-native-histograms shipped as experimental in 1.18.0, so
1.22.0 is four minors and roughly two years of exposure later. docs/guides/native-histograms.md
already carries no experimental language, which contradicted v1-guarantees.md. Graduate the
ingestion flag; the flag name does not change, so no deprecation shim is needed. This does not
touch -blocks-storage.tsdb.out-of-order-cap-max, which has its own bullet.

Also record the experimental flags this cycle added without a v1-guarantees entry:

- -distributor.num-query-workers (#7623), marked EXPERIMENTAL in its flag help.
- The whole Thanos PromQL engine surface. -querier.thanos-engine says "Experimental." in its
  help but nothing from pkg/engine/config.go was listed.
- The whole parquet flag surface; only -parquet-converter.max-num-columns was listed. Grouped
  by prefix rather than enumerated, since the cache flags are boilerplate.
- -querier.query-protection.rejection (#7442) and
  -querier.query-protection.eviction.max-evictions-per-cycle (#7488).

Parquet and the tenant-federation regex matcher are deliberately not graduated: the parquet
proposal is still "Status: Proposed", docs/guides/parquet-mode.md still warns it is experimental,
and #7610 changed the file format this cycle. The regex matcher has only two minors of exposure
and #7562 fixed cross-tenant stale cache reads this cycle.

Refs #5922

Signed-off-by: Charlie Le <charlie_le@apple.com>

* Drop [EXPERIMENTAL] from -blocks-storage.tsdb.enable-native-histograms help

Addresses PR review feedback: the flag is graduated in this PR, so the
CLI help text should no longer mark it experimental. Regenerated config
docs and JSON schema.

Signed-off-by: Charlie Le <charlie_le@apple.com>

---------

Signed-off-by: Charlie Le <charlie_le@apple.com>

Author: CharlieTLe

CHANGELOG.md

@@ -13,6 +13,7 @@
   - `-blocks-storage.tsdb.wal-compression-enabled` (deprecated in 1.19.0; use `-blocks-storage.tsdb.wal-compression-type`)
   - `-ingester.max-series-per-query` (a chunks-storage limit, ignored since blocks storage; use `-querier.max-fetched-series-per-query`)
 * [CHANGE] Ingester: Formally deprecate `-blocks-storage.tsdb.max-exemplars`, scheduled for removal in v1.24.0. Use the per-tenant `max_exemplars` limit instead. The flag still works as the global fallback when `max_exemplars` is 0, but setting it now logs a warning and increments `deprecated_flags_inuse_total`. #7793
+* [CHANGE] Ingester: Graduate native histogram ingestion (`-blocks-storage.tsdb.enable-native-histograms`) from experimental. #7789
 * [CHANGE] Querier: Make query time range configurations per-tenant: `query_ingesters_within`, `query_store_after`, and `shuffle_sharding_ingesters_lookback_period`. Uses `model.Duration` instead of `time.Duration` to support serialization but has minimum unit of 1ms (nanoseconds/microseconds not supported). #7160
 * [CHANGE] Cache: Setting `-blocks-storage.bucket-store.metadata-cache.bucket-index-content-ttl` to 0 will disable the bucket-index cache. #7446
 * [CHANGE] HA Tracker: Move `-distributor.ha-tracker.failover-timeout` from a global config to a per-tenant runtime config. The flag name and default value (30s) remain the same. #7481

docs/configuration/config-file-reference.md

@@ -4623,7 +4623,7 @@ The `limits_config` configures default and per-tenant limits imposed by Cortex s
 # series matching its matchers and exposes the count as a metric.
 [active_series_trackers: <list of ActiveSeriesTrackerConfig> | default = []]
 
-# [EXPERIMENTAL] True to enable native histogram.
+# True to enable native histogram.
 # CLI flag: -blocks-storage.tsdb.enable-native-histograms
 [enable_native_histograms: <boolean> | default = false]
 

docs/configuration/v1-guarantees.md

@@ -52,6 +52,7 @@ Currently experimental features are:
   - Do not extend writes on unhealthy ingesters (`-distributor.extend-writes=false`)
   - Accept multiple HA pairs in the same request (enabled via `-experimental.distributor.ha-tracker.mixed-ha-samples=true`)
   - Accept Prometheus remote write 2.0 request (`-distributor.remote-writev2-enabled=true`)
+  - Use a goroutine worker pool for query fan-out calls to ingesters (`-distributor.num-query-workers`)
 - Tenant Deletion in Purger, for blocks storage.
 - Blocks storage user index
 - Store Gateway max concurrent fetched data bytes limit
@@ -103,8 +104,6 @@ Currently experimental features are:
   - Ingest delta temporality OTLP metrics (`-distributor.otlp.allow-delta-temporality=true`)
 - Persistent tokens in the Ruler Ring:
   - `-ruler.ring.tokens-file-path` (path) CLI flag
-- Native Histograms
-  - Ingestion can be enabled by setting `-blocks-storage.tsdb.enable-native-histograms=true` on Ingester.
 - String interning for metrics labels
   - Enable string interning for metrics labels by setting `-ingester.labels-string-interning-enabled` on Ingester.
 - Query-frontend: query rejection (`-frontend.query-rejection.enabled`)
@@ -113,9 +112,10 @@ Currently experimental features are:
 - Query-frontend: dynamic query splits
   - `querier.max-shards-per-query` (int) CLI flag
   - `querier.max-fetched-data-duration-per-query` (duration) CLI flag
-- Ingester/Store-Gateway: Query rejection
+- Ingester/Store-Gateway/Querier: Query rejection
   - `-ingester.query-protection.rejection`
   - `-store-gateway.query-protection.rejection`
+  - `-querier.query-protection.rejection`
 - Distributor/Ingester: Stream push connection
   - Enable stream push connection between distributor and ingester by setting `-distributor.use-stream-push=true` on Distributor.
   - Enable stream push authentication on Distributor/Ingester. (`-distributor.sign-write-requests-keys`)
@@ -141,6 +141,7 @@ Currently experimental features are:
   - `-querier.query-protection.eviction.cooldown-period` (int)
   - `-querier.query-protection.eviction.eviction-metric` (string)
   - `-querier.query-protection.eviction.min-query-age` (duration)
+  - `-querier.query-protection.eviction.max-evictions-per-cycle` (int)
 - Ingester: Active Series Tracker
   - Per-tenant `active_series_trackers` configuration in runtime config overrides
   - Counts active series matching PromQL label matchers and exposes `cortex_ingester_active_series_per_tracker` metric
@@ -154,6 +155,19 @@ Currently experimental features are:
   - `-ingester.head-queried-series-metrics-windows` time windows to report (default: 2h)
   - `-ingester.head-queried-series-metrics-window-duration` HLL sub-window size
   - `-ingester.head-queried-series-metrics-sample-rate` query sampling rate
-- Parquet Converter: Maximum number of columns per file
-  - `-parquet-converter.max-num-columns` (int) CLI flag
-  - Automatically shards parquet files when the number of columns exceeds the configured limit
+- Parquet storage
+  - Parquet Converter: the `-parquet-converter.*` CLI flags, including `-parquet-converter.enabled`,
+    `-parquet-converter.max-num-columns` (automatically shards parquet files when the number of columns
+    exceeds the configured limit) and the `-parquet-converter.ring.*` ring configuration
+  - Querier: `-querier.parquet-queryable-default-block-store`, `-querier.parquet-queryable-fallback-disabled`,
+    the `-querier.parquet-queryable.max-fetched-*` limits and `-querier.parquet-shard-cache-*`
+  - Store Gateway: `-blocks-storage.bucket-store.parquet-query-concurrency`,
+    `-blocks-storage.bucket-store.parquet-shard-cache-*` and the
+    `-blocks-storage.bucket-store.parquet-labels-cache.*` /
+    `-blocks-storage.bucket-store.parquet-row-ranges-cache.*` cache configuration
+- Querier/Ruler: Thanos PromQL engine
+  - `-querier.thanos-engine` / `-ruler.thanos-engine` (boolean) CLI flags
+  - `-querier.enable-x-functions` / `-ruler.enable-x-functions` (boolean) CLI flags
+  - `-querier.optimizers` / `-ruler.optimizers` (string) CLI flags
+  - `-querier.decoding-concurrency` / `-ruler.decoding-concurrency` (int) CLI flags
+  - `-querier.selector-batch-size` / `-ruler.selector-batch-size` (int) CLI flags

pkg/util/validation/limits.go

@@ -313,7 +313,7 @@ func (l *Limits) RegisterFlags(f *flag.FlagSet) {
 	f.IntVar(&l.MaxGlobalSeriesPerMetric, "ingester.max-global-series-per-metric", 0, "The maximum number of active series per metric name, across the cluster before replication. 0 to disable.")
 	f.IntVar(&l.MaxLocalNativeHistogramSeriesPerUser, "ingester.max-native-histogram-series-per-user", 0, "The maximum number of active native histogram series per user, per ingester. 0 to disable. Supported only if ingester.active-series-metrics-enabled is true.")
 	f.IntVar(&l.MaxGlobalNativeHistogramSeriesPerUser, "ingester.max-global-native-histogram-series-per-user", 0, "The maximum number of active native histogram series per user, across the cluster before replication. 0 to disable. Supported only if -distributor.shard-by-all-labels and ingester.active-series-metrics-enabled is true.")
-	f.BoolVar(&l.EnableNativeHistograms, "blocks-storage.tsdb.enable-native-histograms", false, "[EXPERIMENTAL] True to enable native histogram.")
+	f.BoolVar(&l.EnableNativeHistograms, "blocks-storage.tsdb.enable-native-histograms", false, "True to enable native histogram.")
 	f.IntVar(&l.MaxExemplars, "ingester.max-exemplars", 0, "Enables support for exemplars in TSDB and sets the maximum number that will be stored. less than zero means disabled. If the value is set to zero, cortex will fallback to the deprecated blocks-storage.tsdb.max-exemplars value; that fallback is removed in v1.24.0.")
 	f.Var(&l.OutOfOrderTimeWindow, "ingester.out-of-order-time-window", "[Experimental] Configures the allowed time window for ingestion of out-of-order samples. Disabled (0s) by default.")
 

schemas/cortex-config-schema.json

@@ -5670,7 +5670,7 @@
         },
         "enable_native_histograms": {
           "default": false,
-          "description": "[EXPERIMENTAL] True to enable native histogram.",
+          "description": "True to enable native histogram.",
           "type": "boolean",
           "x-cli-flag": "blocks-storage.tsdb.enable-native-histograms"
         },