[jvoegele]

Each push now adds a scope extra_label for the metric set it carries, in both the single-pusher and split-pusher setups. Observability can then drop tenant metrics from the central forward without touching cluster metrics.

Part of REAL-1050.

What kind of change does this PR introduce?

Feature.

What is the current behavior?

The pusher sends global and tenant metrics as separate requests, but both carry the same labels (service=realtime,cluster=<name>). VictoriaMetrics can't tell the two sets apart, so Observability has no way to drop one and keep the other. See the Slack thread.

What is the new behavior?

Each push adds extra_label=scope=global or extra_label=scope=tenant, based on the metric set in that request. This works with today's single pusher and with the split pushers. No config changes are needed.

Additional context

• A new label starts new series for every pushed metric. Queries that aggregate with sum by (...) are unaffected. Panels that plot raw series may show a break at deploy.
• Tested with mix test test/realtime/metrics_pusher_test.exs. A new test covers the single-pusher case, and the :global and :tenant tests now also check the label.

🤖 Generated with Claude Code

[github-actions]

CRAP Score Report

Summary: files=247 functions=1375 scored=1375 worst_score=15944.76

** (Mix) CRAP threshold failed: max_score=30.00
High scores: 30
  lib/extensions/postgres_cdc_rls/subscription_manager.ex Extensions.PostgresCdcRls.SubscriptionManager.handle_info/2 score=37.16
  lib/realtime/adapters/postgres/oid_database.ex Realtime.Adapters.Postgres.OidDatabase.name_for_type_id/1 score=15944.76
  lib/realtime/application.ex Realtime.Application.setup_region_mapping/0 score=47.11
  lib/realtime/nodes.ex Realtime.Nodes.default_region_mapping/1 score=129.85
  lib/realtime/operations.ex Realtime.Operations.rebalance/0 score=42.00
  lib/realtime/operations.ex Realtime.Operations.kill_connections_to_tenant_id/2 score=90.00
  lib/realtime/tenants/connect.ex Realtime.Tenants.Connect.handle_info/2 score=41.41
  lib/realtime_web/channels/auth/jwt_verification.ex RealtimeWeb.JwtVerification.generate_signer/3 score=30.78
  lib/realtime_web/channels/realtime_channel.ex RealtimeWeb.RealtimeChannel.handle_info/2 score=51.06
  lib/realtime_web/dashboard/feature_flags.ex RealtimeWeb.Dashboard.FeatureFlags.handle_event/3 score=112.70
  lib/realtime_web/dashboard/node_info.ex RealtimeWeb.Dashboard.NodeInfo.fetch_node_data/2 score=42.00
  lib/realtime_web/dashboard/recon_trace.ex RealtimeWeb.Dashboard.ReconTrace.handle_event/3 score=552.00
  lib/realtime_web/dashboard/recon_trace.ex RealtimeWeb.Dashboard.ReconTrace.handle_info/2 score=210.00
  lib/realtime_web/dashboard/recon_trace.ex RealtimeWeb.Dashboard.ReconTrace.render_value/2 score=42.00
  lib/realtime_web/dashboard/recon_trace.ex RealtimeWeb.Dashboard.ReconTrace.load_module_functions/1 score=42.00
  lib/realtime_web/dashboard/recon_trace.ex RealtimeWeb.Dashboard.ReconTrace.parse_and_start/2 score=110.00
  lib/realtime_web/dashboard/recon_trace.ex RealtimeWeb.Dashboard.ReconTrace.format_value/1 score=552.00
  lib/realtime_web/dashboard/recon_trace.ex RealtimeWeb.Dashboard.ReconTrace.sort_entries/2 score=42.00
  lib/realtime_web/dashboard/sql_inspector.ex RealtimeWeb.Dashboard.SqlInspector.handle_event/3 score=72.00
  lib/realtime_web/dashboard/sql_inspector.ex RealtimeWeb.Dashboard.SqlInspector.execute_read_only/1 score=72.00
  lib/realtime_web/dashboard/sql_inspector.ex RealtimeWeb.Dashboard.SqlInspector.mask_sensitive_columns/1 score=42.00
  lib/realtime_web/dashboard/sql_inspector.ex RealtimeWeb.Dashboard.SqlInspector.compare_cells/2 score=42.00
  lib/realtime_web/dashboard/sql_inspector.ex RealtimeWeb.Dashboard.SqlInspector.format_cell/1 score=72.00
  lib/realtime_web/dashboard/tenant_migrations.ex RealtimeWeb.Dashboard.TenantMigrations.handle_info/2 score=156.00
  lib/realtime_web/live/components.ex RealtimeWeb.Components.input/1 score=90.00
  lib/realtime_web/live/inspector_live/conn_component.ex RealtimeWeb.InspectorLive.ConnComponent.handle_event/3 score=53.83
  lib/realtime_web/live/inspector_live/event_log_component.ex RealtimeWeb.InspectorLive.EventLogComponent.category_variant/1 score=35.00
  lib/realtime_web/live/inspector_live/event_log_component.ex RealtimeWeb.InspectorLive.EventLogComponent.event_label/1 score=51.43
  lib/realtime_web/live/inspector_live/index.ex RealtimeWeb.InspectorLive.Index.handle_event/3 score=30.84
  lib/realtime_web/live/status_live/index.ex RealtimeWeb.StatusLive.Index.handle_event/3 score=42.00

[coveralls]

Coverage is 92.466% — jvoegele/real-1050-metrics-scope-label into main. No base build found for main.

[jvoegele]

Closing: Observability is dropping all service=realtime metrics from the central forward, and the existing tenant label covers any tenant matching. See the #team-realtime thread.