Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Prometheus: variables in adhoc filters not working since 10.2 version #87979

Open
cazeaux opened this issue May 16, 2024 · 3 comments · May be fixed by #88626
Open

Prometheus: variables in adhoc filters not working since 10.2 version #87979

cazeaux opened this issue May 16, 2024 · 3 comments · May be fixed by #88626
Labels
datasource/Prometheus

Comments

@cazeaux
Copy link

cazeaux commented May 16, 2024 •
edited

What happened?

When you use a variable as a value for an adhoc filter, the variable is not evaluated and the filter is applied with the variable name. The problem is seen with Prometheus datasource.

Problem seen first in 10.2.1, and confirmed in 10.4.2

What did you expect to happen?

The variable should be evaluated, and the filter applied with the variable value instead of its name.

Did this work before?

It was working at least up to version 10.1.2.

How do we reproduce it?

  1. Create a query variable, let's call it $platform. For example, a variable which provides values of a given label of a metric. For the example we use: `$platform = label_values(up, platform)
  2. Create an adhoc variable, and in this variable apply the filter : platform=~$platform
  3. In 10.1.2, the panels are correctly filtered with the values of $platform. See inspect below:

image

  1. In 10.2+, the panels are filtered with platform=~$platform. See inspect below:

image

An example dashboard to reproduce the bug (you might change platform label to another label):

{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": {
          "type": "grafana",
          "uid": "-- Grafana --"
        },
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "type": "dashboard"
      }
    ]
  },
  "editable": true,
  "fiscalYearStartMonth": 0,
  "graphTooltip": 0,
  "id": 1012,
  "links": [],
  "liveNow": false,
  "panels": [
    {
      "datasource": {
        "type": "prometheus",
        "uid": "paasquerier"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 0,
        "y": 0
      },
      "id": 1,
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "mode": "single",
          "sort": "none"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "paasquerier"
          },
          "editorMode": "code",
          "expr": "sum(up) by (platform)",
          "instant": false,
          "legendFormat": "__auto",
          "range": true,
          "refId": "A"
        }
      ],
      "title": "Panel Title",
      "type": "timeseries"
    }
  ],
  "refresh": "",
  "schemaVersion": 38,
  "style": "dark",
  "tags": [],
  "templating": {
    "list": [
      {
        "current": {
          "selected": true,
          "text": [
            "qp",
            "testbed",
            "tb-sd"
          ],
          "value": [
            "qp",
            "testbed",
            "tb-sd"
          ]
        },
        "datasource": {
          "type": "prometheus",
          "uid": "paasquerier"
        },
        "definition": "label_values(up,platform)",
        "hide": 0,
        "includeAll": true,
        "multi": true,
        "name": "platform",
        "options": [],
        "query": {
          "qryType": 1,
          "query": "label_values(up,platform)",
          "refId": "PrometheusVariableQueryEditor-VariableQuery"
        },
        "refresh": 1,
        "regex": "",
        "skipUrlSync": false,
        "sort": 0,
        "type": "query"
      },
      {
        "datasource": {
          "type": "prometheus",
          "uid": "paasquerier"
        },
        "filters": [
          {
            "key": "platform",
            "operator": "=~",
            "value": "$platform"
          }
        ],
        "hide": 0,
        "name": "platform_filter",
        "skipUrlSync": false,
        "type": "adhoc"
      }
    ]
  },
  "time": {
    "from": "now-6h",
    "to": "now"
  },
  "timepicker": {},
  "timezone": "browser",
  "title": "Grafana Bug",
  "uid": "ddlp24t4jkm4gd",
  "version": 11,
  "weekStart": ""
}

Is the bug inside a dashboard panel?

No

Environment (with versions)?

Grafana: 10.2.1 and recent versions
OS: Windows 10
Browser: Edge, Vivaldi, Chrome, Firefox

Grafana platform?

Kubernetes

Datasource(s)?

prometheus
@cazeaux
Copy link
Author

cazeaux commented May 27, 2024

Just an update to report that the issue is also present in Grafana 11.0.0

@cazeaux
Copy link
Author

cazeaux commented Jun 3, 2024 •
edited

Hello

I have found where the regression happened.
It has been introduced in v10.1.6, with this commit: ae58128#diff-79a8e0c726aee282de46b93b30ea4022261351ea9c0d46e6f988296cfcf5c900

We see in this commit that the replace happens before the enhanceExprWithAdHocFilters. So this is why the variables are not replaces anymore.

The same inversion is also there in the last version, even if the code has significantly changed :

grafana/packages/grafana-prometheus/src/datasource.ts

Lines 844 to 851 in c73bbf1

const expr = this.templateSrv.replace(target.expr, variables, this.interpolateQueryExpr);
return {
...target,
...(config.featureToggles.promQLScope ? { adhocFilters: this.generateScopeFilters(filters) } : {}),
expr: config.featureToggles.promQLScope ? expr : this.enhanceExprWithAdHocFilters(filters, expr),
interval: this.templateSrv.replace(target.interval, variables),
legendFormat: this.templateSrv.replace(target.legendFormat, variables),

@cazeaux cazeaux changed the title Dashboards: variables in adhoc filters not working since 10.2 version Prometheus: variables in adhoc filters not working since 10.2 version Jun 3, 2024
cazeaux added a commit to cazeaux/grafana that referenced this issue Jun 3, 2024
@cazeaux
Prometheus: replace variables on adhoc filters
ace86e9 
Fixes grafana#87979

Signed-off-by: Stéphane Cazeaux <stephane.cazeaux@orange.com>
cazeaux added a commit to cazeaux/grafana that referenced this issue Jun 3, 2024
@cazeaux
Prometheus: replace variables on adhoc filters
b081a9b 
Fixes grafana#87979

Signed-off-by: Stéphane Cazeaux <stephane.cazeaux@orange.com>
@cazeaux cazeaux linked a pull request Jun 3, 2024 that will close this issue
Open
@tonypowa
Copy link
Contributor

tonypowa commented Jun 3, 2024

thanks for reporting this issue and for your contribution @cazeaux 🎉
I've added the issue to the team's board

cazeaux added a commit to cazeaux/grafana that referenced this issue Jun 3, 2024
@cazeaux
Prometheus: replace variables on adhoc filters
8f01489 
Fixes grafana#87979

Signed-off-by: Stéphane Cazeaux <stephane.cazeaux@orange.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
datasource/Prometheus
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Prometheus: replace variables on adhoc filters cazeaux/grafana
2 participants
@cazeaux @tonypowa