OIDC for Grafana with Helm

I've avoided installing an observability stack on my Kubernetes homelab¹ — it's always seemed excessive in terms of the ratio of resources consumed to the scale of my operation.

Of course, as these things go, I have begun to rely on my self-hosted infrastructure more and more.

This became apparent when my GoToSocial instance (OSS fediverse server, compatible with Mastodon) went down due to the PostgreSQL instance running out of space on its PVC.

My backups — which yes, I have previously tested — had stopped working². 🙀

Cut off from the fediverse, I couldn't even complain, so I was forced to repair things in a very un-cloud way (kubectl exec && 🙏🏻).

During the post-mortem, which was absolutely not blameless³, I took an action item to set up an observability stack and here we are!

I use Authelia for SSO, so wanted to set up Grafana as part of kube-prometheus-stack to use OpenID Connect (OIDC / oauth2_generic in the Grafana config) for authentication.

🧠 The Authelia docs are a great, security-minded reference for all things authentication

After the "initial configuration", or, as I like to call it, copy 🍝 from Authelia's OIDC & Grafana guide, I was able to log in, but could not see or do anything. Realizing it was an RBAC/permissions issue, I headed to Grafana's generic OAuth2 guide.

ℹ️ An Important Note on Helm values.yaml: grafana/grafana vs prometheus-community/kube-prometheus-stack

In all the following YAML examples, if you are installing Grafana via kube-prometheus-stack (which I recommend!), these should be under the grafana: key.

If you see this snippet in this blog post...

a:
  b: "c"

...and you're using the Grafana Helm chart directly, use as-is.

..and you're using kube-prometheus-stack, in your values.yaml, nest these under a grafana key:

grafana:
  a:
    b: "c"

Binding Authelia OIDC groups Scope to Grafana Roles

Instead of manually logging in as a special Grafana user and granting my SSO user admin permissions, I decided to quickly create a couple LDAP groups, grafana_admin and grafana_editor.

With these, I was able to tweak the example query slightly, set a couple other related options, and successfully log in as a server admin.

NOTE: Server Admin is a more powerful role than Org Admin in Grafana.

grafana.ini:
  auth.generic_oauth:
    # there's about to be a 👻 JSON query, by setting this to strict,
    # Grafana will reject logins if the expression errors
    # (fail closed)
    role_attribute_strict: true

    # LDAP group -> Grafana role mapping:
    # `grafana_admin` -> Grafana **Server** Admin
    # `grafana_editor` -> Grafana Editor
    # * -> Grafana Viewer
    role_attribute_path: contains(groups[*], 'grafana_admin') && 'GrafanaAdmin' || contains(groups[*], 'grafana_editor') && 'Editor' || 'Viewer'

    # required for the above query ⬆️ to be able to map to Grafana **Server** Admin
    allow_assign_grafana_admin: true

At this point, after logging out from Authelia, I was able to log back in with my new groups from LDAP applied and access Grafana as a server admin.

Although I was excited to get things working, I was a bit disappointed to not find much more guidance beyond that.

Removing Alternate Forms of Authentication

In my case, I want NO possible login except via OIDC. (If OIDC misbehaves, I'll patch the deployment/Helm chart. The risk of "break-the-glass" credentials is not worth it to me.)

First off, I wanted to get rid of the login form.

More importantly, I wanted to remove the possibility to use any sort of non-SSO credentials.

But also...I really wanted to get rid of the login form! I have enough RSI, thank you very much, I do not need to click an extra "Login with SSO" button every time.

grafana.ini:
  auth:
    # remove the login form from the sign-in page 
    disable_login_form: true
  
  auth.basic:
    # disable HTTP Basic Authentication
    enabled: false

  auth.oauth_generic:
    # go directly to SSO! 🎉 (hint: `/login?disableAutoLogin`)
    auto_login: true

Also, because I'm paranoid and don't like that there's a weak default value, I went ahead and also set the (supposedly now unusable) admin account's password to a long, randomly-generated value⁴:

adminPassword: ||replace-me-with-a-secure-value||

🙅🏻 No More Secrets in values.yaml

Even though my flux2 repository is private, I treat it as though it's public and don't store any credentials in there.

I was surprised to see so many guides online setting the client_secret for OIDC in plaintext directly in values.yaml.

Eventually, I discovered that Grafana allows setting config parameters via environment variables.

After an initially clumsy and fragile postRenderer JSON 6902 patch, I realized that the Helm chart already supported loading environment variables from a secret.

Now, I use the 1Password/onepassword-operator, so I first created a secret in my connect vault with the fields GF_AUTH_GENERIC_OAUTH_CLIENT_ID and GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET.

Then, the 1password CRD:

apiVersion: onepassword.com/v1
kind: OnePasswordItem
metadata:
  name: grafana-oidc
  namespace: monitoring
spec:
  itemPath: "vaults/notaphish-connect/items/grafana-oidc"

Lastly, reference it in your Helm values.yaml:

envFromSecret: grafana-oidc

You can create the Secret however you want — the important part is that the key names match the GF_ field names above.

✅

I feel pretty good about my Grafana deployment now!