OIDC for Grafana with HelmOctober 30, 2023 -
I've avoided installing an observability stack on my Kubernetes homelab1 — it's always seemed excessive in terms of the ratio of resources consumed to the scale of my operation.
Of course, as these things go, I have begun to rely on my self-hosted infrastructure more and more.
This became apparent when my GoToSocial instance (OSS fediverse server, compatible with Mastodon) went down due to the PostgreSQL instance running out of space on its PVC.
My backups — which yes, I have previously tested — had stopped working2. 🙀
Cut off from the fediverse, I couldn't even complain, so I was forced to repair things in a very un-cloud way (
kubectl exec && 🙏🏻).
During the post-mortem, which was absolutely not blameless3, I took an action item to set up an observability stack and here we are!
I use Authelia for SSO, so wanted to set up Grafana as part of
kube-prometheus-stack to use OpenID Connect (OIDC /
oauth2_generic in the Grafana config) for authentication.
🧠 The Authelia docs are a great, security-minded reference for all things authentication
After the "initial configuration", or, as I like to call it, copy 🍝 from Authelia's OIDC & Grafana guide, I was able to log in, but could not see or do anything. Realizing it was an RBAC/permissions issue, I headed to Grafana's generic OAuth2 guide.
ℹ️ An Important Note on Helm
In all the following YAML examples, if you are installing Grafana via
kube-prometheus-stack (which I recommend!), these should be under the
If you see this snippet in this blog post...
a: b: "c"
...and you're using the Grafana Helm chart directly, use as-is.
..and you're using
kube-prometheus-stack, in your
values.yaml, nest these under a
grafana: a: b: "c"
Binding Authelia OIDC
groups Scope to Grafana Roles
Instead of manually logging in as a special Grafana user and granting my SSO user admin permissions, I decided to quickly create a couple LDAP groups,
With these, I was able to tweak the example query slightly, set a couple other related options, and successfully log in as a server admin.
NOTE: Server Admin is a more powerful role than Org Admin in Grafana.
grafana.ini: auth.generic_oauth: # there's about to be a 👻 JSON query, by setting this to strict, # Grafana will reject logins if the expression errors # (fail closed) role_attribute_strict: true # LDAP group -> Grafana role mapping: # `grafana_admin` -> Grafana **Server** Admin # `grafana_editor` -> Grafana Editor # * -> Grafana Viewer role_attribute_path: contains(groups[*], 'grafana_admin') && 'GrafanaAdmin' || contains(groups[*], 'grafana_editor') && 'Editor' || 'Viewer' # required for the above query ⬆️ to be able to map to Grafana **Server** Admin allow_assign_grafana_admin: true
At this point, after logging out from Authelia, I was able to log back in with my new groups from LDAP applied and access Grafana as a server admin.
Although I was excited to get things working, I was a bit disappointed to not find much more guidance beyond that.
Removing Alternate Forms of Authentication
In my case, I want NO possible login except via OIDC. (If OIDC misbehaves, I'll patch the deployment/Helm chart. The risk of "break-the-glass" credentials is not worth it to me.)
First off, I wanted to get rid of the login form.
More importantly, I wanted to remove the possibility to use any sort of non-SSO credentials.
But also...I really wanted to get rid of the login form! I have enough RSI, thank you very much, I do not need to click an extra "Login with SSO" button every time.
grafana.ini: auth: # remove the login form from the sign-in page disable_login_form: true auth.basic: # disable HTTP Basic Authentication enabled: false auth.oauth_generic: # go directly to SSO! 🎉 (hint: `/login?disableAutoLogin`) auto_login: true
Also, because I'm paranoid and don't like that there's a weak default value, I went ahead and also set the (supposedly now unusable) admin account's password to a long, randomly-generated value4:
🙅🏻 No More Secrets in
Even though my flux2 repository is private, I treat it as though it's public and don't store any credentials in there.
I was surprised to see so many guides online setting the
client_secret for OIDC in plaintext directly in
Eventually, I discovered that Grafana allows setting config parameters via environment variables.
After an initially clumsy and fragile
postRenderer JSON 6902 patch, I realized that the Helm chart already supported loading environment variables from a secret.
Now, I use the 1Password/onepassword-operator, so I first created a secret in my connect vault with the fields
Then, the 1password CRD:
apiVersion: onepassword.com/v1 kind: OnePasswordItem metadata: name: grafana-oidc namespace: monitoring spec: itemPath: "vaults/notaphish-connect/items/grafana-oidc"
Lastly, reference it in your Helm
You can create the
Secret however you want — the important part is that the key names match the
GF_ field names above.
I feel pretty good about my Grafana deployment now!
Running on my forked Talos Linux for the Rockchip RK3588
Why are you suggesting that the restore from the backup had anything to do with the backups no longer working?
I have a cat and he's judgy
docker run authelia/authelia authelia crypto hash generate pbkdf2 --variant sha512 --random --random.length 72 --random.charset rfc3986 — but really refer to Authelia's docs on Generating Secure Values