Possible Grafana Plugin configuration deadlock

[fculpo]

What went wrong?

What happened:

You can deadlock grafana oncall plugin if first GRAFANA_API_URL is set to something unreachable or invalid TLS.

The plugin config screen tries to sync, fails, retries...

You see the wrong callback url in engine and celery logs

Disabling or uninstalling grafana plugin does not purge this broken config, if you reinstall it, it tries immediately to resync a broken config.

Celery and engine are still trying to reach invalid url.

I had to psql DELETE from plugin_settings WHERE plugin=oncall-app in grafana db to wipe any trace of this and reinstall plugin.

What did you expect to happen:

Oncall should update its callback url when changing GRAFANA_API_URL value.

How do we reproduce it?

Related to #2329

Grafana OnCall Version

at least 1.2.44 and 1.2.46

Product Area

Alert Flow & Configuration, Helm, Other

Grafana OnCall Platform?

Kubernetes

User's Browser?

No response

Anything else to add?

No response

[xom4ek]

You can change it via DB oncall engine

select grafana_url  from user_management_organization umo ;

update user_management_organization set grafana_url='https://mynewdomain.example.com' where id =1

[mderynck]

Recently we made some changes to the way Grafana OnCall is initialized. Use 1.9.22, there were quite a few changes along the way from 1.9.0-1.9.22 to get things working.

• If you are running Grafana 11 and newer you must have externalServiceAccounts feature toggle enabled.
  This has already been enabled in the docker compose files and helm charts in the oncall repo.
• Plugin settings must be provided to the plugin using an API call if you are installing for the first time (Note: credentials and hostnames need to be adjusted for your configuration, stackId and orgId are expected to be the listed constants in a self-hosted configuration)

curl -X POST 'http://admin:admin@localhost:3000/api/plugins/grafana-oncall-app/settings' -H "Content-Type: application/json" -d '{"enabled":true, "jsonData":{"stackId":5, "orgId":100, "onCallApiUrl":"http://engine:8080/", "grafanaUrl":"http://grafana:3000/"}}'

• Once settings are configured use this API call to install:

curl -X POST 'http://admin:admin@localhost:3000/api/plugins/grafana-oncall-app/resources/plugin/install'

Grafana OnCall should now be ready to use.
For additional troubleshooting see here