High Availability Setup (Spec)

[ntk148v]

HA Architecture (Quick Draft)

• Grafana HA under Loadbalancer (Haproxy/Nginx)

• To run Prometheus in a highly available manner, two (or more) instances need to be running with the same configuration, that means they scrape the same targets, which in turn means they will have the same data in memory and on disk, which in turn means they are answering requests the same way. These Prometheus instance is under Loadbalancer (Haproxy/Nginx).

• Prometheus instances only point to Grafana 1st instance as its data source. When Grafana 1st goes down, update existing datasource to Grafana 2nd instance.

• Alertmanager HA solution.

How to implement it? Here is the useful links to follow.

• AlertManager HA

• High Availability Prometheus Alerting and notifications (2 Prometheus - 2 Alertmanager).

• Setup Grafana HA.

• Update an existing data source.

• Scaling and Federate Prometheus

Things to do - Work items

• Update prometheus.yaml template if multiple prometheus and alertmanager was defined in inventory.

• Update Grafana datasource config.

• Add one more service to docker-compose: MySQL as Grafana dashboard database. Allow user to config it with grafana_database even it's in single mode.

• Setup Loadbalancers with Grafana and Promtheus instances as backend.

• Alertmanager HA config and setup!

Refer

• Gitlab Monitoring Architecture.

• Gitlab Issue

• Gitlab Handbook Monitoring

• Scaling out with Prometheus

[ntk148v]

• Loadbalancer: HaProxy + Keepalived.

  • Create role to setup HAProxy + Keepalived.
  • Update inventory.

• Grafana dashboard database.

  • Allow use external database -> Update group_vars
  • Setup MariaDB cluster with mariadb-galera image.