Diagnostic service and application stage four

bridgehead/README.md

@@ -5,6 +5,8 @@ This guide provides instructions for deploying the core UDMI services bundle. Th
 - etcd server 
 - udmis service 
 - validator service (includes registrar tool)
+- InfluxDB timeseries database
+- Grafana visualisation tool
 
 *Note: docker logs display file paths as seen inside the container. These are internal paths and do not match your local machine's file system. Files with local access exist via volume mapping; check your docker-compose.yml for the corresponding host path. Example: The common container path /root/site_model/ exists locally under bridgehead/udmi_site_model*
 
@@ -16,16 +18,23 @@ This guide provides instructions for deploying the core UDMI services bundle. Th
 
 3. **Get default site model:** In you terminal, run `sudo git clone https://github.com/faucetsdn/udmi_site_model.git`.
 
-4. **Add your host ip:** Open the docker-compose.yml file and locate the line `HOST_IP: <YOUR_IP>` inside the mosquitto service block. Replace `<YOUR_IP>` with your hosts ip address. You can find this by running `sudo hostname -I`. This is required in order tp allow connections to the broker externally from the docker compose environment. 
+4. **Edit compose file:** Open the docker-compose.yml file in your chosen editor.
+    1. **Add Host IP:** Locate the line `HOST_IP: <YOUR_IP>` inside the **mosquitto** service block. Replace `<YOUR_IP>` with your hosts ip address. You can find this by running `sudo hostname -I`. This is required in order to allow connections to the broker externally from the docker compose environment. 
+    2. **Update InfluxDB Credentials:** Under the **influxdb** service, in environment variables, set the values of `DOCKER_INFLUXDB_INIT_USERNAME`, `DOCKER_INFLUXDB_INIT_PASSWORD` and `DOCKER_INFLUXDB_INIT_ADMIN_TOKEN`. For `DOCKER_INFLUXDB_INIT_ADMIN_TOKEN`, run `openssl rand -hex 32`. *Note: The password for influx must be between 8-72 characters.*
+    3. **Update Grafana credentials:** Under the **grafana** service, set the values of `GF_SECURITY_ADMIN_USER` and `GF_SECURITY_ADMIN_USER`. Update the value of `INFLUXDB_TOKEN` with the token generated in step 2.
+    4. **Update udmis credentials:** Under the **udmis** service, set the value of `INFLUXDB_TOKEN` with the token generated in step 2.
 
 5. **Deploy the service:** Execute the following command to build the custom images (if needed) and start the containers in detached mode.
     * **First time/after changes:** Run `sudo docker compose up -d --build`
     * **Standard run:** Run `sudo docker compose up -d`
+
 6. **Confirm all containers are running:** Run `sudo docker ps` in the terminal, you should see the following containers in any order:
    - validator
    - udmis
    - mosquitto
    - etcd
+   - grafana
+   - influxdb
 
 ## Tools
 
@@ -63,9 +72,14 @@ Pubber is running successfully if there are no obvious error messages or retries
 A successful run will not end on its own, you can press `Ctrl` + `C` on your keyboard to exit. 
 
 You can stop the pubber container by running `sudo docker stop pubber`
+
+### Validator
+
+To run the validator service in the background, execute `sudo docker exec -d validator bin/validator site_model/ //mqtt/mosquitto`
+
 ### Discovery
 
-In the following example we are passing GAT-123 as the discovery node ip to doscovery.sh.
+In the following example we are passing GAT-123 as the discovery node ip to discovery.sh.
 
 Run the following commands to complete a discovery sequence: 
 
@@ -96,6 +110,66 @@ Summary:
 Out of 4 total.
 ```
 
+## Diagnostics
+
+### InfluxDB
+
+1. **Access:** Navigate to the InfluxDB endpoint at `http://localhost:8086`.
+2. **Login:** Use the credentials specified in the Docker Compose file.
+3. **Query:** Go to the Data Explorer (left panel). Locate the home bucket at the bottom to begin querying data.
+
+Although InfluxDB is available and accessible, there should be no need to directly interact with it.
+
+### Grafana 
+
+1. **Access:** Navigate to the Grafana endpoint at `http://localhost:3000`.
+2. **Login:** Use the credentials specified in the Docker Compose file.
+3. **View Dashboard:**  Select Dashboards from the left-hand menu. The pre-configured `bridgehead` dashboard includes a basic overview of the running services.
+
+Note about the default dashboard: The MQTT graphs only display data that falls within the currently selected time range. For instance, if the connection count has been stable for the last hour, but the dashboard time range is set to the last five minutes, the graphs will remain empty. To see the history, simply expand the grafana time range to include the last time activity was recorded.
+
+#### Alerts
+
+The following instructions will setup an alert in Grafana for when a container goes down.
+
+1. **Create new contact point:** 
+    1. In the left-hand pane go to `Alerting -> Contact points`
+    2. Navigate to `https://webhook.site` and copy the URL under "Your unique URL".
+    3. In Grafana, click "Create contact point", name it Webhook, set the type to webhook and paste in the URL you just copied. Test your webhook with the test button, you should see a test webhook appear in `https://webhook.site`. Save the new contact point.
+2. **Create new alert rule:** 
+    1. In the left-hand pane go to `Alerting -> Alert rules`. 
+    2. Click on "New alert rule" and name the rule `Container Down`.
+    3. Set query data source to be InfluxDB, then paste the following flux query:
+    ```
+    from(bucket: "home")
+    |> range(start: -5m) 
+    |> filter(fn: (r) => r["_measurement"] == "docker_container_status")
+    |> filter(fn: (r) => r["_field"] == "uptime_ns") 
+    |> filter(fn: (r) => r["com.docker.compose.project"] == "bridgehead")
+    |> group(columns: ["container_name"]) 
+    |> last()
+    |> map(fn: (r) => ({
+        r with
+        alert_value: if r.container_status == "exited" or r.container_status == "dead" then 1.0 else 0.0
+    }))
+    |> keep(columns: ["container_name", "alert_value"]) 
+    |> group()
+    ```
+
+    This query summarizes the status of Docker containers. It finds the most recent status for each container and outputs a value of 1.0 if the container is currently 'exited' or 'dead', and 0.0 otherwise. This binary output is then used to trigger the alert.
+
+    4. Set the `Alert condition` to be `WHEN Last OF QUERY IS EQUAL TO 1`. When you click `Preview alert rule condition` you should see each container in a "Normal" state.
+    5. In section 3, `Add folder and labels`, create a new folder "Alerts" and select this folder.
+    6. in section 4, `Set evaluation behaviour`, create an evaluation group "Bridgehead Alerts" and set the evaluation interval to 30s. Set the pending period to 30s, Keep firing for to 0s.
+    7. In section 5, select "Webhook" as the contact point and then save.
+
+3. **Edit default notification policy:** By default a notification won’t be sent for 5 min. Update this by going to `Alerting -> Notification policies` edit the default policy so that the Group interval is set to 30s, and update.
+
+4. **Test:** Go back to the bridgehead dashboard and confirm all containers are running. In your terminal, run `sudo docker stop validator`. Keep an eye on the validator container, it should go red and show an uptime of 0s. Navigate to `https://webhook.site`, you should have received a notification with the line `"title": "[FIRING:1] Container Down Alerts (validator)",` (This could take a minute or two).
+7. Run validator container again with `sudo docker start validator`
+
+Use the same method to setup any number of alerts, you can find useful information here: https://grafana.com/docs/grafana/latest/alerting/
+
 ## Shutting down the docker environment
 
 To gracefully stop and remove the container, run: `sudo docker compose down`

bridgehead/docker-compose.yml

@@ -38,6 +38,7 @@ services:
       - ./var/etcd:/root/udmi/var/etcd
       - ./var/mosquitto/log:/var/log/mosquitto
       - ./var/mosquitto/certs:/etc/mosquitto/certs
+      - /var/run/docker.sock:/var/run/docker.sock:ro
     depends_on:
       etcd:
         condition: service_started
@@ -47,9 +48,12 @@ services:
     networks:
       - udminet
     environment:
-      ETCD_CLUSTER: etcd
-      MQTT_HOST: mosquitto
-      MQTT_PORT: 8883
+      - ETCD_CLUSTER=etcd
+      - MQTT_HOST=mosquitto
+      - MQTT_PORT=8883
+      - INFLUXDB_TOKEN=
+      - INFLUXDB_ORG=bridgehead
+      - INFLUXDB_BUCKET=home
 
   validator:
     build: 
@@ -66,7 +70,49 @@ services:
     restart: always
     networks:
       - udminet
-
+
+  influxdb:
+    container_name: influxdb
+    image: influxdb:2
+    depends_on:
+      udmis:
+        condition: service_started
+    volumes:
+      - ./var/influxdb-storage:/var/lib/influxdb
+      - ./var/mosquitto/certs:/etc/mosquitto/certs
+    environment:
+      - DOCKER_INFLUXDB_INIT_MODE=setup
+      - DOCKER_INFLUXDB_INIT_USERNAME=bridgehead
+      - DOCKER_INFLUXDB_INIT_PASSWORD=password
+      - DOCKER_INFLUXDB_INIT_BUCKET=home
+      - DOCKER_INFLUXDB_INIT_ORG=bridgehead
+      - DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=
+    restart: always
+    ports:
+      - "8086:8086"
+    networks:
+      - udminet
+
+  grafana:
+    container_name: grafana
+    image: grafana/grafana:12.3
+    volumes:
+      - ./grafana/datasources:/etc/grafana/provisioning/datasources
+      - ./grafana/dashboards:/etc/grafana/provisioning/dashboards
+    depends_on:
+      - influxdb
+    environment:
+      - INFLUXDB_ORGANISATION=bridgehead
+      - INFLUXDB_BUCKET=home
+      - INFLUXDB_TOKEN=
+      - GF_SECURITY_ADMIN_USER=bridgehead
+      - GF_SECURITY_ADMIN_PASSWORD=password
+    restart: always
+    ports:
+      - "3000:3000"
+    networks:
+      - udminet
+
 networks:
   udminet:
     name: udminet