Let's continue from the previous article and write yet another watcher. In my previous experience, this watcher has helped to keep track of the status of my server without logging into Kibana every day.

Here is the watcher that is triggered every 24hr, and alerts when the treshold is above 90% of CPU or memory:

{
  "trigger": {
    "schedule": {
      "interval": "24h"
    }
  },
  "input": {
    "search": {
      "request": {
        "indices": [ "metricbeat-*" ],
        "body": {
          "query": {
            "bool": {
              "filter": {
                "range": {
                  "@timestamp": {
                    "gte": "now-24h",
                    "lt": "now"
                  }
                }
              }
            }
          },
          "aggs": {
            "cpu_usage": {
              "avg": {
                "field": "system.cpu.user.pct"
              }
            },
            "memory_usage": {
              "avg": {
                "field": "system.memory.used.pct"
              }
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.aggregations.cpu_usage.value": {
        "gt": 90
      }
    }
  },
  "actions": {
    "send_email": {
      "email": {
        "to": "admin@example.com",
        "subject": "High CPU Usage Alert",
        "body": "The average CPU usage over the last 24 hours was {{ctx.payload.aggregations.cpu_usage.value}}%. The average memory usage was {{ctx.payload.aggregations.memory_usage.value}}%."
      }
    }
  }
}

Hope this helps you and be ready with preventive action in advance. You may choose to change the condition to a lower number such as 70 or 80, in case you need more time to come up with a plan to fix it.

Feel free to share your thoughts and feedback on comments or Twitter at @smit_shah_95