AI in monitoring: faster alarm handling

AI is no longer a "vision of the future," but an everyday reality: machine learning, deep learning, language processing, and machine vision have been with us for years, and in recent years, generative AI has brought spectacular leaps in content automation, efficiency gains, and faster decision support. The question in IT operations is simple: how can this be translated into tangible benefits in daily error handling and reducing monitoring noise?

Why is central monitoring key?

Almost every organization needs central monitoring because it:

• the status of all systems, devices, and services can be viewed in one place,
• provides a real-time picture of operations,
• supports rapid fault detection and immediate alerts,
• forms the basis for proactive operation in the long term.

In addition, monitoring is not just a "red-green light": we also rely on service monitoring and SLA measurement, trend analysis, future planning, and integrations (e.g., CMDB, ticketing, log collectors).

Zabbix as a foundation: stable, scalable, flexible

Zabbix is the obvious choice when comprehensive infrastructure monitoring is required:

• template-based operation (quick implementation),
• advanced visualization and dashboarding,
• scalable architecture (front-end/back-end structure, proxy function),
• manufacturer and platform independence,
• agent and agentless solutions,
• open-source code, and LTS version available for large enterprise environments.

However, classic monitoring tasks remain unchanged: system status monitoring, early detection of anomalies, alarm management and prioritization, incident management support, performance and capacity monitoring.

The challenge begins when, due to the number and complexity of alerts, an increasing amount of the team's time is spent "sorting through the noise."

AI support to speed up alarm handling

Two typical requirements arise during operation:

1. Quick interpretation of individual alerts and support for next steps
2. Highlighting truly critical alerts from among many alerts to maintain focus

1) Alert Assist: "What does this alert mean, and what should I do now?"

Alert Assist is a frontend module integrated into the Zabbix interface that uses a language model to interpret the selected problem and provides the operator with brief, easy-to-read support.

The structure of the response follows operational logic explicitly:

• Possible causes: for example, service outage, network access/problem, firewall filtering, misconfiguration, or overload.
• Verification steps with commands: recommended commands/checks that can be used to quickly validate the suspicion.
• Recommended sequence of actions: what to check first, what can be ruled out immediately.
• Prevention recommendations: how to reduce the chance of the error recurring.

The goal is not to provide the team with a "long essay," but to make the decisions necessary for daily operational work faster—this is especially useful for training, junior colleagues, or when knowledge is scattered.

2) Priority Manager: when you need to focus on 5 out of 10–20 pages of alerts

In larger, productive environments, it is typical for monitoring to be "full" of events. In such cases, the most difficult question is what requires immediate intervention.

Priority Manager is an interface-integrated component that examines all active problems as a set and then ranks the most critical ones based on a linguistic model (and optionally internal knowledge).

In practice, this means that:

• a Top N list is created (configurable: 5/10/15, etc.),
• the type of problem and the affected environment are visible for each item,
• recommended actions can be requested separately for the problems on the list, formulated in plain language,
• it is possible to filter by response length and review the history of alerts,
• and problems of the same type can also be grouped together (rather than viewing the situation on a "per server" basis).

The added value here is that the team quickly finds the "real fire" and the noise coming from monitoring becomes more manageable.

Technical background: flexible model selection, controlled data exposure

An important consideration for both solutions is that organizations have different attitudes toward cloud AI. For this reason, the design supports two approaches:

• On-prem execution: with HPE GPU server, local models (e.g., ollama/localai).
• Cloud LLM: to multiple service provider APIs as needed (e.g., OpenAI, Google, Amazon, Microsoft).

Control is particularly important from a practical, operational point of view:

• sensitive infrastructure identifiers (e.g., server names, IP addresses) are not necessarily included in the model; the approach may use anonymization,
• the quality of the response is aided by a deliberately designed system prompt: it is short, operator-focused, and seeks to minimize erroneous "overspending" (hallucination).

The ranking page can also be set up as workflow-based automation (not based on scripts), so it can be easily supplemented with internal knowledge, rules, and organizational specifics.

Operational benefits: faster response, lower load, less "noise"

The most important tangible benefits of AI-supported monitoring:

• Faster alert handling: focus on critical issues, less time spent on sorting.
• Reducing the burden on operators: routine cycles and initial investigations are simplified.
• Strengthening the L1/helpdesk level: with the support of the AI assistant, more errors can be checked without higher-level escalation.
• Reducing the risk of "shadow AI": if colleagues are already using AI, it is worth establishing a controlled, organizational framework for what we give to the model and how.

Vision for the future: agentic operation and "dialogue" with monitoring

The next step is not only in the proposals, but also in the agentic direction: when the system not only explains, but also performs tasks in certain, regulated steps (for example, automating repetitive, low-risk tasks, or reducing noise based on historical interpretation of alarms).

At the same time, there is also a need to go beyond simply "clicking" on the interface: the concept of an MCP (Modell Context Protocol) server aims to enable monitoring in natural language, which can also shorten the learning curve, especially for devices that are often criticized for their complex usability.