Projects

ARI is InsightFinder's Predictive Reliability Agent: Predictive AI SRE for trustworthy production. Observability tells teams what changed; AI SRE helps them understand why. ARI uses InsightFinder's patented unsupervised AI engine to predict failures hours before impact, identify root causes and resolve issues across the stack, from AI agents and applications to IT infrastructure, turning reactive incident response into proactive prevention.

ARI builds a unified context from metrics, logs, traces, alerts, change events, tickets, service dependencies and operational history, surfaces fewer, higher-value signals with early-warning patterns, and separates likely root causes from downstream symptoms with causal chains. It answers natural-language questions about predicted and active incidents in Slack, Microsoft Teams or ARI Mobile, creates JIRA tickets, starts verification workflows and recommends or initiates policy-approved actions such as rollback. Every action is explainable, permissioned and auditable; engineers stay in the loop, and each validated incident feeds back into the agent.

AI Evaluation is InsightFinder's evaluation workspace: a controlled environment where data scientists and ML engineers create, configure, chat with, monitor and evaluate LLM prompts and responses, with built-in analytics and safety measures, so they can select the best model for their needs. An LLM session pairs model configuration with real-time evaluation and performance metrics, using built-in models or API-connected models from providers such as OpenAI and Anthropic. Prompts run as an interactive chat, where every response is evaluated as it arrives, or as a batch from an upload, manual entry or the saved Prompt Library. Typical uses: testing a new LLM version against the previous one, comparing different LLMs on the same prompts for output quality, token efficiency and edge cases, trust & safety analysis, and keeping an eye on token consumption and latency.

Each response is scored from 1.0 to 5.0 by LLM-as-a-judge evaluations for Hallucination & Relevance (logical inconsistency, answer irrelevance, hallucination, factual inaccuracy) and Safety, Guardrails & Toxicity (anomalous outliers, jailbreaks and prompt injection, topic guardrails, tone, toxicity, PII/PHI leakage), checked pass/fail for gender, racial and cultural bias, and judged against custom evaluations written as your own evaluation prompts, such as brand tone or required information. The Model Compare dashboard runs prompts across models and lines up hallucination & relevance, safety & toxicity, bias and total/input/output tokens; A/B testing turns versioned prompt templates, linked datasets and models into a test matrix on quality, tokens and latency, with a Winner badge, an evaluation summary, history and a "Pick Your Favorite" step that promotes a prompt version to production. The same AI reliability line adds multi-agent trace & observability, an LLM Gateway and domain-specific fine-tuning of small language models.

ITBench is an open-source framework from IBM Research for systematically benchmarking AI agents on real-world IT automation tasks, toward AI-driven IT automation that is correct, safe and fast. It covers three personas: Site Reliability Engineering (availability and resiliency, e.g. resolve "High error rate on service order-management": detect the alert, trace its source, propose a fix), Compliance & Security Operations (e.g. assess the compliance posture for a new control rule for RHEL 9, turning natural-language regulation into checks) and Financial Operations (cost efficiency and ROI, e.g. resolve "IT spend exceeded the budget"). The ICML version has 102 scenarios (SRE 42, CISO 50, FinOps 10) built from real-world major incidents, compliance controls and FinOps cases, and is easily extended by community contributions.

Scenario environments are Kubernetes-based: push-button deployment tooling stands them up and replicates the incident, compliance control or cost event; a scenario framework recreates IT scenarios; reference SRE and CISO agents (built on CrewAI, with configurable LLMs and natural-language tools to interact with the environment) serve as baselines; and a managed leaderboard lets researchers run agents on their own systems and submit outputs for evaluation with push-button workflows and interpretable metrics, rating whether the agent handled the issue and how quickly. Agents powered by state-of-the-art models resolve only 11.4% of SRE scenarios, 25.2% of CISO scenarios and 25.8% of FinOps scenarios (excluding anomaly detection; FinOps anomaly detection F1 0.35), leaving ample room for progress. ITBench is the work of a large IBM Research team; I am one of the co-authors.