Project archive

ScriptSmith is a unified, reference-free and execution-free LLM framework for generating, assessing and refining Bash scripts for IT operations, built on CodeSift. The last stage of an incident-remediation pipeline (root-cause diagnosis, action recommendation, action automation) turns recommended actions into scripts an SRE can run, and Bash is the tool of choice. LLMs can draft those scripts, but validating Bash by execution is often impossible: a test environment may not exist, and values such as free memory depend on the live system, so unit-test-based generate-and-fix approaches do not transfer. ScriptSmith instead assesses and refines scripts with LLMs alone, cutting the manual writing and debugging effort of SRE teams.

For each recommended action step the framework first searches a knowledge catalog of SRE-verified scripts (descriptions embedded as vectors, approximate nearest-neighbour search); a high-confidence hit is shown directly. Otherwise a code LLM generates a script, CodeSift assesses it without execution, and if it fails the LLM is asked to explain why and the explanation is fed back as a prompt to regenerate the script; the SRE approves, edits or rejects the result and approved scripts join the catalog. On the CodeSift Bash dataset (100 tasks, scored by execution) and InterCode Bash (153 tasks, scored by domain experts), the framework improves script generation by 7–10% overall. A peer-review setup, where a smaller model (Llama3 8B) judges a larger model's (Llama3 70B) scripts, beats self-reflection (assessment accuracy 74% vs 69% and 61% vs 54%), the largest refinement gain is 17% when a 70B model refines what an 8B model generated and assessed, and open-source models with ScriptSmith match raw closed-source (Gemini 1.5) generation. In a user study with four domain experts on the 153 InterCode tasks, first-pass accuracy of 42% rose to 76% after feedback-driven refinement, and experts judged the model's failure explanations correct in 69% of the failing cases. ScriptSmith is being deployed in IBM Instana's intelligent remediation pipeline as a tech preview.

CodeSift is an IBM Research framework that uses an LLM as a first-line filter for the functional correctness of generated code, without execution, reference code or human feedback. LLMs have made code generation cheap (SREs increasingly let them draft incident-remediation scripts), but checking that the code actually does what the task asked is still slow and error-prone: match-based metrics such as BLEU and CodeBLEU need a reference solution, execution-based checks need test cases and a secure sandbox, and neither scales to the volume of code now being generated. CodeSift validates the task-code pair directly, so only code that passes goes on to expert review or a controlled environment.

Generated code first passes a syntax check (ShellCheck for Bash, PyLint for Python; on an error the LLM is re-prompted with the error message and usually fixes it). The same LLM then runs three prompts: it rewrites the code as a text description of its purpose and logic (the code-func), judges whether that functionality accomplishes the task (similarity analysis) and lists any discrepancies between the two (difference analysis); the ensemble marks code correct only if it is similar and shows no discrepancies. The key insight is that LLMs compare text to text far better than text to code. On HumanEval, MBPP+ and a new 100-task Bash benchmark with test cases (contributed with the paper), judging code from Starcoder, Codellama, GPT-3.5 and Mistral, CodeSift with Mistral or Mixtral evaluators outperforms the ICE-Score and reference-guided grading baselines in accuracy, catches issues unit tests miss (e.g. "do not use recursion"), and can rank code-generation models without any ground truth. Deployed in an internal script-catalog pipeline, three subject-matter experts agreed with its validation output 83% of the time (and with the extracted functionality 78%) over 105 instances. ScriptSmith (AAAI 2025) builds on it.

Resistance Strength, also called Health Check by GTS, is the best practice for hardening IT infrastructure. It is governed by standard tech specs such as CIS benchmarks and STIG documents.

Customers' running environments vary, which results in many observed deviations, so prioritizing those risks with the customer's environment in mind matters. We quantify environment-aware resistance strength risk to help customers handle the volume of non-compliant controls. The quantification scheme is integrated with IBM X-Force Red Vulnerability Management Services (VMS).

IBM Cloud Pak for Multi Cloud Management (CP4MCM) is a digital consumption and delivery platform with integration and orchestration layers that supports multiple technology stacks across a multi-vendor platform. Governance, Risk and Compliance (GRC) is a critical component of CP4MCM that lets customers manage compliance with pre-defined and custom policies.

We worked with the GRC team to enhance the component:

• End-to-end solution with Ansible Operators that extends the policy controller to Virtual Machines and Cloud Providers.
• Policy controllers for the CIS Red Hat Enterprise Linux (RHEL) 7, CIS AWS and CIS Azure benchmarks.
• Communication between the Ansible Operator and Ansible Tower, enabling customer-specified policies written in an inventory.
• Design and visualization of scan and risk data in the GRC Dashboard.

Infrastructure code supports embedded scripting languages such as Shell and PowerShell to manage infrastructure resources and conduct life-cycle operations. Risky patterns in embedded scripts have widespread negative impact across the whole infrastructure.

We propose an analysis framework that automatically extracts and composes the embedded scripts from infrastructure code, then detects their risky code patterns with correlated severity levels and impacts. SecureCode implements the framework for Ansible playbooks. Integrated with the DevOps pipeline in IBM Cloud and tested on 45 IBM Services community repositories, it identified 3419 true issues (1691 high severity) with 116 false positives in minutes.

Software hang bugs are notoriously difficult to debug and often cause serious service outages in cloud systems.

HangFix automatically fixes a hang bug that is triggered and detected in production. It localizes the hang function with stack trace analysis, classifies the bug by matching likely root-cause patterns, and generates code patches for the identified pattern. Evaluated on 42 real-world hang bugs in 10 commonly used cloud server applications, HangFix fixed 40 of them in seconds.

Cloud server systems such as Hadoop and Cassandra power many data-intensive applications, but they present data-corruption and performance problems that are hard to debug for lack of diagnosis information.

DScope statically detects data-corruption hang bugs. It analyzes I/O operations and loops, identifies loops whose exit conditions can be affected by I/O through returned data, error codes or exception handling, then prunes false positives with loop bound and stride analysis. On 9 common cloud server systems it detected 42 real hang bugs, 29 of them new, most of which existing tools miss.

Server applications in production cloud infrastructures are prone to performance problems such as hangs and slowdowns, and developers often have little to go on when diagnosing them.

Hytrace combines rule-based static analysis with runtime inference to localize performance bugs more accurately than purely static or purely dynamic approaches. It needs no source code and works on both compiled and interpreted programs such as C/C++ and Java. Experiments with real bugs from seven common server applications show significantly better diagnosis accuracy than existing techniques.

Timeouts are commonly used to handle unexpected failures in server systems, but improper use can make systems hang or degrade. Our study of 156 real-world timeout problems in 11 cloud server systems shows they are widespread.

TScope identifies timeout bugs using kernel-level system call tracing with machine-learning-based anomaly detection and feature extraction; it correctly classified 18 of 19 real server performance bugs with a 0.8% false positive rate. TFix is a drill-down protocol that narrows down the root-cause variable of a timeout bug and recommends a fix; on 13 real cloud timeout bugs it localized the misused timeout variables and suggested proper values.

On a blockchain network, transaction data are exposed to all participants. FabZK preserves privacy and confidentiality while keeping data immutable: it stores only encrypted transaction data (e.g., payment amount) on the shared ledger and anonymizes the relationship between payer and payee.

It achieves both privacy and auditability with verifiable Pedersen commitments and zero-knowledge proofs. FabZK extends open-source Hyperledger Fabric with APIs for client code and chaincode, and supports on-demand, automated auditing on encrypted data with reasonable performance.

Containers are increasingly popular for deploying cloud applications but are prone to security attacks. This study evaluates static and dynamic attack detection schemes on 28 real-world vulnerability exploits common in Docker images.

Static scanning detected only 3 of 28; dynamic anomaly detection caught 22; combining both reached 86% (24 of 28). Dynamic detection also gave more than 20 seconds of lead time for common attacks that try to gain a shell and execute arbitrary code.

Cloud infrastructures and distributed applications keep growing more complex, which calls for an easy-to-use deployment tool that stays elastic in dynamic environments: geographically distributed hosts, system anomalies and shifting workloads.

CCM provides automatic component composition and instantiation, plus an elastic auto-scaling mechanism that handles overload, resource contention and system anomalies.