Introducing Lev-3B: Turn Log Evidence into Structured Decisions
Artificial Intelligence Nick Ma , Uday ShahKey takeaways
- This exploratory research develops Lev-3B, an open 3-billion-parameter model, to investigate System 1 approaches to two log tasks: anomaly detection and MITRE ATT&CK technique identification. Lev scores described alternatives directly without generating answer text.
- Observability: we suggest trying Lev-3B off the shelf for log anomaly detection. It led the decision models and prompted frontier LLMs compared in the paper on anomaly F1 across four public log datasets. These benchmark results do not establish production outcomes; test it on your logs and share feedback.
- Security: we suggest evaluating a hybrid architecture for MITRE ATT&CK technique identification. Let Lev handle decisions that meet a locally validated confidence policy, and route uncertain cases to a larger language model. The combined workflow’s quality and savings have not yet been measured.
Observability and security teams need useful decisions from logs: is this session anomalous, and which attack technique best matches this activity? The right model strategy depends on the question—and on how much confidence the evidence supports.
We’re introducing Lev-3B, a compact, open research model adapted for log analysis. Give it log evidence, a question, and described alternatives, and it returns a probability for each alternative. That structured output gives developers a way to explore direct anomaly scoring for observability and confidence-based routing for security.
Two Use Cases, Two Model Strategies
Example 1: Observability — Try Direct Log Anomaly Scoring
A kernel failure among routine messages. A BGL log session example supplied by the research team contains RAS KERNEL FATAL rts: kernel terminated for reason 1001, alongside INFO messages about signal handling and interrupts. The question is straightforward: “Is this log session anomalous?”
A score the workflow can use. In this research-team example, Lev returned {"false": 0.03, "true": 0.97}, ranking the anomalous option first. The application receives a structured score it can use to flag the logs for triage. This is one example, not a measure of performance on a customer’s logs.
Our suggestion: try the released model on your log anomaly task. Lev can supply this classification directly, without a frontier LLM stage. Start off the shelf, then evaluate false alarms, missed anomalies, and probability reliability on your own sources. The benchmark evidence is encouraging. We welcome real-world datasets and feedback to guide further evaluation and improvement.
Example 2: Security — Evaluate Confidence-Based Routing
Firewall changes in the logs. A security example supplied by the research team contains ESXi shell commands including esxcli network firewall set --enabled false. An analyst may want to identify which ATT&CK technique best describes the activity before deciding what to investigate next. Technique identification alone does not establish whether the activity is malicious.
Keep the full decision in view. The paper’s security evaluation offers 130 MITRE ATT&CK parent techniques for every question. Lev returns a probability distribution over those alternatives. Those probabilities are conditional on the candidate set; a high score is not a probability that the event is malicious. No model output is supplied for this firewall example.
Our suggestion: evaluate a hybrid architecture. Use Lev for the initial technique assessment. When a decision meets a confidence policy validated on your data, the application can use it directly under its review policy. Route uncertain cases to a larger language model with the original logs, full candidate set, and Lev scores for a second assessment. Calibrate that policy locally and check high-confidence errors as well as uncertain cases: confidence can be unreliable on unfamiliar data. The paper evaluates the models separately, so this architecture remains a proposal to test.
Evaluation: Compare Decision Models and Frontier LLMs
The paper compares Lev-3B with three decision models—Jev, Microsoft Decision-1, and OpenAI Decisions—and two frontier LLMs—GPT-5.6 Luna and Sol. The charts also include IBM Granite base and instruction-tuned baselines. “Zero-shot” means no examples are supplied in the prompt; “five-shot” means five examples are supplied. The labels show which setting each result uses.
Observability: Leading Anomaly F1, With a Transfer Caveat
For observability, try Lev-3B off the shelf for log anomaly detection. Validate its decisions on your own log sources before connecting them to operational actions.
The goal is to flag unusual log sessions while limiting missed anomalies and false alarms. The public datasets test two situations: separate, held-out records from three sources used in training—BGL, Spirit, and Thunderbird—and records from HDFS, a distributed-storage source not used to train or tune Lev. F1 combines how many anomalies are caught with how often flagged sessions are truly anomalous; higher is better.
Figure 1. Log-session anomaly classification, from paper Table 2. Anomaly-class F1 (%); higher is better. BGL, Spirit, and Thunderbird use held-out test sessions from known sources; HDFS tests an unseen source. Asterisks mark results evaluated on all anomalous examples and a 10% subsample of benign examples, reweighted to represent the full test population. See the paper for complete results and evaluation details.
Start with HDFS, the unseen source in the chart’s last column. Lev reaches 43.32% anomaly F1, compared with 25.91% for OpenAI Decisions zero-shot, the strongest external comparator. Lev leads on this new source, although its absolute score is much lower than on the known sources.
Lev also leads on the held-out test records from known sources: 99.36% F1 on BGL, 88.33% on Spirit, and 88.20% on Thunderbird. These results support trying the model, while the weaker HDFS score—and its high-confidence misses—show why customer logs need their own validation. Read more in the paper for the full evaluation and limitations.
Security: A Decision-Model Lead, With a Frontier-Model Tradeoff
For security, evaluate a hybrid: use Lev for decisions that meet a locally validated confidence policy, and send uncertain cases to a larger language model. Test the complete workflow on your logs before using it for operational decisions.
The security test asks which of 130 MITRE ATT&CK technique categories best describes the activity in a log. It measures technique identification, not whether the activity is malicious. A sourcetype identifies a log’s format. The test includes 134 held-out records from known formats used in training (“represented sourcetypes”) and 178 records from 18 unseen formats excluded from training and model tuning (“withheld sourcetypes”). Accuracy is the share of records assigned the correct technique.
Figure 2. MITRE ATT&CK parent-technique classification, from paper Table 3. All scores are percentages; macro-F1 averages an F1 score for each technique that appears in that test set. Each question offers 130 techniques. See the paper for complete results and evaluation details.
Lev leads the three compared decision models on accuracy and macro-F1 for both known and unseen log formats. On unseen formats, its accuracy is 42.13%, compared with 41.6% for OpenAI Decisions five-shot—a small lead across 178 records. On known formats, the corresponding scores are 35.07% and 32.8%.
GPT-5.6 Sol five-shot is more accurate on this task, reaching 45.5% on known formats and 56.2% on unseen formats. A separate research-team analysis also finds lower macro recall for Lev—the share of examples of each technique it correctly identifies, averaged across techniques. Lev scores 34.55% / 30.29% on known / unseen formats, versus 42.03% / 47.53% for GPT-5.6 Sol zero-shot. That gap is a reason to validate the full hybrid workflow, including high-confidence errors. The models were tested separately; the hybrid’s combined quality, latency, and savings remain unmeasured. Read more in the paper for evaluation details.
How To Implement the Two Workflows
Follow the model-card setup for Linux x86-64, Python 3.12, and a supported NVIDIA GPU environment. From the downloaded checkpoint directory, load the full Lev runtime and define a single-question helper:
import json
import torch
from kev.api import SystemOneRequest, to_record
from kev.evaluate import load
tokenizer, model = load(".", dev="cuda", dtype=torch.bfloat16)
model.head.temperature = 1.0
def score(state, name, question):
request = {"state": state, "questions": {name: question}}
record, _ = to_record(SystemOneRequest.model_validate(request))
encoded = model.encode(tokenizer, record, strict=True)
probabilities = model.probs(encoded)[0].tolist()
Observability: Get Anomaly Probabilities Directly
Use two shortened messages from the BGL example to illustrate the interface:
anomaly_scores = score(json.dumps([
"RAS KERNEL FATAL rts: kernel terminated for reason 1001",
"RAS KERNEL INFO 10696 total interrupts.",
]), "is_anomalous", {
"type": "noul",
"instructions": "Is this log session anomalous?",
"criteria": {"false": "Normal operation", "true": "Anomalous operation"},
})
print(anomaly_scores)
The result has the shape {"false": p_normal, "true": p_anomalous}. This snippet has not been run; the earlier 0.03 / 0.97 scores belong to the research team’s full example.
Security: Score the Full Catalog and Prepare a Route
Use the research-team ESXi example as the evidence. The function below accepts your complete technique catalog and a threshold selected through local validation. To match the paper’s task, supply its fixed 130 parent-technique alternatives and prepared inputs. No benchmark catalog or threshold is inferred from these two log lines.
security_logs = json.dumps([
"Jun 27 13:39:27 192.168.8.233 2025-06-27T13:38:15.354Z "
"localhost.localdomain shell[1514467]: [root]: "
"esxcli network firewall set --enabled false",
"Jun 27 13:39:22 192.168.8.233 2025-06-27T13:38:10.037Z "
"localhost.localdomain shell[1514467]: [root]: "
"esxcli network firewall set --enabled f",
])
def prepare_security_route(original_logs, techniques, threshold):
# techniques maps every candidate ID to its description.
technique_scores = score(original_logs, "technique", {
"type": "choice",
"instructions": "Which offered technique best matches this behavior?",
"criteria": techniques,
})
top_technique, confidence = max(
technique_scores.items(), key=lambda item: item[1]
)
review_input = {
"original_logs": original_logs,
"candidates": techniques,
"lev_probabilities": technique_scores,
}
return {
"route": "lev" if confidence >= threshold else "frontier_review",
"top_technique": top_technique,
"review_input": review_input,
}
This sketch prepares a route; application code must implement the LLM call and the review policy. Choose and validate a confidence policy for your sources before accepting Lev decisions directly, and retain analyst review where needed. The function has not been run, and no technique prediction or probability is claimed for this example.
Limitations of This Study
This exploratory study evaluates two tasks: log-session anomaly detection and MITRE ATT&CK parent-technique identification. The results do not establish performance across other log-analysis tasks or production environments. Confidence-based hybrids are a proposed research direction for both tasks; their combined quality, latency, and savings have not been measured.
Data coverage is limited to a small set of public log datasets and records derived from Splunk’s attack-data repository. Tests on held-out records and unseen sources help examine generalization, but they do not cover the diversity of customer log formats, systems, or operating conditions. Broader evaluation is needed to understand where Lev is useful and where it fails. High confidence does not guarantee correctness on unfamiliar logs.
Start With the Model Strategy Your Task Needs
Explore Lev-3B on Hugging Face and the paper. For observability, try the released model directly on a log anomaly task and assess the results on your sources. For security, evaluate a confidence-based hybrid with a larger language model. Share real-world evaluation feedback and datasets with us so we can better understand where the model is useful and where it needs improvement.
The Lev adapter and pointer-head weights are Apache 2.0 licensed; the base model and data sources retain their own terms. The pinned base is obtained separately, and the adapters alone do not restore the full decision interface.
Related Articles

Detecting DNS Exfiltration with Splunk: Hunting Your DNS Dragons

Staff Picks for Splunk Security Reading December 2023
