Next Gen SIEM: Five Essential Features

Many enterprise security teams have rebuilt their detection architecture twice to address the same bottleneck. The first generation consolidated its storage, search, and detection rules to a single data platform, licensed based on ingest volume. The second separated storage from the SIEM and used pipeline tools to route raw telemetry to a data lake. The drawback with both approaches was that detection logic was executed through saved searches against data at rest. Costs ballooned with the rule count and data volume.

A new third generation moves enrichment and detection into the security data pipeline itself. In this architecture, a dedicated engine evaluates each event once, on-stream, and the data lake stores the results. That means that your SIEM, SOAR, and AI tools consume context-rich findings versus raw logs, as shown below.

Within a next-generation SIEM, enrichment and detection are decoupled from storage to reduce cost and increase coverage. This article discusses the importance of a dedicated engine running at the data layer, beneath the SIEM and other workflows.

Summary of Essential Next-gen SIEM Features

The following sections examine five key properties of a next-generation SIEM.

Enrichment Before Detection

Where enrichment happens

When enrichment occurs after an alert, an analyst opens a finding, say one indicating that a host contacted a suspicious domain, then looks up the domain's age, its reputation, how many other hosts have contacted it, and what else that endpoint did. A SOAR playbook can run the lookups instead, but the work happens after the alert either way.

When enrichment happens before the alert, the detection logic scores an event that already carries the right context. Consider a single DNS query: with pre-detection enrichment, it reveals whether the destination domain was registered recently, whether it resolves to an ASN unique within your network, whether it is targeted by only one host, and whether it exhibits repetitive query patterns over time.

These four contextual signals work together as follows:

• Recently registered domain: Derived from event-level lookups when the DNS destination is evaluated.
• Unique ASN destination: Derived from event-level lookups against the network's ASN baseline.
• Single-host targeting (local prevalence): Derived from the engine's aggregated state, counting unique endpoints reaching the domain.
• Repetitive query patterns (beaconing interval): Derived from the engine's aggregated state, measuring query intervals over time.

Signals and how they combine

Each signal provides context that raises or lowers the likelihood of malicious activity. Individually, a single signal is weak because benign exceptions exist: new domains are registered daily, and rare ASNs host legitimate services. Combined, these four signals determine whether an alert fires and at what severity level, while suppressing false positives that raw events alone would trigger.

A Sigma rule written against enriched fields demonstrates this approach, matching only when all four signals are present on the event:

title: DNS query to rare, newly registered beaconing domain
status: experimental
logsource:
  category: dns
detection:
  selection:
    enrichments.domain_age_days|lte: 7
    enrichments.prevalence.global|lte: 1
    enrichments.prevalence.local|lte: 1
    enrichments.beacon_interval_sec|gte: 60
  condition: selection
level: high

Without upstream enrichment, none of these fields exist when the rule is evaluated. The same detection would need four saved searches, a correlation rule to join them, and a lookup step for each field, all running after the raw DNS alert had already fired.

Prevalence

Prevalence is the enrichment that matters most for threats nobody has classified yet. It measures how rare an event is: globally rare, locally rare (within your environment), or unique to a single tenant. A destination, an API caller, a login source, or an administrative action stands out when nothing else in your environment, or in anyone else's, has seen it.

That said, event rarity is not an indication of malicious activity. It ranks what deserves attention, and it works when a threat feed has nothing to match because the domain, IP address, or file hash is too new to appear on one.

When evaluating a tool, check whether threat intelligence, reputation scoring, and prevalence are included or sold as separate subscriptions. Each add-on carries an annual fee and a key to rotate, and each one you skip is context your analysts need to recover by hand.

AlphaSOC runs its own threat intelligence platform and enriches events upstream of the SIEM. Its engine scores every event across six dimensions, as shown below.

Prevalence is measured both within your environment and across customer environments, which is how a domain seen by one tenant gets flagged.

On-stream Detection, Decoupled from Storage

Where a rule can run

There are three places a detection rule can run, and where it occurs sets the costs and how deep it can look:

• The centralized model moves all telemetry into one store and runs rules there. This is what most SIEMs assume, and the cost scales with the volume you want inspected.
• The federated model leaves data in Microsoft Sentinel, Splunk, CrowdStrike, Snowflake, or S3 and queries it where it sits. There is no movement cost, but detection depth is limited to what each source exposes and what a saved search can do.
• The on-stream model puts a dedicated engine in the pipeline, so normalization, enrichment, and detection happen as data flows through, before anything is stored.

The hidden cost of saved searches

The centralized and federated models share a scaling problem for the rules you write yourself. A source such as CrowdStrike runs its own detections in real time, but that only covers its telemetry and rules. Your custom rules, whether they run in a SIEM or against a lake, still execute as saved searches against data at rest.

Each search scans the stored events on its own schedule. If you add a rule, you must add a scan, and if you add a log source, then every scan gets longer. A team running thousands of rules over a lake runs thousands of queries every interval, most of them re-reading the same events the previous interval read.

On-stream evaluation

On-stream evaluation inverts the cost because each event passes through the engine once, and every rule evaluates it during that pass. Because each event is handled once, the engine can afford to do more with it: fingerprinting, reputation, prevalence, time series, feature classification, and threat intelligence all run on the same event before it is stored. A saved search cannot do this because it sees a stored event with no context attached and has a fixed compute budget per run.

New rules still need to reach old data. When a Sigma rule is written or a threat feed updates, the engine runs it against the events retained in the data lake, so the same rule works forward on the stream and backward on history.

AlphaSOC runs the on-stream engine as a managed service. Prebuilt parsers for AWS, Azure, CrowdStrike, SentinelOne, Okta, GitHub, and Slack help generate initial findings quickly, without a pipeline-building project. The engine writes findings, evidence, and events to the data lake as OCSF records and escalates findings to your SIEM, SOAR, and AI tools.

Open Standards as the Native Format

A next-generation SIEM stores data, expresses detections, and answers queries in formats you could take to another vendor. There are three standards that cover those three jobs: OCSF, Sigma, and KQL.

OCSF

The Open Cybersecurity Schema Framework is a Linux Foundation project that defines a common language for security data. Contributors include Amazon, Cisco, CrowdStrike, IBM, Microsoft, Okta, and Splunk. Vendors use it in three ways, and few do all three: normalizing incoming telemetry to OCSF classes, emitting results as OCSF Detection Finding objects, and storing telemetry, enrichment, and findings together in an OCSF data lake.

AlphaSOC does all three, which is why a finding it emits carries the same field names as the event that produced it and the evidence attached to it. In addition to OCSF, there are two other schemas to be aware of: ECS and OSSEM.

Sigma

If OCSF standardizes data formats, Sigma standardizes detection logic. It allows you to write a rule once in YAML, store it in Git, peer-review it, deploy it through CI/CD, and make it portable across platforms. The SigmaHQ repository has thousands of community rules that can be pulled and used directly or with minimal changes.

However, converting Sigma rules into SPL, KQL, or CQL before they run adds overhead, which is typically due to:

• Field mappings that don't survive the target backend
• Simplified logic because a converter doesn't support a modifier
• Source rules updated while converted copies drift
• Engineers debugging the translation instead of the detection

Native Sigma execution removes the translation layer to run rules from SigmaHQ in their original form:

title: Okta User Account Locked Out
status: test
logsource:
  product: okta
  service: okta
detection:
  selection:
    displaymessage: Max sign-in attempts exceeded
  condition: selection
level: medium

AlphaSOC reads such files directly, with no converter and no platform-specific copy to maintain. Custom rules get the same controls as managed ones, such as severity, suppression, and scoping to specific systems.

KQL

Detection rules and interactive searches perform distinct roles. While an automated rule runs continuously across all incoming events, an interactive query represents an analyst or AI agent investigating historical data stored within the lake.

To support interactive queries, data lakes require a specific query language. The Kusto Query Language (KQL) has an open-source parser and is widely known among security personnel familiar with Azure Data Explorer or Microsoft Sentinel. Used alongside Sigma, which provides portable, vendor-neutral detection logic, KQL lets you query historical data in the lake.

When evaluating a tool, ask yourself whether it supports these standards natively or via a compatibility layer. A tool that imports Sigma through a converter, exports findings in its own JSON with an OCSF option, or exposes search through a proprietary language with a KQL translator can drift in three places.

Storage Separated from Compute

The previous two sections assumed a data lake that holds everything the engine produces. Whether you can afford to keep all data and findings in the lake depends on how the platform prices storage.

Ingest-time indexing ties storage cost to collection volume. Every event indexed at arrival is an event you pay to keep, whether or not anyone queries it, so each new log source adds cost. Usually, the response to cut costs is to drop the highest-volume telemetry. Items like cloud audit logs, DNS, network flow, and EDR details are deleted first because they are the largest. But they are also where beaconing, tunneling, and credential misuse are logged, so each cut is an added detection gap justified by cost reduction.

A decoupled architecture puts object storage underneath and charges for compute when a query runs. This way, retention and detection don't compete. Security teams also get to keep data they can't yet justify, which is needed for retrospective work like threat hunting and investigation.

For these tasks to be effective, engineers need data. AlphaSOC stores four kinds of OCSF records so that a query can move from a finding to the underlying event:

• Findings, which are the AlphaSOC alerts generated in the environment.
• Evidence, which describes the indicators seen within the environment.
• Activity, providing details (e.g., source, user) associated with each evidence hit.
• Events, which return the raw underlying telemetry to power deep investigation.

A data lake that stores findings and evidence is good for triage, but a data lake that stores activity and raw events is good for hunting because the rule you write next month needs events that produced nothing this month.

AlphaSOC's data lake searches use KQL. An analyst investigating a finding takes the affected device from it and pulls everything that device did in the surrounding day, as follows:

Events
| where device_uid == "<device_uid from the finding>"
| where time between (ago(24h) .. now())
| project time, class_name, src_endpoint, dst_endpoint, enrichments

An Agent-accessible Data Layer

Everything discussed so far ends up in one place: a data lake of OCSF records, a set of Sigma rules, and a query language over both. But who gets to use this data layer?

A couple of approaches are in use today. One approach is to build the AI into the product: Anvilogic, Panther, and RunReveal ship their own agents for triage and investigation. The other approach is to build a standards-based data layer that AI tools can plug into to drive agentic threat hunting and investigations.

The first looks more integrated, but the trade-off is that agents have to keep pace with the model providers, and the customer is tied to whatever the vendor ships. The second approach bets that agents will be commoditized and that the durable value sits in the data underneath them.

That bet changes how the work gets done. For a growing share of tasks, there is less and less need for a console. For example, SOC analysts can now submit queries directly via Slack, detection engineers can execute hunting campaigns straight from terminal interfaces, and automated agents in environments like Claude Code can autonomously author and push new detection rules. So, overhead such as switching between apps or memorizing complex query syntax is increasingly disappearing.

For the agent-accessible data layer to work, the platform must allow an agent to do what an analyst does. It needs to retrieve findings with their evidence, read detection logic and its metadata (such as MITRE ATT&CK technique IDs, severity levels, and associated tags), write and deploy new rules, query the data lake mid-investigation, run a detection retrospectively over stored events, and hand a case to a human with the evidence intact. A read-only API blocks the agent at the first step: it can read a queue, but nothing else.

The Model Context Protocol (MCP) is what makes the connection between external AI agents (such as Claude Code or Slack bots) and the security data platform reusable. An agent that speaks MCP can access the data lake without a custom integration for each tool and can work with whichever agent the team prefers in the future. AlphaSOC exposes its data lake via MCP and the REST API, and every action available in the console is also available via the API.

Connecting an agent is one command, for example, in Claude Code:

claude mcp add --transport http alphasoc https://api.alphasoc.net/mcp

After the browser sign-in, the agent has these tools available:

Any client that speaks MCP over Streamable HTTP with OAuth connects the same way, with nothing to install. Rule deployment and the remaining console actions go through the REST API.

Conclusion

A few questions are worth asking when comparing next-generation SIEM platforms:

• Where does detection run: on-stream as raw events arrive, or as saved searches against data that's already stored?
• Does threat intelligence and enrichment ship with the tool, or is it available as separate subscriptions and API keys?
• What happens to the bill when the telemetry volume doubles?
• Which standards are native: OCSF for storage and findings, Sigma without a conversion step, and a documented query language for the data lake?
• Can AI agents reach the data layer directly through MCP?

Knowing the gaps in your current stack is a good starting point for identifying which questions are worth asking in your case. If analysts enrich by hand, if rules are ported between query languages, if log sources are dropped to hold the budget, the fix is not another dashboard; it is a better data layer beneath the tools your team already uses.

AlphaSOC is built as that layer, and you can test the claims in this article against your own telemetry. Get started for free.