Introducing the AlphaSOC Security Data Lake
The AlphaSOC security data lake indexes normalized OCSF events at write time across four datasets, searchable from Splunk, Cribl, or AI agents.
A security data lake is a central store for security telemetry, kept in a common schema and searchable over long periods, so analysts can investigate history without loading every log into a SIEM. AlphaSOC builds one into its detection pipeline, designed for analysts and AI agents.
It answers one of the most common questions in an investigation, and one of the most cumbersome: has this indicator ever appeared in my logs? The logs are usually spread across tools, kept in a SIEM for a short window, or parked in cold storage that only a data engineer can search.
This post covers what the lake holds, its four datasets, the three ways to query it, how it is optimized for agentic querying, and how it compares with a SIEM.
What the Security Data Lake Holds
The lake is the last stage of the AlphaSOC pipeline. Supported telemetry is mapped to the Open Cybersecurity Schema Framework (OCSF), enriched with threat intelligence and prevalence data, scored to produce findings, then indexed and stored. OCSF gives fields a shared meaning across sources: a hostname means the same thing in a cloud audit log, a DNS log, or an endpoint event. Each source populates only the fields it can supply, so coverage varies by event class.
Records AlphaSOC does not yet map to a specific OCSF class are stored as OCSF Base Events with the original log in raw_data, so unmapped telemetry still appears in results.
Four Layers, From Findings to Raw Events
The lake holds four datasets, each answering a different question. Findings holds detection outputs, Evidence and Activity hold selected observations, and Events holds the normalized input events.
| Dataset | Question it answers | What it holds |
|---|---|---|
| Findings | What did AlphaSOC detect? | Aggregated OCSF Detection Findings, each with a severity, its observables, a MITRE ATT&CK mapping where one applies, and samples of the events behind it |
| Evidence | Was this indicator ever seen? | One record per observation AlphaSOC selected for Wisdom: domains, IP addresses, URLs, TLS certificates, and file hashes, indexed for checking a specific indicator |
| Activity | What is the indicator associated with? | The same observations joined with the device, user, and log source behind them, where available |
| Events | What exactly happened? | The full normalized event with the raw log in raw_data, for questions that need the complete record, such as whether traffic to a destination was periodic |
Three Ways to Search the Lake
Each integration queries the lake in place through federated search, without copying data into another platform.
| Integration | How it works |
|---|---|
| AlphaSOC for Splunk | A Splunk search command queries the Events, Evidence, and Activity datasets, and results continue through your SPL pipeline |
| AlphaSOC Data Lake for Cribl Search | The lake appears as a federated dataset with the same commands, and results pipe into Cribl KQL operators |
| AlphaSOC MCP server | Findings, the lake, and Wisdom threat intelligence become tools an AI agent can call, scoped to your workspace roles |
Optimized for Agentic Querying
Agents investigate in many small, iterative questions rather than a few large queries, and the lake is built for that pattern. It indexes selected OCSF fields as events arrive, so there are no partitions to tune or reindexing jobs to schedule; a new data source still needs a parser and an OCSF mapping before its fields reach the index. Bloom filters sit in front of the index. They can rule a value out but never confirm one, so a check such as "has this IP been seen?" can skip whole time periods without opening them. The index then skips any block of data that cannot hold a match, so a search for one hostname across weeks of data reads only the periods that might contain it.
The four datasets give agents four depths to query, each with a different cost profile. Findings and Evidence are compact and indexed, so queries against them are light and fast. Activity adds the join with device and user identity, and Events carries full records and raw logs, so each step deeper takes more processing. An agent starts at the shallowest depth that could answer its question and moves deeper only when a result gives it a reason to: a finding points to an indicator, the evidence shows it was seen, the activity names the host, and only then do the raw events show exactly what happened. When the answer is no, such as an indicator that was never seen, the search stops at Evidence and the deeper datasets are never read. The animation below shows both paths.
Security Data Lake vs SIEM
A SIEM is built for alerting, correlation, and response over the data it ingests, and retention is where it falls short for investigations. Most SIEMs keep 7 to 30 days of data in hot, searchable storage. Older data moves to archive tiers that are slow to query or is dropped, so an investigation into activity from two months ago often starts with a restore request. A self-built data lake keeps more, typically 1 to 3 months hot, but someone has to build and maintain the pipeline that gathers, normalizes, and loads each source.
AlphaSOC keeps costs low by running that pipeline for you. It collects, normalizes, enriches, indexes, and stores your telemetry, so there is no separate data platform to build or maintain, and it keeps 18 months hot by default.
- SIEM
- High
- Data Lake
- Medium
- Low
- SIEM
- High
- Data Lake
- High
- Low
- SIEM
- 7-30 days
- Data Lake
- 1-3 months
- 18 months
- SIEM
- Minutes
- Data Lake
- Seconds to minutes
- Seconds
† The total cost of software licensing, setup, data storage, and processing.
‡ The ongoing cost to gather, normalize, and load events from different sources.
| SIEM | Data Lake |  | |
|---|---|---|---|
| Overall platform cost† | High | Medium | Low |
| Maintenance overhead‡ | High | High | Low |
| Default hot retention | 7-30 days | 1-3 months | 18 months |
| Average query speed | Minutes | Seconds to minutes | Seconds |
† The total cost of software licensing, setup, data storage, and processing.
‡ The ongoing cost to gather, normalize, and load events from different sources.
The lake complements your SIEM rather than replacing it. AlphaSOC runs detection outside the SIEM and routes findings to supported SIEM and response destinations, while the telemetry behind them stays in the lake, searchable without ingesting it into the SIEM. Detection engineers can also check whether an indicator has been seen before writing a rule for it.
Learn More
From AlphaSOC
- Index & Store
- A Guide to Advanced Threat Detection Tools
- AlphaSOC for Splunk documentation
- AlphaSOC Data Lake for Cribl Search documentation
- AlphaSOC MCP server introduction