Faster Data Lake Queries
Improvement
Data lake queries now skip the storage they can prove holds no match, so a filtered search reads less data and comes back sooner. This is on by default for every data lake query and it is deliberately conservative: a block of data is set aside only when it cannot contain a match, and anything uncertain is still read in full. Your results are the same as before.
What Is Faster
- Indicator searches over long windows: Looking for one hostname, address, or hash across weeks of data narrows to the periods that can hold it rather than reading the whole window.
- Filters combined with
or: A query matching any of several observable values or types narrows on all of them together instead of falling back to a full scan. - Repeated filters on one field: Filtering the same field more than once narrows the search rather than widening it, and a combination that cannot match is answered without reading anything at all.
- Detection level findings queries: Asking only for detection rows no longer decodes the sample events attached to each finding. Queries that do want the samples return exactly what they returned before.