A bit of background on this feature might be needed - and a lot of credit has to be given to Olaf Hartong and FalconForce for this. Through a (now 6) part blog series on MDE internals they outlined some limitations in the MDE agent. I suggest starting at post 0x01 over at the FalconForce medium account, but if you want to save some time I suggest 0x03 and the new 0x06 to learn about the limitations and the new custom data collection rule feature.
To summarize the issue, we can quote the 0x03 article directly:
As mentioned before, a lot of the events have a local cap of 1 per 24 hours that get forwarded to the cloud, based on a set of distinct field values.
In general when looking at configuration for events, like Microsoft-Windows-TCPIP provider for the Connect Complete event in the example from the 0x03 article we can see that it generates an event for each connection where all the fieldNames in the event are unique as a set. If one value changes and the new set of values are unique, a new event is created. Again, according to the article:
All distinct sets expire in 24 hours and there is a total limit of 4000 distinct events in this timeframe.
So as we can see, there’s both a hard cap of 4000 distinct events per 24 hours and a certain level of event aggregation going on. It’s clarified in article 0x03 that this is for the EDR telemetry component, and that there are other components such as the AV that might augment the logs with additional events given certain sitations, such as a process being flagged.
Custom Data Collection
So how does the custom data collection play into this? The short answer is simple, it allows us to define custom logging without the cap (the cap for the custom logging is currently set at 25000 per rule per device). Most of us won’t be likely to ever hit that cap, however.
Currently, the following tables are supported:
DeviceProcessEvents
DeviceNetworkEvents
DeviceFileEvents
DeviceImageloadEvents
DeviceScriptEvents(1)
(1) - No AMSI-events in Advanced Hunting
The tables will change names (to separate them from the native MDE tables) by adding the word Custom:
DeviceCustomProcessEvents
DeviceCustomNetworkEvents
DeviceCustomFileEvents
DeviceCustomImageloadEvents
DeviceCustomScriptEvents
The potential for increased monitoring of important workloads or even as a part of a response action during investigation could prove to be very beneficial.
Setting it up
Setting it up is pretty straight forward. First, we need to make sure we cover the prereqs:
- Defender P2 license for the workloads you want to capture data from
- A connected Microsoft Sentinel workspace (unified experience)
- Dynamic tags configured for device targetting
With that out of the way, head over to the Microsoft Defender XDR Settings\Endpoints\Custom Data Collection and you should find the Custom Data Collection blade:
Clicking create rule takes us into the standard Defender XDR wizard experience, allowing us to set a name, description and state (enabled/not enabled):
Moving on we get to create the rule for targetting events. We can pick one of the supported tables and a corresponding action. For instance, the DeviceProcessEvents allows us to pick only the action ProcessCreated. Once that is selected, we can set rule trigger condition filtering. To make this example easy, we’ve borrowed from TelemtryCollectionManager (see next paragraph for more information) and the example lsass_starting_lsass:
After that we are presented with the scoping where we can either pick all or a specific device tag as mentioned in the prereqs:
After that we are presented with a summary and can click finish to deploy, which should result in a success like below:
And we are presented with the rule being present in the custom data collection blade:
That’s really all there is to it. My prediction is that we will see a lot of contributions to the TelemetryCollectionManager-project up until we get a working API, so watch that space for ideas. If you have an idea or an inkling of what extended telemetry you want to collect I also suggest contributing there directly.
Limitations
As mentioned above, there are some limitations like the 25000 events per device cap per 24 hours. All of the limitations (according to Microsoft Learn) are as follows:
- Each collection rule can capture up to 25,000 events per device within a 24-hour rolling window. Once the device reaches the limit, telemetry for the specific rule on the specific device stops until the window resets.
- If the device reaches the threshold early in the cycle, it can take up to 24 hours for telemetry to resume. For example, if the device reaches the limit one hour after the window resets, telemetry resumes after 23 hours.
- If the device reaches the threshold near the end of the window, the delay is shorter. For example, if the device reaches the limit two hours before the window resets, telemetry resumes after two hours.
- Rule deployment typically takes 20 minutes to one hour.
- Custom collection operates alongside default Defender for Endpoint configuration without interference.
Another big thing here is that will generate Microsoft Sentinel ingestion costs. No cost aside from that as the functionality is included in DFE P2, but the raw log cost is going to be on the Microsoft Sentinel end.
API? Nope!
Currently there’s no API, but FalconForce has you covered again (not sponsored, by the way) - they created a tool called TelemetryCollectionManager that is:
CLI tool to convert YAML custom collection rule definitions into Defender for Endpoint (MDE) rule JSON, validated against the bundled schema model/mde-model.json. An included JSON Schema (rule-schema.json) enables VS Code IntelliSense.
Until an API is in place it looks like this is going to be the way to do it - or you do it old school and just click.
Summary
Not much to say - this is a great addition to Defender for Endpoint. Big shoutout again to FalconForce for their research and contributions in this space.