TL;DR: Microsoft introduced near-real-time analytic rules in Microsoft Sentinel in 2021 and a few years later they were incorporated in Defender as well. During that time, Microsoft has constantly updated and increased the scope of the feature. At FalconForce, we are always looking to enhance our detection engineering practices. So, we wanted to get a better understanding of how NRT works, what happens under the hood and see what the requirements are for converting an existing detection to an NRT one. In this blog, we present the applied research that was done and our observations on NRT in practice. Hope you find this blog useful in setting up NRT in your environment!

Introduction

The main advantage of near-real-time rules is that they provide faster detection speed compared to Sentinel’s normal scheduled analytics rules and Defender’s custom detection rules. According to Microsoft, NRT rules in Sentinel are designed to query events based on their ingestion time, with much smaller delays compared to scheduled rules. A similar principal is used by NRT rules in Defender, with their goal being, as the name suggests, to achieve near-real-time detection. In this post, we will explore how the feature works in Sentinel and Defender in distinct sections, since there are significant differences between its implementation for the two products.

NRT in Sentinel

The theory

According to the . However, they differ to scheduled analytics rules in other aspects as well. Instead of having a default 5-minute built-in delay (to account for ingestion time lag) as in scheduled query rules, NRT detections have a 2-minute built-in delay. Furthermore, NRT rules have a fixed look-back period of 1 minute and are querying by default on the ingestion time, as mentioned above.

Based on this, each NRT detection is querying events ingested 2-3 minutes before the time of the query’s execution.

Graph indicating how NRT detections in Sentinel work in theory (2-minute built-in delay and 1-minute look-back period)

In the example presented in the picture above, consider a query that is executed at 11:30:00. Taking into account the 2-minute built-in delay and the 1-minute look-back period, the rule will query events that were ingested between 11:27:00 and 11:28:00. The query will be executed again after 1 minute, moving the look-back window 1 minute forward.

What’s going on in reality?

So, let’s verify whether the theory about NRT rules holds up in practice. For our main tests, we set up a scheduled task running every 1 minute, while monitoring the events generated by the task with an NRT detection that would trigger an alert for each one of the events. Specifically, the scheduled task was executing a simple PowerShell script which triggers the SACL on a file, generating events with ID 4663. The idea is to correlate the logs of the following three tables to see what is going on in the background:

• SecurityEvents: keep track of the events generated by the scheduled task.
• SecurityAlerts: correlate the scheduled activity with the alerts that are generated from it.
• SentinelHealth: correlate the execution of the detection with the data of the other two tables.

Creating a dataset for testing

Using the SecurityEvents, SecurityAlerts and SentinelHealth tables, we can create a dataset to help us understand the inner workings of NRT. The table below includes a sample of this dataset.

Sample dataset based on the SecurityEvents, SecurityAlerts and SentinelHealth tables.

The columns in the above table show:

• Time Generated: the time the monitored event was generated on the host (collected from the SecurityEvents and SecurityAlerts tables, fields TimeGenerated and StartTime respectively).
• Ingestion Time: the time when the event was ingested in Sentinel (collected from the SecurityEvents table using ingestion_time()).
• Alert Time: the time when the NRT rule triggered an alert for that event (collected from the SecurityAlerts table, field TimeGenerated).
• Look-back Start: the timestamp indicating the start (left-most edge in the graph) of the look-back period (collected from the SentinelHealth table, field QueryStartTimeUTC).
• Look-back End: the timestamp indicating the end (right-most edge in the graph) of the look-back period (collected from the SentinelHealth table, field QueryEndTimeUTC).
• Look-back Period: the duration of the look-back period (derived from the previous two columns).

Note: According to Microsoft’s documentation QueryStartTimeUTC refers to “The UTC time the query completed its run.” and QueryEndTimeUTC refers to “The UTC time the query began to run”. Apart from the obvious mix up where their meanings are reversed, the logs clearly indicate that these fields don’t refer to the start and end times of the query’s execution, but the start and end timestamps of the look-back period.

Analyzing the look-back period

Based on this table we can already see that the duration of the look-back period is not always exactly 1 minute. During our tests we generated more than 2500 events in a low-traffic environment, for which the average look-back duration was 1 minute and 1 second, with the maximum one being almost 2 minutes, and the minimum being 55 seconds. This is crucial, as it indicates that the 1-minute look-back period is not a strict rule. We know that detections are not always executed at the exact time as defined by their frequency, and this should be expected for NRT ones as well.

If the look-back period was a fixed value, gaps would appear where the ingested events would not be “picked-up” by any rule execution.

Therefore, our first observation focuses on the gaps that appear between executions of the same rule. Having a flexible look-back period to cover them is necessary to avoid false negatives and NRT detections automatically handle such situations. During our testing we never observed a gap between the look-back periods of two rule executions.

Dealing with larger ingestion delays

Table indicating how ingestion delays are handled. See how the alert in the second row is outside the look-back period of the relevant rule execution.

A few minutes forward and as we can see above, the event ingested at 10:18:35 has a larger ingestion delay than usual, meaning that it is outside the look-back period (10:17:10 – 10:18:11) of the relevant rule execution. This results in 0 alerts being generated for that period (last column). The next event (ingested at 10:19:12) also misses “its” look-back period (10:18:11 – 10:19:11). However, the previous event does trigger an alert this time, as it falls in the new look-back window. The last row in the table above indicates that for the rule execution that handles events ingested between 10:19:11 and 10:20:12, two alerts were generated. This is expected based on the ingestion timestamps of the last two events (see the graph below).

Our second observation clarifies how ingestion delays are handled. Based on the above, a future query execution will “pick up” alerts with larger than expected delays.

Graph indicating how ingestion delays are handled. Observe how both ingested events are inside the look-back period of the second rule execution, resulting in two alerts for that execution instance.

Built-in delays

Next, we will cover the built-in delay. Based on the tables above, we can already see that the 2-minute built-in delay is also flexible.

Specifically, during our testing, the average duration of the delay was 2 minutes and 2 seconds. Note that this is based on the alert generation timestamp, and not on the rule execution time, as SentinelHealth (or any other table to our knowledge) does not provide a timestamp for that.

The largest built-in delay (where built-in delay in this case is defined as the difference between the alert generation timestamp and the end of the look-back period) we observed was 4 minutes and 7 seconds. For that specific case, the duration between the ingestion of the event and the alert generation was 4 minutes and 55 seconds (also the largest one). According to the documentation, each rule execution should handle the events ingested 2-3 minutes before the execution, but as we can see that is not the case here.

This brings us to our third observation. Similarly to the look-back period and the execution frequency, the built-in delay is not a strict value, but instead adapts to the current situation to avoid false negatives.

Limited duplicate alerts

Finally, during our testing we observed a few duplicate alerts (less than 0.25% of the total in our low-traffic environment). For example, the following two alerts were matched to the same event. This is something that should be considered and possibly a de-duplication process should be used to prevent such alerts from triggering.

Based on the SystemAlertID values we can see that two different alerts were generated for the event ingested at 6:32:51.

NRT limitations in Sentinel

Besides the KQL syntax limitations that are explained in a later section in this blog, NRT rules in Sentinel have a few more limitations. We’ll discuss these here.

Join/union operators

The join and union operators are available for use on NRT queries for Sentinel. However, due to different ingestion delays between different log sources should be avoided. Consider the join operation of the following query:

let Executions = DeviceProcessEvents 
| where ActionType =~ "ProcessCreated" 
[...];
Executions 
| join kind=leftsemi (
    DeviceLogonEvents 
    | where ActionType =~ "LogonSuccess"
    [...]
) on DeviceId, LogonId

• We have an NRT detection that queries table DeviceProcessEvents and table DeviceLogonEvents for “ProcessCreated” events and successful login events, respectively, and performs a join on them. When both events are present an alert is triggered.
• What will happen if the ingestion time of one of the two events is outside the look-back period?
• Since the queried events do not include, for example, any “ProcessCreated” event , no alert will be triggered.

If no “ProcessCreated” event is ingested in the same look-back period as a “LogonSuccess” event, no alert will be triggered.

Note: there are detections where the join/union operator is used only to provide enrichment. The enrichment information may be missing however, depending on whether the event(s) used for the enrichment were ingested during the same look-back period as the event that triggered the alert. In such situations, false negatives will not be generated, but missing the enrichment will reduce the value of the detection while triaging.

Multi-event alerts

A similar situation to the one explained above may occur when multiple events (of the same type) are required to trigger an NRT detection. For example, if the threshold set in the NRT query is 5 events, but 4 occur during the look-back window and 1 just outside, no alert will be triggered. Therefore, NRT rules are ideal for single-event alerts.

Event grouping

Event grouping for NRT rules is the same as for normal scheduled query rules, but with a smaller limit. Instead of having a limit of 150 alerts, a single NRT detection can generate up to 30 alerts. The last of these alerts will be a summary for all those that would be generated after the 29^th.

Number of rules

Currently, no more than 50 rules per workspace can be enabled at the same time. This, along with other limitations of the NRT feature, can be subject to change as Microsoft introduces new updates.

Note: there is a way to bypass this limitation by using multiple workspaces in the same Resource Group. However, it may not be practical.

Log sources with ingestion delay

As NRT rules are designed to run near-real-time, log sources with significant ingestion delays may not work as expected. Specifically, according to Microsoft: “[…] NRT rules will only work properly on log sources with an ingestion delay of less than 12 hours.”

Conclusions on NRT in Sentinel

Based on the above, we can highlight the most important aspects of NRT rules for Sentinel:

• Generally, in our environment, NRT detections run every ~1 minute with a built-in delay of ~2 minutes and a look-back period of ~1 minute (+/- a few seconds). Importantly, the values of these parameters are not fixed; meaning that this flexibility prevents any blind spots from appearing and eliminates false positives.
• Duplicate alerts may occur, but their amount may vary depending on the environment’s traffic.
• The join and union operators should not be used when events from both queried tables are required to trigger an alert, in order to avoid false negatives.
• NRT rules are ideal for single-event alerts.
• Log sources with ingestion delays of more than 12 hours are not compatible with NRT rules.

NRT in Defender

Due to the different way that events are ingested in Defender (in batches instead of in a constant stream), NRT rules do not function the same as described above for Sentinel. However, in the case of Defender, NRT is a lot more straightforward: no look-back periods, built-in delays or anything like that!

NRT detections in Defender are executed when a new batch arrives, querying only the events of that batch (i.e., there is no specific look-back period). Defender will preselect the Continuous (NRT) option when creating a custom detection rule if the query is NRT-compatible. It is as simple as that!

Part of the UI in the Defender portal when deploying a new custom detection (if the rule is NRT-compatible).

To verify what Microsoft suggests in the documentation, we performed tests similar to . Specifically, a scheduled task was set to execute every one minute, creating and deleting a file with a specific filename. This activity was picked up and put into the DeviceFileEvents table in Defender, which in combination with the AlertInfo and AlertEvidence tables enabled us to see when an event was generated, when it was ingested and when the relevant alert was generated (or with which alert it was correlated). A query like the following can be used for that:

AlertInfo
| where Title == "NRT Testing"
| project AlertTime = Timestamp, AlertId, Title
| join kind=inner ( AlertEvidence
    | project AlertId, DeviceId, Timestamp
) on AlertId
| join kind=inner ( DeviceFileEvents
    | project DeviceId, Timestamp, EventIngestionTime = ingestion_time()
) on DeviceId, Timestamp
| extend IngestionDelay = EventIngestionTime - Timestamp
| project  EventGenerationTime = Timestamp, EventIngestionTime, AlertTime, AlertId, IngestionDelay
| order by EventIngestionTime desc

Note: Our setup for Defender was a bit different compared to Sentinel, as we don’t use Windows Events to track the activity (as we did with event 4663 during the Sentinel tests). Instead, we relied on the information from DeviceFileEvents table.

After executing the aforementioned query, we discovered that the process we followed for Sentinel will not return similar results here, mainly due to how Defender processes new, identical events.

Multiple events related to the same scheduled task activity are aggregated into the same alert, even if they are ingested after the alert’s creation.

As we can see in this table,, meaning that no new alerts are created – even for events that are ingested after the alert’s generation. This is the expected behavior for Defender and there is no option to create a new alert for each one of the events that is ingested. This means that there is no clear indication when a detection is executed and which events it queries.

To bypass this restriction, we had to manually change the status of the alerts as “Resolved” as soon as they were created; meaning that the next batch of events would generate a new one. Based on this, and with a grain of salt, we concluded that NRT rules are querying events, and creating alerts, as soon as the batch they are part of is ingested.

NRT limitations in Defender

As for restrictions, we can focus on the following:

• Join/union operators are not allowed. Unlike Sentinel, where theoretically they can be used, using these operators in Defender will lead to the aforementioned Continuous (NRT) option being unavailable.
• As explained before, NRT detections are ideal for single-event alerts. This holds true for the Defender version of the feature as well.
• Unlike Sentinel, Defender has no fixed limit on the NRT rules that can be created.

Conclusions on NRT in Defender

While the idea behind NRT rules for Sentinel and Defender is the same (to speed up the detection process), the way this is achieved is not the same. NRT detections in Defender are as near-real-time as possible, depending mainly on the ingestion delay and not when the rule is executed or the built-in delay.

Exploring how to create new NRT rules or convert existing rules to NRT

Now that we have a solid understanding of how NRT rules work in the background, we are going to explore syntax limitations that must be considered when creating such rules or trying to convert already existing ones. Components and limitations You might think “Aligning our custom KQL rules with the aforementioned NRT restrictions should enable conversion of the rules to NRT”.

It turns out that the KQL syntax you use can affect whether a rule is convertible to NRT, depending on the rule’s target platform. And the reasons for this are not always clear. For example, using the join operator in Defender will cause Microsoft to immediately mark your custom rule as incompatible with NRT, which, based on what we discussed earlier, makes perfect sense. However, using binary_all_and(), binary_all_or(), or binary_all_xor() will also cause Microsoft to disqualify your custom rule from NRT conversion. This does not seem logical, since similar functions as binary_and(), binary_or(), and binary_xor() are compatible with NRT. Similar examples exist for Sentinel as well.

What is documented?

When a rule fails to meet the requirements for NRT convertibility, Microsoft displays an error message:

Defender

Sentinel

Enlightening as this error may be, browsing Microsoft’s NRT documentation eventually led us to this page: Create custom detection rules. Here, it is stated which tables can be used for Continuous (NRT) frequency and refers us to Supported KQL features in Azure Monitor transformations for the KQL syntax allowed in NRT (even if the post’s title does not indicate so). While reading the documentation and playing around with NRT and the KQL syntax, a new question was born — one for which, deep down, we already knew the answer: “Can we solely rely on Microsoft’s documentation on NRT syntax?”

And there are two good reasons for that:

• Microsoft’s documentation can sometimes be outdated/inaccurate or lacking in sufficient detail. . Examples of such syntax are the functions not() and  strcat_array(), even though there is a reference to strcat(), which is quite similar to the latter. Microsoft itself states that “The syntax for this type of rule is gradually evolving”, which could mean more KQL syntax will be convertible in the future.
• The referenced links appear to apply to Defender, but it’s unclear whether they apply to Sentinel.

Making our own documentation

To create NRT rules, we needed to be certain about the KQL syntax that can and cannot be used. This led us to create our own documentation by following five simple steps:

1. Take a deep breath.
2. Browse through Microsoft’s KQL documentation.
3. Evaluate NRT convertibility in Defender.
4. Evaluate NRT convertibility in Sentinel.
5. Repeat.

After many deep breaths, we ended up with more than 500 KQL syntax elements evaluated and sorted these into four categories: tables, functions, operators, and statements. For every category, we created a JSON file and populated it with our findings. The JSON follows the format:

"m365": {
  "Accepted": [],
  "Prohibited": []
},
"Sentinel": {
  "Accepted": [],
  "Prohibited": []
}

You can find those JSON files at https://github.com/FalconForceTeam/NRT-KQL.

The results of our KQL syntax elements evaluation have been summarized in the tables below. Truth be told, Defender ended up being the troublemaker, as a large portion of the KQL components were marked as non–NRT convertible, while Sentinel accepted (most of) them without issue.

The number of KQL elements accepted and prohibited for NRT usage in Defender based on their category.

The number of KQL elements accepted and prohibited for NRT usage in Sentinel based on their category.

Note: while we tested most of the KQL syntax, a few uncommon elements were not verified with regards to their NRT compatibility.

Evaluating our FalconFriday rules

Having reached that point, we couldn’t resist using the aforementioned limitations to evaluate some of our beloved FalconFriday rules. Specifically, we used the KQLAnalyzer tool to break down each KQL rule into its tables, functions, operators, and statements, and then compared those components against the KQL syntax we documented. The results can be summarized into three categories:

1. Non-convertible

Rules that consist of one or more KQL syntax elements that cannot be converted, replaced or deleted. For example:

0xFF-0148-ASR_Bypass_Executable_Content-Win in m365
Functions: ['has_any_index', 'replace_string', 'FileProfile', 'coalesce']
Operators: ['mv-apply', 'invoke']
ingestion_time() function is used: ingestion time is used by default

2. Unknown

Rules that consist of one or more KQL syntax elements that are either custom or not yet documented by Microsoft. For example:

0xFF-0060-Azure_AD_Rare_UserAgent_App_Sign-in in sentinel
Non-registered function: ExtractBrowserTypeFromUA
Non-registered function: QueryUserAgents

3. Eligible

Rules that consist of one or more KQL syntax elements that cannot be converted, but can be replaced or removed. For example:

0xFF-0015-Masquerading-Renamed-executables-of-interest-Win in m365
ingestion_time() function is used: ingestion time is used by default
dynamic() cannot be converted in NRT

From eligible to convertible

We encountered several FalconFriday rules whose NRT convertibility was blocked, simply because of the ingestion_time() function and dynamic expressions (i.e, dynamic()), such as in 0xFF-0015-Masquerading-Renamed-executables-of-interest-Win. For the former, since NRT uses ingestion time by default, we can simply remove the function from the query. For the latter, during the evaluation of Microsoft documentation, we verified that pack_array, which “packs all input values into a dynamic array”, and bag_pack, which “creates a dynamic property bag object from a list of keys and values”, can both be converted to NRT. Based on that, we can replace dynamic() with either pack_array() or bag_pack(), depending on whether a list or a dictionary was required.

Detection query after being converted from eligible to compatible.

Overall conclusion and what’s next

By now it should be clear that there are several things to consider before you can use NRT detections. Even before testing their efficiency, your detection pool will be significantly narrowed due to the reduced supported KQL syntax. Even your current custom rules that comply with the allowed syntax may end up being inefficient, due to the limitations that come with the NRT feature, such as not being able to use join/union.

Yes, custom detection rules can be automatically converted in Defender according to Microsoft, but you should have a good understanding of the NRT feature and its limitations before doing so. We hope that our analysis can assist you throughout your NRT journey.

Special thanks

Extra special thanks go out to Agapios Tsolakis and Rogier Boon for their contributions to this research. 

Take your threat detection to the next level

Want to have access to our repository with over 600 advanced detections? Please have a look at our commercial offering and reach out via [email protected].

Knowledge center

Other articles

FalconForce realizes ambitions by working closely with its customers in a methodical manner, improving their security in the digital domain.

KVK 76682307
BTW NL860745314B01