From simple rules to building correlation rules engines and Domain Specific Languages for event data detection and processing Офлайн 2021

There are plenty of applications that process data based on rules; such as those for log processing, event detection, anomaly detection, and event processing. Data that is encountered in such applications is usually in the form of text or JSON. In this talk, we will focus on rules based data processing techniques for JSON data. Similar techniques can be used to process textual data if the format of the data is known and the data can be tokenized into fields on which rules can be applied.

The nature of JSON data allows it to get more and more complex by allowing it to define recursively nested objects, which can make it difficult to be processed based on rules.

The talk is about how python can be used to write simple rules to discover JSON data that obeys or disobeys those rules, progressing into creating rules to process recursively nested data, designing appropriate data formats to define the rules themselves for being evaluated on recursively nested objects, structuring the code to improve overall efficiency of object discovery using short-circuit evaluation methods, adding logical rule evaluation capability, and culminating in creating a Domain Specific Language (DSL) for defining rules that can be evaluated on any type of recursively nested JSON data. The DSL will make the application flexible to addition, revision, or deletion of rules on-the-fly.