Overview of data processing functions

Updated at：2025-11-03

Description

Data processing functions can be freely combined to complete scenarios such as log cleaning, structuring, filtering, distribution, and desensitization.

Function overview

Field value extraction functions

Extract fields/field values from log text.

Function name	Function description	Function syntax description	Return value type
e_regex	Extract field values based on regular expressions	e_regex("source field name", regex="regular expression", fields_info="mapping list of fields and types", mode="overwrite")	Return the extracted log (LOG)
e_json	Extract field values in JSON string format	ext_json("source field name", depth=100, prefix="", suffix="", format="simple", sep="", mode="overwrite")	Return the extracted log (LOG)
e_sep	Extract field value content based on separators	e_sep("source field name", "mapping list of fields and types", sep="", quote="parts not involved in splitting", restrict=False, mode="overwrite")	Return the extracted log (LOG)
e_csv	Extract field value content based on separators, with the default separator being a half-width comma	e_csv("source field name", "mapping list of fields and types", sep=",", quote="parts not involved in splitting", restrict=False, mode="overwrite")	Return the extracted log (LOG)
e_psv	Extract field value content based on separators, with the default separator being a vertical bar	e_psv("source field name", "mapping list of fields and types", sep="	", quote="parts not involved in splitting", restrict=False, mode="overwrite")
e_tsv	Extract field value content based on separators, with the default separator being a tab character	e_tsv("source field name", "mapping list of fields and types", sep="\t", quote="parts not involved in splitting", restrict=False, mode="overwrite")	Return the extracted log (LOG)
e_kv	Extract field values based on two-level separators	e_kv("source field name", "regular expression", "key position", "value position", fields_info="mapping list of fields and types", mode="overwrite")	Return the extracted log (LOG)

Mapping enrichment functions

Add new fields according to rules based on existing fields.

Function name	Function description	Function syntax description	Return value type
e_dict_map	Use the Dict structure to match field values in logs. When the value of the specified field is the same as the Key in the Dict, assign the Value corresponding to this Key to another field in the log.	e_dict_map("JSON dictionary", "source field name", "target field", caseInsensitive=true, missing="", mode="overwrite")	Return the extracted log (LOG)

Process control function class

Used for conditional judgment.

Function name	Function description	Function syntax description	Return value type
e_compose	A composite operation function, similar to the ability to combine branch code blocks. It can combine multiple operation functions and execute them in sequence, and can be used with branch and output functions	e_compose("function1", "function2", ...)	Return the extracted log (LOG)
e_if	Process logs that meet the condition using the corresponding function; no processing is performed on logs that do not meet the condition	e_if("condition", function)	Return the extracted log (LOG)
e_if_else	Perform different function processing based on conditional judgment	e_if_else("condition", function1, function2)	Return the extracted log (LOG)
e_switch	Perform different function processing based on multi-branch conditions; if there is data that does not meet all conditions, it will be discarded	t_switch("condition1", function1, "condition2", function2, ...)	Return the extracted log (LOG)

Event operation functions

Used for log distribution, discarding, and splitting

Function name	Function description	Function syntax description	Return value type
e_drop	Discard logs that meet the conditions	e_drop(condition="condition")	Return the extracted log (LOG)
e_keep	Retain logs that meet the conditions	e_keep(condition="condition")	Return the extracted log (LOG)

Field operation functions

Used for adding, deleting, modifying, querying, and renaming fields.

Function name	Function description	Function syntax description	Return value type
v	Get the field value and return the corresponding string	v(field name)	Return the value of the field
e_set	Used to set field values or add new fields	e_set(field name 1, field value 1, field name 2, field value 2, ..., mode="overwrite")	Return the extracted log (LOG)
e_drop_fields	Match by field name and delete the matched fields	fields_drop(field name 1, field name 2, ...)	Return the extracted log (LOG)
e_rename	Rename fields	e_rename(field name 1, new field name 1, field name 2, new field name 2, ...)	Return the extracted log (LOG)

Description of field extraction modes

The following table describes the different values and explanations of the mode parameter for field extraction modes. Original log: { "a": "", "b": 100 }

Parameter value	Description	Processing statements	Processing results
fill	Set the target field when the target field does not exist or its value is empty.	e_set("a", "123",mode="fill")	{"a":"123","b":"100"}
fill-auto	Set the target field when the new value is not empty and the target field does not exist or its value is empty.	e_set("a", "123",mode="fill-auto")	{"a":"123","b":"100"}
add	Set the target field when the target field does not exist.	e_set("c", 200,mode="add")	{"a":"","b":"100","c":"200"}
add-auto	Set the target field when the new value is not empty and the target field does not exist.	e_set("c", "",mode="add-auto")	{"a":"","b":"100"}
overwrite	Always set the target field.	e_set("a", "123",mode="overwrite")	{"a":"123","b":"100"}
overwrite-auto	Set the target field when the new value is not empty.	e_set("b", "123",mode="overwrite-auto")	{"a":"","b":"123"}

Real-Time Consumption

Process control function

BLS

BLS