Logging

The Snippet Framework includes logging as a standard feature that was designed to help troubleshoot issues easily. It is execution-aware which enables the Snippet Framework to determine when it is appropriate to log to the file system or the console. It also enables targeted debug through Log Policies which enables pin-pointing logging during standard collections.

Reading a log message

A Snippet Framework log message contains information that helps identify where it came from. This can be useful during debugging to more quickly troubleshoot an issue. A Snippet Framework log message may contain the following information:

Log Format Definitions

Log Message Variable	Description
time	Local time of the log message
unique_id	Unique identifier of the current Snippet Framework execution
module	Module that called the log statement
lineno	line number within the module that called the log statement
log_level	Level of log message (debug, info, etc)
message	Message from the log statement

Some messages will be context-aware to help track the lifecycle of the collection and allow for more targeted debug. A context-aware message contains [a:<app_id>,d:<did>,c:<obj_id>,s:<step_name>] as part of its message.

Context-Aware Definitions

Character	Named Value
a	Application ID
d	Device ID
c	Collection ID
s	Step Name

Assuming you had the following context-aware information, [a:1006,d:4,c:10231,s:http], you would be able to know the following information:

Header Example

Header Portion	Definition
a:1006	Dynamic Application: 1006
d:4	Device: 4
c:10231	Collection Object: 10231
s:http	Step: http

UIDebug

UIDebug provides instant feedback for debugging a collection or Dynamic Application. This is helpful when developing a Snippet Framework based Dynamic Application or trying to troubleshoot a problem. Running in UIDebug will inform the Snippet Framework to run in debug mode, regardless of Log Policies. UIDebug outputs all log messages with the format: <time> - <message>.

Example Output

2024-01-11 14:49:41,006 - Registering http to silo namespace [silo.low_code_steps.rest.network_request]
2024-01-11 14:49:41,170 - [a:1006,d:4,c:10231,s:http] Performing get @ 'https://10.2.24.122:443/api/api/account/2'. auth=True; capath=None; debug=False; proxies={}; verify=False

As we can see in this example, the first value in the log message is when it was logged. The second portion specifies the message. The first line is not a context-aware message as it does not contain the header specifying which collection is aligned. However the second message is a context-aware message as it contains the header that identifies where it came from.

File System Logging

File System logging helps debug an issue if you are unable to reproduce the issue with UIDebug or it intermittently occurs on the system. File System Logging outputs all log messages with the format <time>,<unique_id>,<module>:<lineno>,<log_level>,<message>.

Logfile Locations

The Snippet Framework utilizes several log files which allows for better organization of messages. The logfile locations and their contents are as follows:

Log filepaths

File path	Usage
/var/log/em7/sl1.log	Logs from third-party libraries
/var/log/em7/snippet_framework.log	Logs from the Snippet Framework core and non-step logs
/var/log/em7/snippet_framework.steps.log	Log messages aligned to any log policy
/var/log/em7/snippet_framework.policy_<policy_name>.log	Log messages for a specific policy where <policy_name> is replaced with the name used during Log Policy creation

Since file system logging is not interactive, we need to enable debug output for the Snippet Framework to generate these messages. To enable debug for the Snippet Framework, a Log Policy must be created.

Note

The Snippet Framework does not respect enabling debug through the system process, Data Collection. It must be enabled using a Log Policy.

Log Policy

A Log Policy instructs the Snippet Framework to log to the file system during the standard collection cycle. It allows a user to specify the problem area they wish to gather debug logs. For example, if a specific Dynamic Application was having issues, you could enable debug for the Dynamic Application that was causing issues. This allows for more targeted debug logs by reducing the noise from other Dynamic Applications. When creating a log policy, the name specified will be part of the filename on the file system. Using a descriptive name helps identify the purpose of the Log Policy.

Note

When running on both OL7 and OL8, the command would be sudo -u s-em7-core /opt/em7/envs/<Env-GUID>/bin/python -m silo.low_code.cli log_policy.

Interacting with Log Policies

Log Policies have CRUD (Create, Read, Update, Delete) operations through a command-line tool that is shipped with the Snippet Framework. A Log Policy’s uniqueness is determined based on the policy name.

To interact with a Log Policy, you must use the correct Python virtual environment. The virtual environment is located at /opt/em7/envs/<env_guid>/bin/python. Refer to Looking up an Execution Environment GUID.

Note

Log Policies must be configured on the collector or AIO where the collection occurs. Running the command on the CDB / DE will not update the collector.

Creating a Log Policy

A Log Policy can be created by calling log_policy create. If the name assigned already exists, the existing policy will be updated to match the provided information.

log_policy create

Create a new log policy

At least one of the following elements must be specified: did, app_id, cobj_id.

Note: Logging can affect collector CPU, memory, and disk performance.

Take care when creating log policies that will match many different criteria.

Note: When running this command, the output will be logged to snippet_framework.log and console.

log_policy create [OPTIONS]

Options

--name <name>

Required The log policy name. Required field to create, update and delete a log policy. If more than one log policy is configured, the name must be unique.

--duration <duration>

The time during which the log policy is active.

Format: “#w, #d, #h, #m, #s”

where # can be any number,

w represents weeks,

d represents days,

h represents hours,

m represents minutes,

s represents seconds.

All combinations, in any order, are valid. Time values must be separated by a comma, e.g. “1w, 3h, 30s”. If the duration has 2 or more values of the same type, they will be summed together to create a final cumulative number (ex: 1h, 2h would equal 3h). If the duration is empty, the default setting will be 30 minutes.

--did <did>

Device identifier. If the device field is empty, all the devices are logged.

--app_id <app_id>

Dynamic Application identifier. If the app_id field is empty, all Dynamic Applications are logged.

--cobj_id <cobj_id>

Collection object identifier. If the cobj_id field is empty, all the collection objects are logged.

--step <step>

Step Identifier. This name should match a registered step within the Snippet Framework. If this is not specified, all steps are valid. This option must be used in conjunction with app_id.

Example

For this example we want to create a Log Policy for Dynamic Application 1234 and Device 5. First we need to review the available parameters for create to determine what to use. To accomplish creating this Log Policy, we would run the following command:

sudo -u s-em7-core  <venv_path>/bin/python -m silo.low_code.cli log-policy create --name example_log_policy --app_id 1234 --did 5

Reading all Log Policies

All configured Log Policies can be viewed by running log_policy list. This will list all Log Policies with the remaining time until they become expired.

Example

sudo -u s-em7-core  <venv_path>/bin/python -m silo.low-code.cli log_policy list

Log Policy Name                         Expires
example_log_policy                      01/11/2024, 18:46:55

Updating an existing Log Policy

A Log Policy can be updated by calling log_policy update. If the name assigned does not exist, an error will be returned stating the Log Policy does not exist. If the policy does exist, it will be updated with the provided parameters.

log_policy update

Update an existing log policy

All values must be specified or they will update to their defaults. At least one of the following elements must be specified: did, app_id, cobj_id.

Note: Logging can affect collector CPU, memory, and disk performance.

Take care when creating log policies that will match many different criteria.

Note: When running this command, the output will be logged to snippet_framework.log and console.

log_policy update [OPTIONS]

Options

--name <name>

Required The log policy name. Required field to create, update and delete a log policy. If more than one log policy is configured, the name must be unique.

--duration <duration>

The time during which the log policy is active.

Format: “#w, #d, #h, #m, #s”

where # can be any number,

w represents weeks,

d represents days,

h represents hours,

m represents minutes,

s represents seconds.

All combinations, in any order, are valid. Time values must be separated by a comma, e.g. “1w, 3h, 30s”. If the duration has 2 or more values of the same type, they will be summed together to create a final cumulative number (ex: 1h, 2h would equal 3h). If the duration is empty, the default setting will be 30 minutes.

--did <did>

Device identifier. If the device field is empty, all the devices are logged.

--app_id <app_id>

Dynamic Application identifier. If the app_id field is empty, all Dynamic Applications are logged.

--cobj_id <cobj_id>

Collection object identifier. If the cobj_id field is empty, all the collection objects are logged.

--step <step>

Step Identifier. This name should match a registered step within the Snippet Framework. If this is not specified, all steps are valid. This option must be used in conjunction with app_id.

Example

For this example we will assume that the Log Policy, example_log_policy already exists. We have not encountered the issue yet so we want to extend the duration of the policy for another 120 minutes for the Dynamic Application 12345 and Device 5.

sudo -u s-em7-core  <venv_path>/bin/python -m silo.low_code.cli log-policy update --name example_log_policy --app_id 1234 --did 5 --duration 120m

Deleting an existing Log Policy

A Log Policy can be deleted by calling log_policy delete. If the name assigned does not exist, an error will be returned stating the Log Policy does not exist. If the policy does exist, it will be removed.

Note

This does not stop any currently-running Snippet Framework executions, but will prevent future iterations from logging.

log_policy delete

Delete an existing log policy

Note: When running this command, the output will be logged to snippet_framework.log and console.

log_policy delete [OPTIONS]

Options

--name <name>

Required The log policy name. Required field to create, update and delete a log policy. If more than one log policy is configured, the name must be unique.

Example

For this example we will assume that the Log Policy, example_log_policy already exists.

sudo -u s-em7-core  <venv_path>/bin/python -m silo.low_code.cli log_policy delete --name example_log_policy