Root Cause Report Outgoing Webhook
Features
- This section provides detailed information on webhook support provided by Skylar Automated RCA so you can build your own custom integrations.
- Root Cause report webhook payloads are sent when data is ingested and our machine learning detects an incident comprised of anomalous events.
- Frequency of Incident webhook depends on data ingest and detection of anomalies.
STEP 1: Determine the Destination Endpoint
The destination endpoint is the endpoint URL that will receive and process the content of the Root Cause Report Outgoing Webhook.
The authentication method for the endpoint can be one of the following:
- None
- Basic authentication
- Token (or Bearer) authentication
The authentication method and its associated configuration parameters will be used in STEP 2.
STEP 2: Create a Root Cause Report Outgoing Webhook Integration in Skylar Automated RCA.
- In the Skylar Automated RCA user interface, go to the Integrations & Collectors page (Settings (
) > Integrations & Collectors). - In the Webhooks section, click the button.
- Click button. The Create Outgoing RCA Webhook dialog appears.
- On the tab, enter an Integration Name for this integration.
- In the Deployment drop-down, select a deployment for the integration.
- In the Service Group(s) drop-down, select a service group for the integration. You can select Any to share suggestions for all Service Groups.
- Enter the Webhook URL that will receive and handle the POST request.
- On the tab, click .
- Enter the Webhook URL that will receive and handle the POST request.
- Select the required Authentication Method for the endpoint and complete the necessary configuration using the information from STEP 1, above.
- Click .
Root Cause Report Outgoing Webhook Payload
Payload
| Name | Type | Description |
|---|---|---|
| account | string | Skylar Automated RCA account name for this customer_name |
| customer_name | string | Customer name of Skylar Automated RCA instance |
| deployment_name | string | Name of the deployment where incident was raised |
| event_type | string | Always: “zebrium_incident” |
| first_occurrence | boolean | First time this incident has been seen |
| incident_bad_level | number | Numeric scale from 0-9 indicating the badness of the core events in the RC report (9 being very bad) |
| incident_desc_alt | string | Unused |
| incident_desc | string | Summarization of the incident assigned by NLP or the user |
| incident_epoch | integer | UTC epoch of incident start |
| incident_epoch_ts | timestamp (yyyy-mm-ddThh:mm:ss.nnnnnnZ) | UTC timestamp of incident start |
| incident_feedback | number | 1-5 Likert rating given to this incident type |
| incident_group | string | Name of the incident group where incident was raised |
| incident_hosts | string | Comma separated list of hosts participating in this incident (Skylar Automated RCA On-Prem only) |
| incident_id | uuid | Unique identifier for the incident |
| incident_jira_url | url encoded string | URL to the Jira Issue linked to this incident type |
| incident_like | url encoded string | API URL to "like" the incident |
| incident_local_offset | string | Local time offset from UTC as depicted in the log event |
| incident_local_timestamp | timestamp (yyyy-mm-ddThh:mm:ss.nnnnnn) | Local time of incident start |
| incident_logs | string | Comma separated list of logs participating in the incident (Skylar Automated RCA On-Prem only) |
| incident_mute | url encoded string | API URL to "mute" the incident |
| incident_name | string | Title of the incident assigned by NLP or the user |
| incident_owner | string | Owner assigned to this incident |
| incident_priority | string | Priority assigned to this incident (P1/P3 ) |
| incident_rare_level | number | Numeric scale from 0-9 indicating the rareness of the core events in the RC report (9 being very rare) |
| incident_repeat_ct | number | Number of times this incident type has been seen |
| incident_repeat_idx | number | Time ordered occurrence of this incident type |
| incident_short_name | string | System generated name for the incident type |
| incident_spam | url encoded string | API URL to tag incident as "spam" |
| incident_state | string | State of the incident (open, muted) |
| incident_summary | string | Summarization of the incident assigned by NLP or the user |
| incident_title | string | Title of the incident assigned by NLP or the user |
| incident_detail | string | Full details of the incident assigned by NLP or the user |
| incident_touches_agent | boolean | Incident is related to a log or metrics collector vs. application |
| incident_touches_k8s | boolean | Incident is related to Kubernetes infrastructure |
| incident_type | uuid | Unique identifier for the incident type |
| incident_url | url encoded string | URL to view incident in the Skylar Automated RCA UI |
| incident_words | word object list | List of words (w) and their rareness/size (s) and badness (b) used in the word cloud |
| service_groups | string list | List of service groups touched by this incident |
| signal_association | string | How is Incident associated to the signal (related or nearby) |
| signal_initiated | boolean | Incident is associated with a signal request |
| signal_timestamp | string | Timestamp of the signal request |
| signal_type | string | What initiated the signal. Could be USER, OPSGENIE, PAGERDUTY, SLACK |
| incident_hallmark_event | event object | Event determined to be the most severe indicator of the incident (Unused) |
| incident_events | event object list | All events in the core RC Report (level 0-2) |
| key_events | event object list | Key events (level 0) in RC Report |
| interesting_events | event object list | Interesting events (level 1) in RC Report |
| nearby_events | event object list | Nearby events (level 3-5) in RC Report |
Event Object
| Name | Type | Description |
|---|---|---|
| app | string | Application name from meta data |
| container_name | string | Container name from meta data |
| epoch | integer | UTC epoch of event |
| epoch_ts | timestamp (yyyy-mm-ddThh:mm:ss.nnnnnnZ) | UTC timestamp of event |
| etype | string | Name of the event type |
| event_context_level | integer | Event level: 0=key, 1=interesting, 2=core, 3,4,5=nearby |
| event_meta_data | set of name value pairs | Name value pairs derived from event meta data |
| event_text | string | Log event text |
| event_uuid | uuid | Unique identifier for the event |
| hallmark | boolean | True if this event is the hallmark event |
| host | string | Host on which event originated |
| incident_group | string | Name of the incident group where anomaly was raised |
| local_offset | string | Local time offset from UTC as depicted in the log event |
| local_timestamp | timestamp (yyyy-mm-ddThh:mm:ss.nnnnnn) | Local timestamp of event |
| log_name | string | Name of log basename (e.g. syslog, error) |
| namespace_name | string | Namespace name from meta data |
| root_cause | boolean | True if this event is the root cause event |
| severity_num | integer | Severity number as defined by syslog |
| severity | string | Severity text as see in the log (e.g. INFO) |
| ze_xid | uuid | Unique external identifier for the event if provided by the log collector (otherwise empty) |
Example Payload
{
"incident_id": "00000000-0000-0000-0000-000000000000",
"incident_type": "00000000-0000-0000-0000-000000000000",
"incident_epoch_ts": "2021-10-15T21:07:13.813857Z",
"incident_epoch": 1634332033813,
"incident_state": "open",
"incident_desc": "Notes let you document details of a report to help colleagues understand your analysis in the future.",
"incident_repeat_ct": 2,
"incident_local_timestamp": "2021-10-15T21:07:13.813857Z",
"incident_local_offset": "+0000",
"incident_touches_k8s": false,
"incident_touches_agent": false,
"incident_name": "SAMPLE - You would normally see An NLP-generated title here",
"incident_short_name": "cfcd2",
"incident_summary": "",
"incident_owner": "Skylar Automated RCA",
"incident_feedback": 5,
"incident_jira_url": "https://www.zebrium.com",
"incident_priority": "P3",
"service_groups": [
"sample"
],
"signal_initiated": false,
"signal_type": "",
"signal_timestamp": "",
"signal_association": "",
"incident_repeat_idx": 2,
"first_occurrence": false,
"incident_hosts": "host1,host2,host3",
"incident_logs": "logtype1,logtype2,zoom_log",
"incident_bad_level": 5,
"incident_rare_level": 5,
"incident_words": [
{
"w": "critical",
"s": 10,
"b": 4
},
{
"w": "peek",
"s": 14,
"b": 4
},
{
"w": "characterize",
"s": 14,
"b": 1
},
{
"w": "rca",
"s": 14,
"b": 2
},
{
"w": "filter",
"s": 12,
"b": 4
},
{
"w": "zoom",
"s": 10,
"b": 1
},
{
"w": "correlated",
"s": 8,
"b": 4
},
{
"w": "enjoy",
"s": 6,
"b": 2
},
{
"w": "useful",
"s": 4,
"b": 4
},
{
"w": "wordcloud",
"s": 2,
"b": 4
},
{
"w": "related",
"s": 2,
"b": 2
},
{
"w": "reports",
"s": 2,
"b": 2
},
{
"w": "data",
"s": 2,
"b": 4
},
{
"w": "zebrium",
"s": 2,
"b": 2
},
{
"w": "raw",
"s": 2,
"b": 1
},
{
"w": "fast",
"s": 2,
"b": 2
}
],
"account": "zebrium465_trial",
"customer_name": "zebrium465",
"deployment_name": "trial",
"incident_group": "sample",
"event_type": "zebrium_incident",
"incident_url": "https://cloud.zebrium.com/root-cause/report?itype_id=00000000-0000-0000-0000-000000000000&inci_id=00000000-0000-0000-0000-000000000000&ievt_level=2",
"incident_like": "https://cloud.zebrium.com /api/v2/incident/setstate/00000000-0000-0000-0000-000000000000/liked/B316BB07D18F63B61AF62416BCD7A73B960D48DD",
"incident_mute": "https://cloud.zebrium.com /api/v2/incident/setstate/00000000-0000-0000-0000-000000000000/muted/B316BB07D18F63B61AF62416BCD7A73B960D48DD",
"incident_spam": "https://cloud.zebrium.com /api/v2/incident/setstate/00000000-0000-0000-0000-000000000000/spam/B316BB07D18F63B61AF62416BCD7A73B960D48DD",
"incident_desc_alt": "Notes let you document details of a report to help colleagues understand your analysis in the future.",
"incident_hallmark_event": {
"root_cause": false,
"hallmark": true,
"epoch_ts": "2021-10-15T21:07:29.833156Z",
"epoch": 1634332049833,
"etype": "line",
"log_name": "logtype2",
"severity_num": 2,
"event_uuid": "00000000-0000-0000-0000-000000000008",
"event_text": "[2021-10-15 21:07:29.833156] CRITICAL: This is the second of two events that are used to characterize the report in the list view",
"metadata_id": "ze_deployment_name=sample,zid_container_name=logtype2,zid_host=host1,zid_log=logtype2",
"metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host1.fqdm.com",
"local_timestamp": "2021-10-15T21:07:29.833156Z",
"local_offset": "+0000",
"ze_xid": "",
"event_context_level": 0,
"host": "host1",
"severity": "Critical",
"app": null,
"container_name": "logtype2",
"namespace_name": null,
"event_meta_data": {
"ze_deployment_name": "sample",
"container_name": "logtype1-359f02372109b4222880d1c7932b717f",
"hostname": "host1.fqdm.com"
}
},
"incident_events": [
{
"root_cause": false,
"hallmark": false,
"epoch_ts": "2021-10-15T21:06:49.790742Z",
"epoch": 1634332009790,
"etype": "line",
"log_name": "logtype1",
"severity_num": 6,
"event_uuid": "00000000-0000-0000-0000-000000000003",
"event_text": "[2021-10-15 21:06:49.790742] INFO: This is a sample root cause report",
"metadata_id": "ze_deployment_name=sample,zid_container_name=logtype1,zid_host=host2,zid_log=logtype1",
"metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host2.fqdm.com",
"local_timestamp": "2021-10-15T21:06:49.790742Z",
"local_offset": "+0000",
"ze_xid": "",
"event_context_level": 1,
"host": "host2",
"severity": "Informational",
"app": null,
"container_name": "logtype1",
"namespace_name": null,
"event_meta_data": {
"ze_deployment_name": "sample",
"container_name": "logtype1-359f02372109b4222880d1c7932b717f",
"hostname": "host2.fqdm.com"
}
},
{
"root_cause": false,
"hallmark": false,
"epoch_ts": "2021-10-15T21:06:57.7982Z",
"epoch": 1634332017798,
"etype": "line",
"log_name": "logtype2",
"severity_num": 6,
"event_uuid": "00000000-0000-0000-0000-000000000004",
"event_text": "[2021-10-15 21:06:57.7982] INFO: Real Root Cause Reports typically have 5-20 \"Core\" log events",
"metadata_id": "ze_deployment_name=sample,zid_container_name=logtype2,zid_host=host2,zid_log=logtype2",
"metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host2.fqdm.com",
"local_timestamp": "2021-10-15T21:06:57.7982Z",
"local_offset": "+0000",
"ze_xid": "",
"event_context_level": 1,
"host": "host2",
"severity": "Informational",
"app": null,
"container_name": "logtype2",
"namespace_name": null,
"event_meta_data": {
"ze_deployment_name": "sample",
"container_name": "logtype1-359f02372109b4222880d1c7932b717f",
"hostname": "host2.fqdm.com"
}
},
{
"root_cause": false,
"hallmark": false,
"epoch_ts": "2021-10-15T21:07:05.805105Z",
"epoch": 1634332025805,
"etype": "line",
"log_name": "logtype2",
"severity_num": 6,
"event_uuid": "00000000-0000-0000-0000-000000000005",
"event_text": "[2021-10-15 21:07:05.805105] INFO: Core events consist of mostly \"rare\" and high-severity events that are correlated across multiple logs",
"metadata_id": "ze_deployment_name=sample,zid_container_name=logtype2,zid_host=host2,zid_log=logtype2",
"metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host2.fqdm.com",
"local_timestamp": "2021-10-15T21:07:05.805105Z",
"local_offset": "+0000",
"ze_xid": "",
"event_context_level": 1,
"host": "host2",
"severity": "Informational",
"app": null,
"container_name": "logtype2",
"namespace_name": null,
"event_meta_data": {
"ze_deployment_name": "sample",
"container_name": "logtype1-359f02372109b4222880d1c7932b717f",
"hostname": "host2.fqdm.com"
}
},
{
"root_cause": true,
"hallmark": true,
"epoch_ts": "2021-10-15T21:07:13.82029Z",
"epoch": 1634332033820,
"etype": "line",
"log_name": "logtype1",
"severity_num": 6,
"event_uuid": "00000000-0000-0000-0000-000000000006",
"event_text": "[2021-10-15 21:07:13.82029] INFO: This is the first of two events that are used to characterize the report in the list view",
"metadata_id": "ze_deployment_name=sample,zid_container_name=logtype1,zid_host=host1,zid_log=logtype1",
"metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host1.fqdm.com",
"local_timestamp": "2021-10-15T21:07:13.82029Z",
"local_offset": "+0000",
"ze_xid": "",
"event_context_level": 0,
"host": "host1",
"severity": "Informational",
"app": null,
"container_name": "logtype1",
"namespace_name": null,
"event_meta_data": {
"ze_deployment_name": "sample",
"container_name": "logtype1-359f02372109b4222880d1c7932b717f",
"hostname": "host1.fqdm.com"
}
},
{
"root_cause": false,
"hallmark": false,
"epoch_ts": "2021-10-15T21:07:21.826703Z",
"epoch": 1634332041826,
"etype": "line",
"log_name": "logtype1",
"severity_num": 3,
"event_uuid": "00000000-0000-0000-0000-000000000007",
"event_text": "[2021-10-15 21:07:21.826703] ERROR: Did you notice this event has error severity?",
"metadata_id": "ze_deployment_name=sample,zid_container_name=logtype1,zid_host=host1,zid_log=logtype1",
"metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host1.fqdm.com",
"local_timestamp": "2021-10-15T21:07:21.826703Z",
"local_offset": "+0000",
"ze_xid": "",
"event_context_level": 1,
"host": "host1",
"severity": "Error",
"app": null,
"container_name": "logtype1",
"namespace_name": null,
"event_meta_data": {
"ze_deployment_name": "sample",
"container_name": "logtype1-359f02372109b4222880d1c7932b717f",
"hostname": "host1.fqdm.com"
}
},
{
"root_cause": false,
"hallmark": true,
"epoch_ts": "2021-10-15T21:07:29.833156Z",
"epoch": 1634332049833,
"etype": "line",
"log_name": "logtype2",
"severity_num": 2,
"event_uuid": "00000000-0000-0000-0000-000000000008",
"event_text": "[2021-10-15 21:07:29.833156] CRITICAL: This is the second of two events that are used to characterize the report in the list view",
"metadata_id": "ze_deployment_name=sample,zid_container_name=logtype2,zid_host=host1,zid_log=logtype2",
"metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host1.fqdm.com",
"local_timestamp": "2021-10-15T21:07:29.833156Z",
"local_offset": "+0000",
"ze_xid": "",
"event_context_level": 0,
"host": "host1",
"severity": "Critical",
"app": null,
"container_name": "logtype2",
"namespace_name": null,
"event_meta_data": {
"ze_deployment_name": "sample",
"container_name": "logtype1-359f02372109b4222880d1c7932b717f",
"hostname": "host1.fqdm.com"
}
},
{
"root_cause": false,
"hallmark": false,
"epoch_ts": "2021-10-15T21:07:37.840903Z",
"epoch": 1634332057840,
"etype": "line",
"log_name": "logtype2",
"severity_num": 6,
"event_uuid": "00000000-0000-0000-0000-000000000009",
"event_text": "[2021-10-15 21:07:37.840903] INFO: Now try the filter bar (above), and highlight bar (below)",
"metadata_id": "ze_deployment_name=sample,zid_container_name=logtype2,zid_host=host2,zid_log=logtype2",
"metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host2.fqdm.com",
"local_timestamp": "2021-10-15T21:07:37.840903Z",
"local_offset": "+0000",
"ze_xid": "",
"event_context_level": 1,
"host": "host2",
"severity": "Informational",
"app": null,
"container_name": "logtype2",
"namespace_name": null,
"event_meta_data": {
"ze_deployment_name": "sample",
"container_name": "logtype1-359f02372109b4222880d1c7932b717f",
"hostname": "host2.fqdm.com"
}
},
{
"root_cause": false,
"hallmark": false,
"epoch_ts": "2021-10-15T21:07:45.851986Z",
"epoch": 1634332065851,
"etype": "line",
"log_name": "logtype1",
"severity_num": 2,
"event_uuid": "00000000-0000-0000-0000-000000000010",
"event_text": "[2021-10-15 21:07:45.851986] CRITICAL: If you do not see enough detail in the Core events, try these things:",
"metadata_id": "ze_deployment_name=sample,zid_container_name=logtype1,zid_host=host1,zid_log=logtype1",
"metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host1.fqdm.com",
"local_timestamp": "2021-10-15T21:07:45.851986Z",
"local_offset": "+0000",
"ze_xid": "",
"event_context_level": 1,
"host": "host1",
"severity": "Critical",
"app": null,
"container_name": "logtype1",
"namespace_name": null,
"event_meta_data": {
"ze_deployment_name": "sample",
"container_name": "logtype1-359f02372109b4222880d1c7932b717f",
"hostname": "host1.fqdm.com"
}
},
{
"root_cause": false,
"hallmark": false,
"epoch_ts": "2021-10-15T21:07:53.858345Z",
"epoch": 1634332073858,
"etype": "line",
"log_name": "logtype1",
"severity_num": 6,
"event_uuid": "00000000-0000-0000-0000-000000000011",
"event_text": "[2021-10-15 21:07:53.858345] INFO: Click the Peek button (at the end of each log line) to see all available lines from just this log stream",
"metadata_id": "ze_deployment_name=sample,zid_container_name=logtype1,zid_host=host2,zid_log=logtype1",
"metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host2.fqdm.com",
"local_timestamp": "2021-10-15T21:07:53.858345Z",
"local_offset": "+0000",
"ze_xid": "",
"event_context_level": 1,
"host": "host2",
"severity": "Informational",
"app": null,
"container_name": "logtype1",
"namespace_name": null,
"event_meta_data": {
"ze_deployment_name": "sample",
"container_name": "logtype1-359f02372109b4222880d1c7932b717f",
"hostname": "host2.fqdm.com"
}
},
{
"root_cause": false,
"hallmark": false,
"epoch_ts": "2021-10-15T21:08:01.864572Z",
"epoch": 1634332081864,
"etype": "line",
"log_name": "logtype2",
"severity_num": 6,
"event_uuid": "00000000-0000-0000-0000-000000000012",
"event_text": "[2021-10-15 21:08:01.864572] INFO: Or zoom out beyond the Core events by clicking a Zoom level in Related Events (at the top)",
"metadata_id": "ze_deployment_name=sample,zid_container_name=logtype2,zid_host=host2,zid_log=logtype2",
"metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host2.fqdm.com",
"local_timestamp": "2021-10-15T21:08:01.864572Z",
"local_offset": "+0000",
"ze_xid": "",
"event_context_level": 1,
"host": "host2",
"severity": "Informational",
"app": null,
"container_name": "logtype2",
"namespace_name": null,
"event_meta_data": {
"ze_deployment_name": "sample",
"container_name": "logtype1-359f02372109b4222880d1c7932b717f",
"hostname": "host2.fqdm.com"
}
},
{
"root_cause": false,
"hallmark": false,
"epoch_ts": "2021-10-15T21:08:09.871442Z",
"epoch": 1634332089871,
"etype": "line",
"log_name": "logtype2",
"severity_num": 6,
"event_uuid": "00000000-0000-0000-0000-000000000013",
"event_text": "[2021-10-15 21:08:09.871442] INFO: Zooming is useful when the Core events do not contain enough information",
"metadata_id": "ze_deployment_name=sample,zid_container_name=logtype2,zid_host=host1,zid_log=logtype2",
"metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host1.fqdm.com",
"local_timestamp": "2021-10-15T21:08:09.871442Z",
"local_offset": "+0000",
"ze_xid": "",
"event_context_level": 1,
"host": "host1",
"severity": "Informational",
"app": null,
"container_name": "logtype2",
"namespace_name": null,
"event_meta_data": {
"ze_deployment_name": "sample",
"container_name": "logtype1-359f02372109b4222880d1c7932b717f",
"hostname": "host1.fqdm.com"
}
},
{
"root_cause": false,
"hallmark": false,
"epoch_ts": "2021-10-15T21:08:17.878258Z",
"epoch": 1634332097878,
"etype": "line",
"log_name": "logtype2",
"severity_num": 6,
"event_uuid": "00000000-0000-0000-0000-000000000014",
"event_text": "[2021-10-15 21:08:17.878258] INFO: Enjoy using Skylar Automated RCA and let us know if you have any questions!",
"metadata_id": "ze_deployment_name=sample,zid_container_name=logtype2,zid_host=host1,zid_log=logtype2",
"metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host1.fqdm.com",
"local_timestamp": "2021-10-15T21:08:17.878258Z",
"local_offset": "+0000",
"ze_xid": "",
"event_context_level": 1,
"host": "host1",
"severity": "Informational",
"app": null,
"container_name": "logtype2",
"namespace_name": null,
"event_meta_data": {
"ze_deployment_name": "sample",
"container_name": "logtype1-359f02372109b4222880d1c7932b717f",
"hostname": "host1.fqdm.com"
}
}
],
"key_events": [
{
"root_cause": true,
"hallmark": true,
"epoch_ts": "2021-10-15T21:07:13.82029Z",
"epoch": 1634332033820,
"etype": "line",
"log_name": "logtype1",
"severity_num": 6,
"event_uuid": "00000000-0000-0000-0000-000000000006",
"event_text": "[2021-10-15 21:07:13.82029] INFO: This is the first of two events that are used to characterize the report in the list view",
"metadata_id": "ze_deployment_name=sample,zid_container_name=logtype1,zid_host=host1,zid_log=logtype1",
"metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host1.fqdm.com",
"local_timestamp": "2021-10-15T21:07:13.82029Z",
"local_offset": "+0000",
"ze_xid": "",
"event_context_level": 0,
"host": "host1",
"severity": "Informational",
"app": null,
"container_name": "logtype1",
"namespace_name": null,
"event_meta_data": {
"ze_deployment_name": "sample",
"container_name": "logtype1-359f02372109b4222880d1c7932b717f",
"hostname": "host1.fqdm.com"
}
},
{
"root_cause": false,
"hallmark": true,
"epoch_ts": "2021-10-15T21:07:29.833156Z",
"epoch": 1634332049833,
"etype": "line",
"log_name": "logtype2",
"severity_num": 2,
"event_uuid": "00000000-0000-0000-0000-000000000008",
"event_text": "[2021-10-15 21:07:29.833156] CRITICAL: This is the second of two events that are used to characterize the report in the list view",
"metadata_id": "ze_deployment_name=sample,zid_container_name=logtype2,zid_host=host1,zid_log=logtype2",
"metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host1.fqdm.com",
"local_timestamp": "2021-10-15T21:07:29.833156Z",
"local_offset": "+0000",
"ze_xid": "",
"event_context_level": 0,
"host": "host1",
"severity": "Critical",
"app": null,
"container_name": "logtype2",
"namespace_name": null,
"event_meta_data": {
"ze_deployment_name": "sample",
"container_name": "logtype1-359f02372109b4222880d1c7932b717f",
"hostname": "host1.fqdm.com"
}
}
],
"interesting_events": [
{
"root_cause": false,
"hallmark": false,
"epoch_ts": "2021-10-15T21:06:49.790742Z",
"epoch": 1634332009790,
"etype": "line",
"log_name": "logtype1",
"severity_num": 6,
"event_uuid": "00000000-0000-0000-0000-000000000003",
"event_text": "[2021-10-15 21:06:49.790742] INFO: This is a sample root cause report",
"metadata_id": "ze_deployment_name=sample,zid_container_name=logtype1,zid_host=host2,zid_log=logtype1",
"metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host2.fqdm.com",
"local_timestamp": "2021-10-15T21:06:49.790742Z",
"local_offset": "+0000",
"ze_xid": "",
"event_context_level": 1,
"host": "host2",
"severity": "Informational",
"app": null,
"container_name": "logtype1",
"namespace_name": null,
"event_meta_data": {
"ze_deployment_name": "sample",
"container_name": "logtype1-359f02372109b4222880d1c7932b717f",
"hostname": "host2.fqdm.com"
}
},
{
"root_cause": false,
"hallmark": false,
"epoch_ts": "2021-10-15T21:06:57.7982Z",
"epoch": 1634332017798,
"etype": "line",
"log_name": "logtype2",
"severity_num": 6,
"event_uuid": "00000000-0000-0000-0000-000000000004",
"event_text": "[2021-10-15 21:06:57.7982] INFO: Real Root Cause Reports typically have 5-20 \"Core\" log events",
"metadata_id": "ze_deployment_name=sample,zid_container_name=logtype2,zid_host=host2,zid_log=logtype2",
"metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host2.fqdm.com",
"local_timestamp": "2021-10-15T21:06:57.7982Z",
"local_offset": "+0000",
"ze_xid": "",
"event_context_level": 1,
"host": "host2",
"severity": "Informational",
"app": null,
"container_name": "logtype2",
"namespace_name": null,
"event_meta_data": {
"ze_deployment_name": "sample",
"container_name": "logtype1-359f02372109b4222880d1c7932b717f",
"hostname": "host2.fqdm.com"
}
},
{
"root_cause": false,
"hallmark": false,
"epoch_ts": "2021-10-15T21:07:05.805105Z",
"epoch": 1634332025805,
"etype": "line",
"log_name": "logtype2",
"severity_num": 6,
"event_uuid": "00000000-0000-0000-0000-000000000005",
"event_text": "[2021-10-15 21:07:05.805105] INFO: Core events consist of mostly \"rare\" and high-severity events that are correlated across multiple logs",
"metadata_id": "ze_deployment_name=sample,zid_container_name=logtype2,zid_host=host2,zid_log=logtype2",
"metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host2.fqdm.com",
"local_timestamp": "2021-10-15T21:07:05.805105Z",
"local_offset": "+0000",
"ze_xid": "",
"event_context_level": 1,
"host": "host2",
"severity": "Informational",
"app": null,
"container_name": "logtype2",
"namespace_name": null,
"event_meta_data": {
"ze_deployment_name": "sample",
"container_name": "logtype1-359f02372109b4222880d1c7932b717f",
"hostname": "host2.fqdm.com"
}
},
{
"root_cause": false,
"hallmark": false,
"epoch_ts": "2021-10-15T21:07:21.826703Z",
"epoch": 1634332041826,
"etype": "line",
"log_name": "logtype1",
"severity_num": 3,
"event_uuid": "00000000-0000-0000-0000-000000000007",
"event_text": "[2021-10-15 21:07:21.826703] ERROR: Did you notice this event has error severity?",
"metadata_id": "ze_deployment_name=sample,zid_container_name=logtype1,zid_host=host1,zid_log=logtype1",
"metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host1.fqdm.com",
"local_timestamp": "2021-10-15T21:07:21.826703Z",
"local_offset": "+0000",
"ze_xid": "",
"event_context_level": 1,
"host": "host1",
"severity": "Error",
"app": null,
"container_name": "logtype1",
"namespace_name": null,
"event_meta_data": {
"ze_deployment_name": "sample",
"container_name": "logtype1-359f02372109b4222880d1c7932b717f",
"hostname": "host1.fqdm.com"
}
},
{
"root_cause": false,
"hallmark": false,
"epoch_ts": "2021-10-15T21:07:37.840903Z",
"epoch": 1634332057840,
"etype": "line",
"log_name": "logtype2",
"severity_num": 6,
"event_uuid": "00000000-0000-0000-0000-000000000009",
"event_text": "[2021-10-15 21:07:37.840903] INFO: Now try the filter bar (above), and highlight bar (below)",
"metadata_id": "ze_deployment_name=sample,zid_container_name=logtype2,zid_host=host2,zid_log=logtype2",
"metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host2.fqdm.com",
"local_timestamp": "2021-10-15T21:07:37.840903Z",
"local_offset": "+0000",
"ze_xid": "",
"event_context_level": 1,
"host": "host2",
"severity": "Informational",
"app": null,
"container_name": "logtype2",
"namespace_name": null,
"event_meta_data": {
"ze_deployment_name": "sample",
"container_name": "logtype1-359f02372109b4222880d1c7932b717f",
"hostname": "host2.fqdm.com"
}
},
{
"root_cause": false,
"hallmark": false,
"epoch_ts": "2021-10-15T21:07:45.851986Z",
"epoch": 1634332065851,
"etype": "line",
"log_name": "logtype1",
"severity_num": 2,
"event_uuid": "00000000-0000-0000-0000-000000000010",
"event_text": "[2021-10-15 21:07:45.851986] CRITICAL: If you do not see enough detail in the Core events, try these things:",
"metadata_id": "ze_deployment_name=sample,zid_container_name=logtype1,zid_host=host1,zid_log=logtype1",
"metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host1.fqdm.com",
"local_timestamp": "2021-10-15T21:07:45.851986Z",
"local_offset": "+0000",
"ze_xid": "",
"event_context_level": 1,
"host": "host1",
"severity": "Critical",
"app": null,
"container_name": "logtype1",
"namespace_name": null,
"event_meta_data": {
"ze_deployment_name": "sample",
"container_name": "logtype1-359f02372109b4222880d1c7932b717f",
"hostname": "host1.fqdm.com"
}
},
{
"root_cause": false,
"hallmark": false,
"epoch_ts": "2021-10-15T21:07:53.858345Z",
"epoch": 1634332073858,
"etype": "line",
"log_name": "logtype1",
"severity_num": 6,
"event_uuid": "00000000-0000-0000-0000-000000000011",
"event_text": "[2021-10-15 21:07:53.858345] INFO: Click the Peek button (at the end of each log line) to see all available lines from just this log stream",
"metadata_id": "ze_deployment_name=sample,zid_container_name=logtype1,zid_host=host2,zid_log=logtype1",
"metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host2.fqdm.com",
"local_timestamp": "2021-10-15T21:07:53.858345Z",
"local_offset": "+0000",
"ze_xid": "",
"event_context_level": 1,
"host": "host2",
"severity": "Informational",
"app": null,
"container_name": "logtype1",
"namespace_name": null,
"event_meta_data": {
"ze_deployment_name": "sample",
"container_name": "logtype1-359f02372109b4222880d1c7932b717f",
"hostname": "host2.fqdm.com"
}
},
{
"root_cause": false,
"hallmark": false,
"epoch_ts": "2021-10-15T21:08:01.864572Z",
"epoch": 1634332081864,
"etype": "line",
"log_name": "logtype2",
"severity_num": 6,
"event_uuid": "00000000-0000-0000-0000-000000000012",
"event_text": "[2021-10-15 21:08:01.864572] INFO: Or zoom out beyond the Core events by clicking a Zoom level in Related Events (at the top)",
"metadata_id": "ze_deployment_name=sample,zid_container_name=logtype2,zid_host=host2,zid_log=logtype2",
"metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host2.fqdm.com",
"local_timestamp": "2021-10-15T21:08:01.864572Z",
"local_offset": "+0000",
"ze_xid": "",
"event_context_level": 1,
"host": "host2",
"severity": "Informational",
"app": null,
"container_name": "logtype2",
"namespace_name": null,
"event_meta_data": {
"ze_deployment_name": "sample",
"container_name": "logtype1-359f02372109b4222880d1c7932b717f",
"hostname": "host2.fqdm.com"
}
},
{
"root_cause": false,
"hallmark": false,
"epoch_ts": "2021-10-15T21:08:09.871442Z",
"epoch": 1634332089871,
"etype": "line",
"log_name": "logtype2",
"severity_num": 6,
"event_uuid": "00000000-0000-0000-0000-000000000013",
"event_text": "[2021-10-15 21:08:09.871442] INFO: Zooming is useful when the Core events do not contain enough information",
"metadata_id": "ze_deployment_name=sample,zid_container_name=logtype2,zid_host=host1,zid_log=logtype2",
"metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host1.fqdm.com",
"local_timestamp": "2021-10-15T21:08:09.871442Z",
"local_offset": "+0000",
"ze_xid": "",
"event_context_level": 1,
"host": "host1",
"severity": "Informational",
"app": null,
"container_name": "logtype2",
"namespace_name": null,
"event_meta_data": {
"ze_deployment_name": "sample",
"container_name": "logtype1-359f02372109b4222880d1c7932b717f",
"hostname": "host1.fqdm.com"
}
},
{
"root_cause": false,
"hallmark": false,
"epoch_ts": "2021-10-15T21:08:17.878258Z",
"epoch": 1634332097878,
"etype": "line",
"log_name": "logtype2",
"severity_num": 6,
"event_uuid": "00000000-0000-0000-0000-000000000014",
"event_text": "[2021-10-15 21:08:17.878258] INFO: Enjoy using Skylar Automated RCA and let us know if you have any questions!",
"metadata_id": "ze_deployment_name=sample,zid_container_name=logtype2,zid_host=host1,zid_log=logtype2",
"metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host1.fqdm.com",
"local_timestamp": "2021-10-15T21:08:17.878258Z",
"local_offset": "+0000",
"ze_xid": "",
"event_context_level": 1,
"host": "host1",
"severity": "Informational",
"app": null,
"container_name": "logtype2",
"namespace_name": null,
"event_meta_data": {
"ze_deployment_name": "sample",
"container_name": "logtype1-359f02372109b4222880d1c7932b717f",
"hostname": "host1.fqdm.com"
}
}
],
"nearby_events": [
{
"root_cause": false,
"hallmark": false,
"epoch_ts": "2021-10-15T21:06:25.77145Z",
"epoch": 1634331985771,
"etype": "line",
"log_name": "zoom_log",
"severity_num": 6,
"event_uuid": "00000000-0000-0000-0000-000000000000",
"event_text": "[2021-10-15 21:06:25.77145] INFO: You are seeing this event because you zoomed into Related Events level 3 (or because you Peeked)",
"metadata_id": "ze_deployment_name=sample,zid_container_name=zoom_log,zid_host=host3,zid_log=zoom_log",
"metadata_cfg": "ze_deployment_name=sample,container_name=zoom_log-a32e129fccd92e3ab19e749655f152a7,hostname=host3.fqdm.com",
"local_timestamp": "2021-10-15T21:06:25.77145Z",
"local_offset": "+0000",
"ze_xid": "",
"event_context_level": 5,
"host": "host3",
"severity": "Informational",
"app": null,
"container_name": "zoom_log",
"namespace_name": null,
"event_meta_data": {
"ze_deployment_name": "sample",
"container_name": "zoom_log-a32e129fccd92e3ab19e749655f152a7",
"hostname": "host3.fqdm.com"
}
},
{
"root_cause": false,
"hallmark": false,
"epoch_ts": "2021-10-15T21:06:33.778395Z",
"epoch": 1634331993778,
"etype": "line",
"log_name": "zoom_log",
"severity_num": 6,
"event_uuid": "00000000-0000-0000-0000-000000000001",
"event_text": "[2021-10-15 21:06:33.778395] INFO: You are seeing this event because you zoomed into Related Events level 2 (or because you Peeked)",
"metadata_id": "ze_deployment_name=sample,zid_container_name=zoom_log,zid_host=host3,zid_log=zoom_log",
"metadata_cfg": "ze_deployment_name=sample,container_name=zoom_log-a32e129fccd92e3ab19e749655f152a7,hostname=host3.fqdm.com",
"local_timestamp": "2021-10-15T21:06:33.778395Z",
"local_offset": "+0000",
"ze_xid": "",
"event_context_level": 4,
"host": "host3",
"severity": "Informational",
"app": null,
"container_name": "zoom_log",
"namespace_name": null,
"event_meta_data": {
"ze_deployment_name": "sample",
"container_name": "zoom_log-a32e129fccd92e3ab19e749655f152a7",
"hostname": "host3.fqdm.com"
}
},
{
"root_cause": false,
"hallmark": false,
"epoch_ts": "2021-10-15T21:06:41.784659Z",
"epoch": 1634332001784,
"etype": "line",
"log_name": "zoom_log",
"severity_num": 6,
"event_uuid": "00000000-0000-0000-0000-000000000002",
"event_text": "[2021-10-15 21:06:41.784659] INFO: You are seeing this event because you zoomed into Related Events level 1 (or because you Peeked)",
"metadata_id": "ze_deployment_name=sample,zid_container_name=zoom_log,zid_host=host3,zid_log=zoom_log",
"metadata_cfg": "ze_deployment_name=sample,container_name=zoom_log-a32e129fccd92e3ab19e749655f152a7,hostname=host3.fqdm.com",
"local_timestamp": "2021-10-15T21:06:41.784659Z",
"local_offset": "+0000",
"ze_xid": "",
"event_context_level": 3,
"host": "host3",
"severity": "Informational",
"app": null,
"container_name": "zoom_log",
"namespace_name": null,
"event_meta_data": {
"ze_deployment_name": "sample",
"container_name": "zoom_log-a32e129fccd92e3ab19e749655f152a7",
"hostname": "host3.fqdm.com"
}
},
{
"root_cause": false,
"hallmark": false,
"epoch_ts": "2021-10-15T21:08:25.885936Z",
"epoch": 1634332105885,
"etype": "line",
"log_name": "zoom_log",
"severity_num": 6,
"event_uuid": "00000000-0000-0000-0000-000000000015",
"event_text": "[2021-10-15 21:08:25.885936] INFO: This is the last event in the Related Events level 1 zoom out",
"metadata_id": "ze_deployment_name=sample,zid_container_name=zoom_log,zid_host=host3,zid_log=zoom_log",
"metadata_cfg": "ze_deployment_name=sample,container_name=zoom_log-a32e129fccd92e3ab19e749655f152a7,hostname=host3.fqdm.com",
"local_timestamp": "2021-10-15T21:08:25.885936Z",
"local_offset": "+0000",
"ze_xid": "",
"event_context_level": 3,
"host": "host3",
"severity": "Informational",
"app": null,
"container_name": "zoom_log",
"namespace_name": null,
"event_meta_data": {
"ze_deployment_name": "sample",
"container_name": "zoom_log-a32e129fccd92e3ab19e749655f152a7",
"hostname": "host3.fqdm.com"
}
},
{
"root_cause": false,
"hallmark": false,
"epoch_ts": "2021-10-15T21:08:33.896882Z",
"epoch": 1634332113896,
"etype": "line",
"log_name": "zoom_log",
"severity_num": 6,
"event_uuid": "00000000-0000-0000-0000-000000000016",
"event_text": "[2021-10-15 21:08:33.896882] INFO: This is the last event in the Related Events level 2 zoom out",
"metadata_id": "ze_deployment_name=sample,zid_container_name=zoom_log,zid_host=host3,zid_log=zoom_log",
"metadata_cfg": "ze_deployment_name=sample,container_name=zoom_log-a32e129fccd92e3ab19e749655f152a7,hostname=host3.fqdm.com",
"local_timestamp": "2021-10-15T21:08:33.896882Z",
"local_offset": "+0000",
"ze_xid": "",
"event_context_level": 4,
"host": "host3",
"severity": "Informational",
"app": null,
"container_name": "zoom_log",
"namespace_name": null,
"event_meta_data": {
"ze_deployment_name": "sample",
"container_name": "zoom_log-a32e129fccd92e3ab19e749655f152a7",
"hostname": "host3.fqdm.com"
}
},
{
"root_cause": false,
"hallmark": false,
"epoch_ts": "2021-10-15T21:08:41.903443Z",
"epoch": 1634332121903,
"etype": "line",
"log_name": "zoom_log",
"severity_num": 6,
"event_uuid": "00000000-0000-0000-0000-000000000017",
"event_text": "[2021-10-15 21:08:41.903443] INFO: This is the last event in the Related Events level 3 zoom out",
"metadata_id": "ze_deployment_name=sample,zid_container_name=zoom_log,zid_host=host3,zid_log=zoom_log",
"metadata_cfg": "ze_deployment_name=sample,container_name=zoom_log-a32e129fccd92e3ab19e749655f152a7,hostname=host3.fqdm.com",
"local_timestamp": "2021-10-15T21:08:41.903443Z",
"local_offset": "+0000",
"ze_xid": "",
"event_context_level": 5,
"host": "host3",
"severity": "Informational",
"app": null,
"container_name": "zoom_log",
"namespace_name": null,
"event_meta_data": {
"ze_deployment_name": "sample",
"container_name": "zoom_log-a32e129fccd92e3ab19e749655f152a7",
"hostname": "host3.fqdm.com"
}
}
]
}