Root Cause Report Outgoing Webhook

Download this manual as a PDF file

Features

  • This section provides detailed information on webhook support provided by Skylar Automated RCA so you can build your own custom integrations.
  • Root Cause report webhook payloads are sent when data is ingested and our machine learning detects an incident comprised of anomalous events.
  • Frequency of Incident webhook depends on data ingest and detection of anomalies.

STEP 1: Determine the Destination Endpoint

The destination endpoint is the endpoint URL that will receive and process the content of the Root Cause Report Outgoing Webhook.

The authentication method for the endpoint can be one of the following:

  • None
  • Basic authentication
  • Token (or Bearer) authentication

The authentication method and its associated configuration parameters will be used in STEP 2.

STEP 2: Create a Root Cause Report Outgoing Webhook Integration in Skylar Automated RCA.

  1. In the Skylar Automated RCA user interface, go to the Integrations & Collectors page (Settings () > Integrations & Collectors).
  2. In the Webhooks section, click the Outgoing RCA button.
  3. Click Create a New Integration button. The Create Outgoing RCA Webhook dialog appears.
  4. On the General tab, enter an Integration Name for this integration.
  5. In the Deployment drop-down, select a deployment for the integration.
  6. In the Service Group(s) drop-down, select a service group for the integration.
  7. Enter the Webhook URL that will receive and handle the POST request.
  8. On the Send Detections tab, click Enabled.
  9. Enter the Webhook URL that will receive and handle the POST request.
  10. Select the required Authentication Method for the endpoint and complete the necessary configuration using the information from STEP 1, above.
  11. Click Save.

Root Cause Report Outgoing Webhook Payload

Payload

Name Type Description
account string Skylar Automated RCA account name for this customer_name
customer_name string Customer name of Skylar Automated RCA instance
deployment_name string Name of the deployment where incident was raised
event_type string Always: “zebrium_incident”
first_occurrence boolean First time this incident has been seen
incident_bad_level number Numeric scale from 0-9 indicating the badness of the core events in the RC report (9 being very bad)
incident_desc_alt string Unused
incident_desc string Summarization of the incident assigned by NLP or the user
incident_epoch integer UTC epoch of incident start
incident_epoch_ts timestamp (yyyy-mm-ddThh:mm:ss.nnnnnnZ) UTC timestamp of incident start
incident_feedback number 1-5 Likert rating given to this incident type
incident_group string Name of the incident group where incident was raised
incident_hosts string Comma separated list of hosts participating in this incident (Skylar Automated RCA On-Prem only)
incident_id uuid Unique identifier for the incident
incident_jira_url url encoded string URL to the Jira Issue linked to this incident type
incident_like url encoded string API URL to "like" the incident
incident_local_offset string Local time offset from UTC as depicted in the log event
incident_local_timestamp timestamp (yyyy-mm-ddThh:mm:ss.nnnnnn) Local time of incident start
incident_logs string Comma separated list of logs participating in the incident (Skylar Automated RCA On-Prem only)
incident_mute url encoded string API URL to "mute" the incident
incident_name string Title of the incident assigned by NLP or the user
incident_owner string Owner assigned to this incident
incident_priority string Priority assigned to this incident (P1/P3 )
incident_rare_level number Numeric scale from 0-9 indicating the rareness of the core events in the RC report (9 being very rare)
incident_repeat_ct number Number of times this incident type has been seen
incident_repeat_idx number Time ordered occurrence of this incident type
incident_short_name string System generated name for the incident type
incident_spam url encoded string API URL to tag incident as "spam"
incident_state string State of the incident (open, muted)
incident_summary string Summarization of the incident assigned by NLP or the user
incident_title string Title of the incident assigned by NLP or the user
incident_detail string Full details of the incident assigned by NLP or the user
incident_touches_agent boolean Incident is related to a log or metrics collector vs. application
incident_touches_k8s boolean Incident is related to Kubernetes infrastructure
incident_type uuid Unique identifier for the incident type
incident_url url encoded string URL to view incident in the Skylar Automated RCA UI
incident_words word object list List of words (w) and their rareness/size (s) and badness (b) used in the word cloud
service_groups string list List of service groups touched by this incident
signal_association string How is Incident associated to the signal (related or nearby)
signal_initiated boolean Incident is associated with a signal request
signal_timestamp string Timestamp of the signal request
signal_type string What initiated the signal. Could be USER, OPSGENIE, PAGERDUTY, SLACK
incident_hallmark_event event object Event determined to be the most severe indicator of the incident (Unused)
incident_events event object list All events in the core RC Report (level 0-2)
key_events event object list Key events (level 0) in RC Report
interesting_events event object list Interesting events (level 1) in RC Report
nearby_events event object list Nearby events (level 3-5) in RC Report

Event Object

Name Type Description
app string Application name from meta data
container_name string Container name from meta data
epoch integer UTC epoch of event
epoch_ts timestamp (yyyy-mm-ddThh:mm:ss.nnnnnnZ) UTC timestamp of event
etype string Name of the event type
event_context_level integer Event level: 0=key, 1=interesting, 2=core, 3,4,5=nearby
event_meta_data set of name value pairs Name value pairs derived from event meta data
event_text string Log event text
event_uuid uuid Unique identifier for the event
hallmark boolean True if this event is the hallmark event
host string Host on which event originated
incident_group string Name of the incident group where anomaly was raised
local_offset string Local time offset from UTC as depicted in the log event
local_timestamp timestamp (yyyy-mm-ddThh:mm:ss.nnnnnn) Local timestamp of event
log_name string Name of log basename (e.g. syslog, error)
namespace_name string Namespace name from meta data
root_cause boolean True if this event is the root cause event
severity_num integer Severity number as defined by syslog
severity string Severity text as see in the log (e.g. INFO)
ze_xid uuid Unique external identifier for the event if provided by the log collector (otherwise empty)

Example Payload

{
  "incident_id": "00000000-0000-0000-0000-000000000000",
  "incident_type": "00000000-0000-0000-0000-000000000000",
  "incident_epoch_ts": "2021-10-15T21:07:13.813857Z",
  "incident_epoch": 1634332033813,
  "incident_state": "open",
  "incident_desc": "Notes let you document details of a report to help colleagues understand your analysis in the future.",
  "incident_repeat_ct": 2,
  "incident_local_timestamp": "2021-10-15T21:07:13.813857Z",
  "incident_local_offset": "+0000",
  "incident_touches_k8s": false,
  "incident_touches_agent": false,
  "incident_name": "SAMPLE - You would normally see An NLP-generated title here",
  "incident_short_name": "cfcd2",
  "incident_summary": "",
  "incident_owner": "Skylar Automated RCA",
  "incident_feedback": 5,
  "incident_jira_url": "https://www.zebrium.com",
  "incident_priority": "P3",
  "service_groups": [
    "sample"
  ],
  "signal_initiated": false,
  "signal_type": "",
  "signal_timestamp": "",
  "signal_association": "",
  "incident_repeat_idx": 2,
  "first_occurrence": false,
  "incident_hosts": "host1,host2,host3",
  "incident_logs": "logtype1,logtype2,zoom_log",
  "incident_bad_level": 5,
  "incident_rare_level": 5,
  "incident_words": [
    {
      "w": "critical",
      "s": 10,
      "b": 4
    },
    {
      "w": "peek",
      "s": 14,
      "b": 4
    },
    {
      "w": "characterize",
      "s": 14,
      "b": 1
    },
    {
      "w": "rca",
      "s": 14,
      "b": 2
    },
    {
      "w": "filter",
      "s": 12,
      "b": 4
    },
    {
      "w": "zoom",
      "s": 10,
      "b": 1
    },
    {
      "w": "correlated",
      "s": 8,
      "b": 4
    },
    {
      "w": "enjoy",
      "s": 6,
      "b": 2
    },
    {
      "w": "useful",
      "s": 4,
      "b": 4
    },
    {
      "w": "wordcloud",
      "s": 2,
      "b": 4
    },
    {
      "w": "related",
      "s": 2,
      "b": 2
    },
    {
      "w": "reports",
      "s": 2,
      "b": 2
    },
    {
      "w": "data",
      "s": 2,
      "b": 4
    },
    {
      "w": "zebrium",
      "s": 2,
      "b": 2
    },
    {
      "w": "raw",
      "s": 2,
      "b": 1
    },
    {
      "w": "fast",
      "s": 2,
      "b": 2
    }
  ],
  "account": "zebrium465_trial",
  "customer_name": "zebrium465",
  "deployment_name": "trial",
  "incident_group": "sample",
  "event_type": "zebrium_incident",
  "incident_url": "https://cloud.zebrium.com/root-cause/report?itype_id=00000000-0000-0000-0000-000000000000&inci_id=00000000-0000-0000-0000-000000000000&ievt_level=2",
  "incident_like": "https://cloud.zebrium.com /api/v2/incident/setstate/00000000-0000-0000-0000-000000000000/liked/B316BB07D18F63B61AF62416BCD7A73B960D48DD",
  "incident_mute": "https://cloud.zebrium.com /api/v2/incident/setstate/00000000-0000-0000-0000-000000000000/muted/B316BB07D18F63B61AF62416BCD7A73B960D48DD",
  "incident_spam": "https://cloud.zebrium.com /api/v2/incident/setstate/00000000-0000-0000-0000-000000000000/spam/B316BB07D18F63B61AF62416BCD7A73B960D48DD",
  "incident_desc_alt": "Notes let you document details of a report to help colleagues understand your analysis in the future.",
  "incident_hallmark_event": {
    "root_cause": false,
    "hallmark": true,
    "epoch_ts": "2021-10-15T21:07:29.833156Z",
    "epoch": 1634332049833,
    "etype": "line",
    "log_name": "logtype2",
    "severity_num": 2,
    "event_uuid": "00000000-0000-0000-0000-000000000008",
    "event_text": "[2021-10-15 21:07:29.833156] CRITICAL: This is the second of two events that are used to characterize the report in the list view",
    "metadata_id": "ze_deployment_name=sample,zid_container_name=logtype2,zid_host=host1,zid_log=logtype2",
    "metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host1.fqdm.com",
    "local_timestamp": "2021-10-15T21:07:29.833156Z",
    "local_offset": "+0000",
    "ze_xid": "",
    "event_context_level": 0,
    "host": "host1",
    "severity": "Critical",
    "app": null,
    "container_name": "logtype2",
    "namespace_name": null,
    "event_meta_data": {
      "ze_deployment_name": "sample",
      "container_name": "logtype1-359f02372109b4222880d1c7932b717f",
      "hostname": "host1.fqdm.com"
    }
  },
  "incident_events": [
    {
      "root_cause": false,
      "hallmark": false,
      "epoch_ts": "2021-10-15T21:06:49.790742Z",
      "epoch": 1634332009790,
      "etype": "line",
      "log_name": "logtype1",
      "severity_num": 6,
      "event_uuid": "00000000-0000-0000-0000-000000000003",
      "event_text": "[2021-10-15 21:06:49.790742] INFO: This is a sample root cause report",
      "metadata_id": "ze_deployment_name=sample,zid_container_name=logtype1,zid_host=host2,zid_log=logtype1",
      "metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host2.fqdm.com",
      "local_timestamp": "2021-10-15T21:06:49.790742Z",
      "local_offset": "+0000",
      "ze_xid": "",
      "event_context_level": 1,
      "host": "host2",
      "severity": "Informational",
      "app": null,
      "container_name": "logtype1",
      "namespace_name": null,
      "event_meta_data": {
        "ze_deployment_name": "sample",
        "container_name": "logtype1-359f02372109b4222880d1c7932b717f",
        "hostname": "host2.fqdm.com"
      }
    },
    {
      "root_cause": false,
      "hallmark": false,
      "epoch_ts": "2021-10-15T21:06:57.7982Z",
      "epoch": 1634332017798,
      "etype": "line",
      "log_name": "logtype2",
      "severity_num": 6,
      "event_uuid": "00000000-0000-0000-0000-000000000004",
      "event_text": "[2021-10-15 21:06:57.7982] INFO: Real Root Cause Reports typically have 5-20 \"Core\" log events",
      "metadata_id": "ze_deployment_name=sample,zid_container_name=logtype2,zid_host=host2,zid_log=logtype2",
      "metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host2.fqdm.com",
      "local_timestamp": "2021-10-15T21:06:57.7982Z",
      "local_offset": "+0000",
      "ze_xid": "",
      "event_context_level": 1,
      "host": "host2",
      "severity": "Informational",
      "app": null,
      "container_name": "logtype2",
      "namespace_name": null,
      "event_meta_data": {
        "ze_deployment_name": "sample",
        "container_name": "logtype1-359f02372109b4222880d1c7932b717f",
        "hostname": "host2.fqdm.com"
      }
    },
    {
      "root_cause": false,
      "hallmark": false,
      "epoch_ts": "2021-10-15T21:07:05.805105Z",
      "epoch": 1634332025805,
      "etype": "line",
      "log_name": "logtype2",
      "severity_num": 6,
      "event_uuid": "00000000-0000-0000-0000-000000000005",
      "event_text": "[2021-10-15 21:07:05.805105] INFO: Core events consist of mostly \"rare\" and high-severity events that are correlated across multiple logs",
      "metadata_id": "ze_deployment_name=sample,zid_container_name=logtype2,zid_host=host2,zid_log=logtype2",
      "metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host2.fqdm.com",
      "local_timestamp": "2021-10-15T21:07:05.805105Z",
      "local_offset": "+0000",
      "ze_xid": "",
      "event_context_level": 1,
      "host": "host2",
      "severity": "Informational",
      "app": null,
      "container_name": "logtype2",
      "namespace_name": null,
      "event_meta_data": {
        "ze_deployment_name": "sample",
        "container_name": "logtype1-359f02372109b4222880d1c7932b717f",
        "hostname": "host2.fqdm.com"
      }
    },
    {
      "root_cause": true,
      "hallmark": true,
      "epoch_ts": "2021-10-15T21:07:13.82029Z",
      "epoch": 1634332033820,
      "etype": "line",
      "log_name": "logtype1",
      "severity_num": 6,
      "event_uuid": "00000000-0000-0000-0000-000000000006",
      "event_text": "[2021-10-15 21:07:13.82029] INFO: This is the first of two events that are used to characterize the report in the list view",
      "metadata_id": "ze_deployment_name=sample,zid_container_name=logtype1,zid_host=host1,zid_log=logtype1",
      "metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host1.fqdm.com",
      "local_timestamp": "2021-10-15T21:07:13.82029Z",
      "local_offset": "+0000",
      "ze_xid": "",
      "event_context_level": 0,
      "host": "host1",
      "severity": "Informational",
      "app": null,
      "container_name": "logtype1",
      "namespace_name": null,
      "event_meta_data": {
        "ze_deployment_name": "sample",
        "container_name": "logtype1-359f02372109b4222880d1c7932b717f",
        "hostname": "host1.fqdm.com"
      }
    },
    {
      "root_cause": false,
      "hallmark": false,
      "epoch_ts": "2021-10-15T21:07:21.826703Z",
      "epoch": 1634332041826,
      "etype": "line",
      "log_name": "logtype1",
      "severity_num": 3,
      "event_uuid": "00000000-0000-0000-0000-000000000007",
      "event_text": "[2021-10-15 21:07:21.826703] ERROR: Did you notice this event has error severity?",
      "metadata_id": "ze_deployment_name=sample,zid_container_name=logtype1,zid_host=host1,zid_log=logtype1",
      "metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host1.fqdm.com",
      "local_timestamp": "2021-10-15T21:07:21.826703Z",
      "local_offset": "+0000",
      "ze_xid": "",
      "event_context_level": 1,
      "host": "host1",
      "severity": "Error",
      "app": null,
      "container_name": "logtype1",
      "namespace_name": null,
      "event_meta_data": {
        "ze_deployment_name": "sample",
        "container_name": "logtype1-359f02372109b4222880d1c7932b717f",
        "hostname": "host1.fqdm.com"
      }
    },
    {
      "root_cause": false,
      "hallmark": true,
      "epoch_ts": "2021-10-15T21:07:29.833156Z",
      "epoch": 1634332049833,
      "etype": "line",
      "log_name": "logtype2",
      "severity_num": 2,
      "event_uuid": "00000000-0000-0000-0000-000000000008",
      "event_text": "[2021-10-15 21:07:29.833156] CRITICAL: This is the second of two events that are used to characterize the report in the list view",
      "metadata_id": "ze_deployment_name=sample,zid_container_name=logtype2,zid_host=host1,zid_log=logtype2",
      "metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host1.fqdm.com",
      "local_timestamp": "2021-10-15T21:07:29.833156Z",
      "local_offset": "+0000",
      "ze_xid": "",
      "event_context_level": 0,
      "host": "host1",
      "severity": "Critical",
      "app": null,
      "container_name": "logtype2",
      "namespace_name": null,
      "event_meta_data": {
        "ze_deployment_name": "sample",
        "container_name": "logtype1-359f02372109b4222880d1c7932b717f",
        "hostname": "host1.fqdm.com"
      }
    },
    {
      "root_cause": false,
      "hallmark": false,
      "epoch_ts": "2021-10-15T21:07:37.840903Z",
      "epoch": 1634332057840,
      "etype": "line",
      "log_name": "logtype2",
      "severity_num": 6,
      "event_uuid": "00000000-0000-0000-0000-000000000009",
      "event_text": "[2021-10-15 21:07:37.840903] INFO: Now try the filter bar (above), and highlight bar (below)",
      "metadata_id": "ze_deployment_name=sample,zid_container_name=logtype2,zid_host=host2,zid_log=logtype2",
      "metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host2.fqdm.com",
      "local_timestamp": "2021-10-15T21:07:37.840903Z",
      "local_offset": "+0000",
      "ze_xid": "",
      "event_context_level": 1,
      "host": "host2",
      "severity": "Informational",
      "app": null,
      "container_name": "logtype2",
      "namespace_name": null,
      "event_meta_data": {
        "ze_deployment_name": "sample",
        "container_name": "logtype1-359f02372109b4222880d1c7932b717f",
        "hostname": "host2.fqdm.com"
      }
    },
    {
      "root_cause": false,
      "hallmark": false,
      "epoch_ts": "2021-10-15T21:07:45.851986Z",
      "epoch": 1634332065851,
      "etype": "line",
      "log_name": "logtype1",
      "severity_num": 2,
      "event_uuid": "00000000-0000-0000-0000-000000000010",
      "event_text": "[2021-10-15 21:07:45.851986] CRITICAL: If you do not see enough detail in the Core events, try these things:",
      "metadata_id": "ze_deployment_name=sample,zid_container_name=logtype1,zid_host=host1,zid_log=logtype1",
      "metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host1.fqdm.com",
      "local_timestamp": "2021-10-15T21:07:45.851986Z",
      "local_offset": "+0000",
      "ze_xid": "",
      "event_context_level": 1,
      "host": "host1",
      "severity": "Critical",
      "app": null,
      "container_name": "logtype1",
      "namespace_name": null,
      "event_meta_data": {
        "ze_deployment_name": "sample",
        "container_name": "logtype1-359f02372109b4222880d1c7932b717f",
        "hostname": "host1.fqdm.com"
      }
    },
    {
      "root_cause": false,
      "hallmark": false,
      "epoch_ts": "2021-10-15T21:07:53.858345Z",
      "epoch": 1634332073858,
      "etype": "line",
      "log_name": "logtype1",
      "severity_num": 6,
      "event_uuid": "00000000-0000-0000-0000-000000000011",
      "event_text": "[2021-10-15 21:07:53.858345] INFO: Click the Peek button (at the end of each log line) to see all available lines from just this log stream",
      "metadata_id": "ze_deployment_name=sample,zid_container_name=logtype1,zid_host=host2,zid_log=logtype1",
      "metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host2.fqdm.com",
      "local_timestamp": "2021-10-15T21:07:53.858345Z",
      "local_offset": "+0000",
      "ze_xid": "",
      "event_context_level": 1,
      "host": "host2",
      "severity": "Informational",
      "app": null,
      "container_name": "logtype1",
      "namespace_name": null,
      "event_meta_data": {
        "ze_deployment_name": "sample",
        "container_name": "logtype1-359f02372109b4222880d1c7932b717f",
        "hostname": "host2.fqdm.com"
      }
    },
    {
      "root_cause": false,
      "hallmark": false,
      "epoch_ts": "2021-10-15T21:08:01.864572Z",
      "epoch": 1634332081864,
      "etype": "line",
      "log_name": "logtype2",
      "severity_num": 6,
      "event_uuid": "00000000-0000-0000-0000-000000000012",
      "event_text": "[2021-10-15 21:08:01.864572] INFO: Or zoom out beyond the Core events by clicking a Zoom level in Related Events (at the top)",
      "metadata_id": "ze_deployment_name=sample,zid_container_name=logtype2,zid_host=host2,zid_log=logtype2",
      "metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host2.fqdm.com",
      "local_timestamp": "2021-10-15T21:08:01.864572Z",
      "local_offset": "+0000",
      "ze_xid": "",
      "event_context_level": 1,
      "host": "host2",
      "severity": "Informational",
      "app": null,
      "container_name": "logtype2",
      "namespace_name": null,
      "event_meta_data": {
        "ze_deployment_name": "sample",
        "container_name": "logtype1-359f02372109b4222880d1c7932b717f",
        "hostname": "host2.fqdm.com"
      }
    },
    {
      "root_cause": false,
      "hallmark": false,
      "epoch_ts": "2021-10-15T21:08:09.871442Z",
      "epoch": 1634332089871,
      "etype": "line",
      "log_name": "logtype2",
      "severity_num": 6,
      "event_uuid": "00000000-0000-0000-0000-000000000013",
      "event_text": "[2021-10-15 21:08:09.871442] INFO: Zooming is useful when the Core events do not contain enough information",
      "metadata_id": "ze_deployment_name=sample,zid_container_name=logtype2,zid_host=host1,zid_log=logtype2",
      "metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host1.fqdm.com",
      "local_timestamp": "2021-10-15T21:08:09.871442Z",
      "local_offset": "+0000",
      "ze_xid": "",
      "event_context_level": 1,
      "host": "host1",
      "severity": "Informational",
      "app": null,
      "container_name": "logtype2",
      "namespace_name": null,
      "event_meta_data": {
        "ze_deployment_name": "sample",
        "container_name": "logtype1-359f02372109b4222880d1c7932b717f",
        "hostname": "host1.fqdm.com"
      }
    },
    {
      "root_cause": false,
      "hallmark": false,
      "epoch_ts": "2021-10-15T21:08:17.878258Z",
      "epoch": 1634332097878,
      "etype": "line",
      "log_name": "logtype2",
      "severity_num": 6,
      "event_uuid": "00000000-0000-0000-0000-000000000014",
      "event_text": "[2021-10-15 21:08:17.878258] INFO: Enjoy using Skylar Automated RCA and let us know if you have any questions!",
      "metadata_id": "ze_deployment_name=sample,zid_container_name=logtype2,zid_host=host1,zid_log=logtype2",
      "metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host1.fqdm.com",
      "local_timestamp": "2021-10-15T21:08:17.878258Z",
      "local_offset": "+0000",
      "ze_xid": "",
      "event_context_level": 1,
      "host": "host1",
      "severity": "Informational",
      "app": null,
      "container_name": "logtype2",
      "namespace_name": null,
      "event_meta_data": {
        "ze_deployment_name": "sample",
        "container_name": "logtype1-359f02372109b4222880d1c7932b717f",
        "hostname": "host1.fqdm.com"
      }
    }
  ],
  "key_events": [
    {
      "root_cause": true,
      "hallmark": true,
      "epoch_ts": "2021-10-15T21:07:13.82029Z",
      "epoch": 1634332033820,
      "etype": "line",
      "log_name": "logtype1",
      "severity_num": 6,
      "event_uuid": "00000000-0000-0000-0000-000000000006",
      "event_text": "[2021-10-15 21:07:13.82029] INFO: This is the first of two events that are used to characterize the report in the list view",
      "metadata_id": "ze_deployment_name=sample,zid_container_name=logtype1,zid_host=host1,zid_log=logtype1",
      "metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host1.fqdm.com",
      "local_timestamp": "2021-10-15T21:07:13.82029Z",
      "local_offset": "+0000",
      "ze_xid": "",
      "event_context_level": 0,
      "host": "host1",
      "severity": "Informational",
      "app": null,
      "container_name": "logtype1",
      "namespace_name": null,
      "event_meta_data": {
        "ze_deployment_name": "sample",
        "container_name": "logtype1-359f02372109b4222880d1c7932b717f",
        "hostname": "host1.fqdm.com"
      }
    },
    {
      "root_cause": false,
      "hallmark": true,
      "epoch_ts": "2021-10-15T21:07:29.833156Z",
      "epoch": 1634332049833,
      "etype": "line",
      "log_name": "logtype2",
      "severity_num": 2,
      "event_uuid": "00000000-0000-0000-0000-000000000008",
      "event_text": "[2021-10-15 21:07:29.833156] CRITICAL: This is the second of two events that are used to characterize the report in the list view",
      "metadata_id": "ze_deployment_name=sample,zid_container_name=logtype2,zid_host=host1,zid_log=logtype2",
      "metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host1.fqdm.com",
      "local_timestamp": "2021-10-15T21:07:29.833156Z",
      "local_offset": "+0000",
      "ze_xid": "",
      "event_context_level": 0,
      "host": "host1",
      "severity": "Critical",
      "app": null,
      "container_name": "logtype2",
      "namespace_name": null,
      "event_meta_data": {
        "ze_deployment_name": "sample",
        "container_name": "logtype1-359f02372109b4222880d1c7932b717f",
        "hostname": "host1.fqdm.com"
      }
    }
  ],
  "interesting_events": [
    {
      "root_cause": false,
      "hallmark": false,
      "epoch_ts": "2021-10-15T21:06:49.790742Z",
      "epoch": 1634332009790,
      "etype": "line",
      "log_name": "logtype1",
      "severity_num": 6,
      "event_uuid": "00000000-0000-0000-0000-000000000003",
      "event_text": "[2021-10-15 21:06:49.790742] INFO: This is a sample root cause report",
      "metadata_id": "ze_deployment_name=sample,zid_container_name=logtype1,zid_host=host2,zid_log=logtype1",
      "metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host2.fqdm.com",
      "local_timestamp": "2021-10-15T21:06:49.790742Z",
      "local_offset": "+0000",
      "ze_xid": "",
      "event_context_level": 1,
      "host": "host2",
      "severity": "Informational",
      "app": null,
      "container_name": "logtype1",
      "namespace_name": null,
      "event_meta_data": {
        "ze_deployment_name": "sample",
        "container_name": "logtype1-359f02372109b4222880d1c7932b717f",
        "hostname": "host2.fqdm.com"
      }
    },
    {
      "root_cause": false,
      "hallmark": false,
      "epoch_ts": "2021-10-15T21:06:57.7982Z",
      "epoch": 1634332017798,
      "etype": "line",
      "log_name": "logtype2",
      "severity_num": 6,
      "event_uuid": "00000000-0000-0000-0000-000000000004",
      "event_text": "[2021-10-15 21:06:57.7982] INFO: Real Root Cause Reports typically have 5-20 \"Core\" log events",
      "metadata_id": "ze_deployment_name=sample,zid_container_name=logtype2,zid_host=host2,zid_log=logtype2",
      "metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host2.fqdm.com",
      "local_timestamp": "2021-10-15T21:06:57.7982Z",
      "local_offset": "+0000",
      "ze_xid": "",
      "event_context_level": 1,
      "host": "host2",
      "severity": "Informational",
      "app": null,
      "container_name": "logtype2",
      "namespace_name": null,
      "event_meta_data": {
        "ze_deployment_name": "sample",
        "container_name": "logtype1-359f02372109b4222880d1c7932b717f",
        "hostname": "host2.fqdm.com"
      }
    },
    {
      "root_cause": false,
      "hallmark": false,
      "epoch_ts": "2021-10-15T21:07:05.805105Z",
      "epoch": 1634332025805,
      "etype": "line",
      "log_name": "logtype2",
      "severity_num": 6,
      "event_uuid": "00000000-0000-0000-0000-000000000005",
      "event_text": "[2021-10-15 21:07:05.805105] INFO: Core events consist of mostly \"rare\" and high-severity events that are correlated across multiple logs",
      "metadata_id": "ze_deployment_name=sample,zid_container_name=logtype2,zid_host=host2,zid_log=logtype2",
      "metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host2.fqdm.com",
      "local_timestamp": "2021-10-15T21:07:05.805105Z",
      "local_offset": "+0000",
      "ze_xid": "",
      "event_context_level": 1,
      "host": "host2",
      "severity": "Informational",
      "app": null,
      "container_name": "logtype2",
      "namespace_name": null,
      "event_meta_data": {
        "ze_deployment_name": "sample",
        "container_name": "logtype1-359f02372109b4222880d1c7932b717f",
        "hostname": "host2.fqdm.com"
      }
    },
    {
      "root_cause": false,
      "hallmark": false,
      "epoch_ts": "2021-10-15T21:07:21.826703Z",
      "epoch": 1634332041826,
      "etype": "line",
      "log_name": "logtype1",
      "severity_num": 3,
      "event_uuid": "00000000-0000-0000-0000-000000000007",
      "event_text": "[2021-10-15 21:07:21.826703] ERROR: Did you notice this event has error severity?",
      "metadata_id": "ze_deployment_name=sample,zid_container_name=logtype1,zid_host=host1,zid_log=logtype1",
      "metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host1.fqdm.com",
      "local_timestamp": "2021-10-15T21:07:21.826703Z",
      "local_offset": "+0000",
      "ze_xid": "",
      "event_context_level": 1,
      "host": "host1",
      "severity": "Error",
      "app": null,
      "container_name": "logtype1",
      "namespace_name": null,
      "event_meta_data": {
        "ze_deployment_name": "sample",
        "container_name": "logtype1-359f02372109b4222880d1c7932b717f",
        "hostname": "host1.fqdm.com"
      }
    },
    {
      "root_cause": false,
      "hallmark": false,
      "epoch_ts": "2021-10-15T21:07:37.840903Z",
      "epoch": 1634332057840,
      "etype": "line",
      "log_name": "logtype2",
      "severity_num": 6,
      "event_uuid": "00000000-0000-0000-0000-000000000009",
      "event_text": "[2021-10-15 21:07:37.840903] INFO: Now try the filter bar (above), and highlight bar (below)",
      "metadata_id": "ze_deployment_name=sample,zid_container_name=logtype2,zid_host=host2,zid_log=logtype2",
      "metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host2.fqdm.com",
      "local_timestamp": "2021-10-15T21:07:37.840903Z",
      "local_offset": "+0000",
      "ze_xid": "",
      "event_context_level": 1,
      "host": "host2",
      "severity": "Informational",
      "app": null,
      "container_name": "logtype2",
      "namespace_name": null,
      "event_meta_data": {
        "ze_deployment_name": "sample",
        "container_name": "logtype1-359f02372109b4222880d1c7932b717f",
        "hostname": "host2.fqdm.com"
      }
    },
    {
      "root_cause": false,
      "hallmark": false,
      "epoch_ts": "2021-10-15T21:07:45.851986Z",
      "epoch": 1634332065851,
      "etype": "line",
      "log_name": "logtype1",
      "severity_num": 2,
      "event_uuid": "00000000-0000-0000-0000-000000000010",
      "event_text": "[2021-10-15 21:07:45.851986] CRITICAL: If you do not see enough detail in the Core events, try these things:",
      "metadata_id": "ze_deployment_name=sample,zid_container_name=logtype1,zid_host=host1,zid_log=logtype1",
      "metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host1.fqdm.com",
      "local_timestamp": "2021-10-15T21:07:45.851986Z",
      "local_offset": "+0000",
      "ze_xid": "",
      "event_context_level": 1,
      "host": "host1",
      "severity": "Critical",
      "app": null,
      "container_name": "logtype1",
      "namespace_name": null,
      "event_meta_data": {
        "ze_deployment_name": "sample",
        "container_name": "logtype1-359f02372109b4222880d1c7932b717f",
        "hostname": "host1.fqdm.com"
      }
    },
    {
      "root_cause": false,
      "hallmark": false,
      "epoch_ts": "2021-10-15T21:07:53.858345Z",
      "epoch": 1634332073858,
      "etype": "line",
      "log_name": "logtype1",
      "severity_num": 6,
      "event_uuid": "00000000-0000-0000-0000-000000000011",
      "event_text": "[2021-10-15 21:07:53.858345] INFO: Click the Peek button (at the end of each log line) to see all available lines from just this log stream",
      "metadata_id": "ze_deployment_name=sample,zid_container_name=logtype1,zid_host=host2,zid_log=logtype1",
      "metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host2.fqdm.com",
      "local_timestamp": "2021-10-15T21:07:53.858345Z",
      "local_offset": "+0000",
      "ze_xid": "",
      "event_context_level": 1,
      "host": "host2",
      "severity": "Informational",
      "app": null,
      "container_name": "logtype1",
      "namespace_name": null,
      "event_meta_data": {
        "ze_deployment_name": "sample",
        "container_name": "logtype1-359f02372109b4222880d1c7932b717f",
        "hostname": "host2.fqdm.com"
      }
    },
    {
      "root_cause": false,
      "hallmark": false,
      "epoch_ts": "2021-10-15T21:08:01.864572Z",
      "epoch": 1634332081864,
      "etype": "line",
      "log_name": "logtype2",
      "severity_num": 6,
      "event_uuid": "00000000-0000-0000-0000-000000000012",
      "event_text": "[2021-10-15 21:08:01.864572] INFO: Or zoom out beyond the Core events by clicking a Zoom level in Related Events (at the top)",
      "metadata_id": "ze_deployment_name=sample,zid_container_name=logtype2,zid_host=host2,zid_log=logtype2",
      "metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host2.fqdm.com",
      "local_timestamp": "2021-10-15T21:08:01.864572Z",
      "local_offset": "+0000",
      "ze_xid": "",
      "event_context_level": 1,
      "host": "host2",
      "severity": "Informational",
      "app": null,
      "container_name": "logtype2",
      "namespace_name": null,
      "event_meta_data": {
        "ze_deployment_name": "sample",
        "container_name": "logtype1-359f02372109b4222880d1c7932b717f",
        "hostname": "host2.fqdm.com"
      }
    },
    {
      "root_cause": false,
      "hallmark": false,
      "epoch_ts": "2021-10-15T21:08:09.871442Z",
      "epoch": 1634332089871,
      "etype": "line",
      "log_name": "logtype2",
      "severity_num": 6,
      "event_uuid": "00000000-0000-0000-0000-000000000013",
      "event_text": "[2021-10-15 21:08:09.871442] INFO: Zooming is useful when the Core events do not contain enough information",
      "metadata_id": "ze_deployment_name=sample,zid_container_name=logtype2,zid_host=host1,zid_log=logtype2",
      "metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host1.fqdm.com",
      "local_timestamp": "2021-10-15T21:08:09.871442Z",
      "local_offset": "+0000",
      "ze_xid": "",
      "event_context_level": 1,
      "host": "host1",
      "severity": "Informational",
      "app": null,
      "container_name": "logtype2",
      "namespace_name": null,
      "event_meta_data": {
        "ze_deployment_name": "sample",
        "container_name": "logtype1-359f02372109b4222880d1c7932b717f",
        "hostname": "host1.fqdm.com"
      }
    },
    {
      "root_cause": false,
      "hallmark": false,
      "epoch_ts": "2021-10-15T21:08:17.878258Z",
      "epoch": 1634332097878,
      "etype": "line",
      "log_name": "logtype2",
      "severity_num": 6,
      "event_uuid": "00000000-0000-0000-0000-000000000014",
      "event_text": "[2021-10-15 21:08:17.878258] INFO: Enjoy using Skylar Automated RCA and let us know if you have any questions!",
      "metadata_id": "ze_deployment_name=sample,zid_container_name=logtype2,zid_host=host1,zid_log=logtype2",
      "metadata_cfg": "ze_deployment_name=sample,container_name=logtype1-359f02372109b4222880d1c7932b717f,hostname=host1.fqdm.com",
      "local_timestamp": "2021-10-15T21:08:17.878258Z",
      "local_offset": "+0000",
      "ze_xid": "",
      "event_context_level": 1,
      "host": "host1",
      "severity": "Informational",
      "app": null,
      "container_name": "logtype2",
      "namespace_name": null,
      "event_meta_data": {
        "ze_deployment_name": "sample",
        "container_name": "logtype1-359f02372109b4222880d1c7932b717f",
        "hostname": "host1.fqdm.com"
      }
    }
  ],
  "nearby_events": [
    {
      "root_cause": false,
      "hallmark": false,
      "epoch_ts": "2021-10-15T21:06:25.77145Z",
      "epoch": 1634331985771,
      "etype": "line",
      "log_name": "zoom_log",
      "severity_num": 6,
      "event_uuid": "00000000-0000-0000-0000-000000000000",
      "event_text": "[2021-10-15 21:06:25.77145] INFO: You are seeing this event because you zoomed into Related Events level 3 (or because you Peeked)",
      "metadata_id": "ze_deployment_name=sample,zid_container_name=zoom_log,zid_host=host3,zid_log=zoom_log",
      "metadata_cfg": "ze_deployment_name=sample,container_name=zoom_log-a32e129fccd92e3ab19e749655f152a7,hostname=host3.fqdm.com",
      "local_timestamp": "2021-10-15T21:06:25.77145Z",
      "local_offset": "+0000",
      "ze_xid": "",
      "event_context_level": 5,
      "host": "host3",
      "severity": "Informational",
      "app": null,
      "container_name": "zoom_log",
      "namespace_name": null,
      "event_meta_data": {
        "ze_deployment_name": "sample",
        "container_name": "zoom_log-a32e129fccd92e3ab19e749655f152a7",
        "hostname": "host3.fqdm.com"
      }
    },
    {
      "root_cause": false,
      "hallmark": false,
      "epoch_ts": "2021-10-15T21:06:33.778395Z",
      "epoch": 1634331993778,
      "etype": "line",
      "log_name": "zoom_log",
      "severity_num": 6,
      "event_uuid": "00000000-0000-0000-0000-000000000001",
      "event_text": "[2021-10-15 21:06:33.778395] INFO: You are seeing this event because you zoomed into Related Events level 2 (or because you Peeked)",
      "metadata_id": "ze_deployment_name=sample,zid_container_name=zoom_log,zid_host=host3,zid_log=zoom_log",
      "metadata_cfg": "ze_deployment_name=sample,container_name=zoom_log-a32e129fccd92e3ab19e749655f152a7,hostname=host3.fqdm.com",
      "local_timestamp": "2021-10-15T21:06:33.778395Z",
      "local_offset": "+0000",
      "ze_xid": "",
      "event_context_level": 4,
      "host": "host3",
      "severity": "Informational",
      "app": null,
      "container_name": "zoom_log",
      "namespace_name": null,
      "event_meta_data": {
        "ze_deployment_name": "sample",
        "container_name": "zoom_log-a32e129fccd92e3ab19e749655f152a7",
        "hostname": "host3.fqdm.com"
      }
    },
    {
      "root_cause": false,
      "hallmark": false,
      "epoch_ts": "2021-10-15T21:06:41.784659Z",
      "epoch": 1634332001784,
      "etype": "line",
      "log_name": "zoom_log",
      "severity_num": 6,
      "event_uuid": "00000000-0000-0000-0000-000000000002",
      "event_text": "[2021-10-15 21:06:41.784659] INFO: You are seeing this event because you zoomed into Related Events level 1 (or because you Peeked)",
      "metadata_id": "ze_deployment_name=sample,zid_container_name=zoom_log,zid_host=host3,zid_log=zoom_log",
      "metadata_cfg": "ze_deployment_name=sample,container_name=zoom_log-a32e129fccd92e3ab19e749655f152a7,hostname=host3.fqdm.com",
      "local_timestamp": "2021-10-15T21:06:41.784659Z",
      "local_offset": "+0000",
      "ze_xid": "",
      "event_context_level": 3,
      "host": "host3",
      "severity": "Informational",
      "app": null,
      "container_name": "zoom_log",
      "namespace_name": null,
      "event_meta_data": {
        "ze_deployment_name": "sample",
        "container_name": "zoom_log-a32e129fccd92e3ab19e749655f152a7",
        "hostname": "host3.fqdm.com"
      }
    },
    {
      "root_cause": false,
      "hallmark": false,
      "epoch_ts": "2021-10-15T21:08:25.885936Z",
      "epoch": 1634332105885,
      "etype": "line",
      "log_name": "zoom_log",
      "severity_num": 6,
      "event_uuid": "00000000-0000-0000-0000-000000000015",
      "event_text": "[2021-10-15 21:08:25.885936] INFO: This is the last event in the Related Events level 1 zoom out",
      "metadata_id": "ze_deployment_name=sample,zid_container_name=zoom_log,zid_host=host3,zid_log=zoom_log",
      "metadata_cfg": "ze_deployment_name=sample,container_name=zoom_log-a32e129fccd92e3ab19e749655f152a7,hostname=host3.fqdm.com",
      "local_timestamp": "2021-10-15T21:08:25.885936Z",
      "local_offset": "+0000",
      "ze_xid": "",
      "event_context_level": 3,
      "host": "host3",
      "severity": "Informational",
      "app": null,
      "container_name": "zoom_log",
      "namespace_name": null,
      "event_meta_data": {
        "ze_deployment_name": "sample",
        "container_name": "zoom_log-a32e129fccd92e3ab19e749655f152a7",
        "hostname": "host3.fqdm.com"
      }
    },
    {
      "root_cause": false,
      "hallmark": false,
      "epoch_ts": "2021-10-15T21:08:33.896882Z",
      "epoch": 1634332113896,
      "etype": "line",
      "log_name": "zoom_log",
      "severity_num": 6,
      "event_uuid": "00000000-0000-0000-0000-000000000016",
      "event_text": "[2021-10-15 21:08:33.896882] INFO: This is the last event in the Related Events level 2 zoom out",
      "metadata_id": "ze_deployment_name=sample,zid_container_name=zoom_log,zid_host=host3,zid_log=zoom_log",
      "metadata_cfg": "ze_deployment_name=sample,container_name=zoom_log-a32e129fccd92e3ab19e749655f152a7,hostname=host3.fqdm.com",
      "local_timestamp": "2021-10-15T21:08:33.896882Z",
      "local_offset": "+0000",
      "ze_xid": "",
      "event_context_level": 4,
      "host": "host3",
      "severity": "Informational",
      "app": null,
      "container_name": "zoom_log",
      "namespace_name": null,
      "event_meta_data": {
        "ze_deployment_name": "sample",
        "container_name": "zoom_log-a32e129fccd92e3ab19e749655f152a7",
        "hostname": "host3.fqdm.com"
      }
    },
    {
      "root_cause": false,
      "hallmark": false,
      "epoch_ts": "2021-10-15T21:08:41.903443Z",
      "epoch": 1634332121903,
      "etype": "line",
      "log_name": "zoom_log",
      "severity_num": 6,
      "event_uuid": "00000000-0000-0000-0000-000000000017",
      "event_text": "[2021-10-15 21:08:41.903443] INFO: This is the last event in the Related Events level 3 zoom out",
      "metadata_id": "ze_deployment_name=sample,zid_container_name=zoom_log,zid_host=host3,zid_log=zoom_log",
      "metadata_cfg": "ze_deployment_name=sample,container_name=zoom_log-a32e129fccd92e3ab19e749655f152a7,hostname=host3.fqdm.com",
      "local_timestamp": "2021-10-15T21:08:41.903443Z",
      "local_offset": "+0000",
      "ze_xid": "",
      "event_context_level": 5,
      "host": "host3",
      "severity": "Informational",
      "app": null,
      "container_name": "zoom_log",
      "namespace_name": null,
      "event_meta_data": {
        "ze_deployment_name": "sample",
        "container_name": "zoom_log-a32e129fccd92e3ab19e749655f152a7",
        "hostname": "host3.fqdm.com"
      }
    }
  ]
}