New Relic Dashboard Integrations

Download this manual as a PDF file

Features

  • You can configure Zebrium to automatically add Root Cause (RCA) reports as Events in New Relic. This allows you to see details of root cause on any New Relic dashboard.
  • This integration automatically adds log metrics in New Relic.
  • Each Zebrium RCA report includes a summary, a word cloud, and a set of log events showing symptoms and root cause, plus a link to the full report in the Zebrium user interface.
  • This means faster Mean Time to Resolution (MTTR) and less time manually hunting for root cause.

How It Works

The recommended mode of operation for observability dashboard integrations is to use the Zebrium Auto-Detect mode as an accurate mechanism for explaining the reason something went wrong. In this mode, you continue to use your existing rules, alerts and metrics as the primary source of problem detection. You can then review Zebrium RCA report findings directly in your New Relic Dashboards alongside other metrics to explain the reason behind problems you were alerted on.

The Zebrium Augment mode is useful when you have Health Rule Violation or Anomaly-based Policy triggers defined in AppDynamics, and you want a Root Cause report automatically generated at the time of the alert. In this mode, Zebrium uses a New Relic webhook as a notification channel and will update your Dashboard with Root Cause reports that coincide with the triggering alert so they’re immediately visible to you as you work the issue.

The two modes of operation are independent. You can configure Auto-Detect and/or Augment modes depending on your operational use-case.

Auto-Detect (recommended): Send Root Cause Detections to your New Relic Dashboards

  1. Zebrium continuously monitors all application logs and uses unsupervised machine learning to find anomalous log patterns that indicate a problem. These are automatically turned into Root Cause reports highlighting details of any problems with over 95% accuracy.
  2. Root Cause report summaries are sent to New Relic using the events API and Root Cause details are visible on your New Relic Dashboards.
  3. With a single click on your Dashboard, you can drill down further into the Zebrium user interface to look at correlated logs across your entire application.
  4. Log metrics are also sent to New Relic using the metric API for visualization on your New Relic Dashboards.

For details, see Adding Zebrium Root Cause reports to New Relic Dashboards.

Augment (advanced users): Receive Signals from New Relic Alert Policies

  1. Any New Relic Alert Policy can trigger a webhook request for Root Cause Analysis from Zebrium.
  2. Zebrium finds anomalous log patterns from your application that coincide with the Incident and creates a Root Cause report.
  3. Root Cause report summaries are sent to New Relic using the event API and Root Cause details are visible on your New Relic Dashboards.
  4. If you need to drill down further to look at correlated logs across your entire app, it’s just one click from your Dashboard.

For details, see Augmenting New Relic with Root Cause reports using Alert Policies.

Adding Zebrium Root Cause reports to New Relic Dashboards

Integration Overview

  1. Create an API Key in New Relic.
  2. Create a New Relic Integration in Zebrium using the information from STEP 1.
  3. Add Zebrium Root Cause report events and Log metrics to your New Relic Dashboard.

Integration Details

STEP 1: Create an API Key in New Relic

  1. In the New Relic user interface, go to the Settings drop-down and select API keys.
  2. Click Create a key.
  3. Choose the appropriate account and make a note of the Account ID for use in STEP 2.
  4. Select Ingest - License as the Key type.
  5. Enter a Name for the API key and click Create a key.
  6. Copy and save the Key for use in STEP 2.

STEP 2: Create a New Relic Outbound Integration in Zebrium

  1. In the Zebrium user interface, go to the Integrations & Collectors page (Settings () > Integrations & Collectors).
  2. In the Observability Dashboards section, click the New Relic button.
  3. Click Create a New Integration button. The Create AppDynamics Dashboard dialog appears.
  4. On the General tab, enter an Integration Name for this integration.
  5. In the Deployment drop-down, select a deployment for the integration.
  6. In the Service Group(s) drop-down, select a service group for the integration.
  7. On the Send Detections tab, click Enabled.
  8. Enter the Account ID from STEP 1, above.
  9. Enter the API Key from STEP 1, above.
  10. Click Save.

STEP 3: Add Zebrium Root Cause report Detections and Log Metrics to your New Relic Dashboards

We recommend adding a few charts to your existing New Relic dashboards so that you can see details of Zebrium Root Cause reports alongside your other observability data.

The image below shows a sample New Relic dashboard with the following Zebrium charts:

  • A count of Zebrium Root Cause report Detections.
  • A Zebrium Root Cause Finder chart with a green vertical bar wherever a detection occurs. This is useful when used on other dashboards as you can visually correlate what you are seeing in other data with what Zebrium has automatically detected.
  • A chart showing log anomaly counts.
  • A chart containing a list of Root Cause report summaries.

Sample New Relic dashboard:

Image of a New Relic Zebrium dashboard

The following images show the definitions for each of the charts above.

Count of Zebrium Root Cause report detections:

Image of Zebrium Root Cause Report Detections

Root cause finder showing where detections occur:

Image of a Root cause finder showing where detections occur

Chart of log anomaly counts:

Image of a Chart of log anomaly counts

List of Root Cause report summaries:

Image of a list of root cause report summaries

Important Metric Names

Metric Name Description

zebrium.logs.all.count

Count of all log events received in a one-minute duration (per service_group).

zebrium.logs.anomalies.count

Count of anomaly log events received in a one-minute duration (per service_group).

zebrium.logs.errors.count

Count of error log events received in a one-minute duration (per service_group).

Zebrium Detections Event Payload

Zebrium Detections are sent to New Relic as Custom Events with eventType of NrAiIncidentExternal.

Here is a sample NrAiIncidentExternal payload for a Zebrium detection with descriptions of each field:

    {
      "aggregationTag.ze_deployment": "this is the deployment name",
      "aggregationTag.ze_first_occurrence": "set to true if this is the first occurrence of this type of incident",
      "aggregationTag.ze_inci_id": "00062722-5f20-0000-0000-5190000353ee",
      "aggregationTag.ze_service_group": "this is the zebrium service group name",
      "aggregationTag.ze_significance": "significance of the detection: low, medium or high",
      "deepLinkUrl": "this is the URL in the Zebrium UI for this root cause report",
      "description": "this is usually the NLP summary for the root cause report",
      "entityName": "zebrium_detections",
      "source": "zebrium_detections",
      "state": "trigger",
      "timestamp": 1651647986000,
      "title": "Zebrium Detected Root Cause report",
      "version": 1
    }

Augmenting New Relic with Root Cause Reports using Alert Policies

Integration Overview

  1. Create an API Key in New Relic.
  2. Create a New Relic Inbound Integration in Zebrium using the information from STEP 1.
  3. Create a Webhook Notification Channel in New Relic using the information from STEP 2.
  4. Add Webhook notifications to your Alert Policies in New Relic.
  5. Add Zebrium Root Cause reports to your New Relic Dashboard.

Integration Details

STEP 1: Create an API Key in New Relic

  1. In the New Relic user interface, go to the Settings drop-down and select API keys.
  2. Click Create a key.
  3. Choose the appropriate Account and make a note of the Account ID for use in STEP 2.
  4. Select Ingest - License as the Key type.
  5. Enter a Name for the API key and click Create a key.
  6. Copy and save the Key for use in STEP 2.

STEP 2: Create a New Relic Inbound Integration in Zebrium

  1. From the User menu area, click on the Settings menu () .
  2. Select Inbound Integrations.
  3. Click the + Create Inbound Integration button.
  4. Select New Relic as the Inbound Integration Type and click Create.
  5. Enter a Name for this integration.
  6. Select the Deployment for the integration.
  7. Select the Service Group(s) for the integration.
  8. Enter the Account ID from STEP 1, above.
  9. Enter the API Key from STEP 1, above.
  10. Click Create.
  11. Copy the Webhook that was just created and save it for use in STEP 3.

STEP 3: Create a Webhook Notification Channel in New Relic

  1. In the New Relic user interface, click the Alerts & AI tab.
  2. From the Main Navigation panel, click Channels.
  3. Click the + New notification channel button.
  4. Select Webhook as the channel type.
  5. Enter a Channel name for this Webhook.
  6. Enter the Base URL generated by Zebrium in STEP 2.
  7. Leave the Basic Auth, Custom Headers, and Use Custom Payload fields empty.
  8. Click Create channel.

STEP 4: Add Webhook Notifications to your Alert Policies in New Relic

  1. In the New Relic user interface, click the Alerts & AI tab.
  2. From the Main Navigation panel, click on Alert conditions (Policies).
  3. Click on any Alert Policy you wish to trigger a Root Cause report request.
  4. Click the Notification channels tab.
  5. Click on Add notification channels.
  6. Click on Webhook.
  7. Select the Webhook notification channel created in STEP 3.