The Installation Process for Skylar AI

The installation and configuration of Skylar AI and Skylar Advisor uses Harbor for the registry and Helm for deployment. The Skylar AI platform is deployed as a single Helm umbrella chart containing over 20 microservices, databases, and supporting infrastructure components.

This chapter explains how to get access to the registry for the installation, how to run the installation, and how to validate your installation.

Registry Access Setup

The information in this section is relevant for both Skylar Analytics and Skylar Advisor deployments.

Step 0: Pre-flight Validation

Perform all of the following validations before continuing to Step 1:

Verify the Kubernetes version; the minimum is version 1.35:

kubectl version
Verify the container runtime version; ContainerD 2.1 or later is required:

containerd --version
Verify that the GPU Operator is running:

kubectl get pods -n gpu-operator
Verify that GPUs are visible to Kubernetes:

kubectl describe nodes | grep -A5 'nvidia.com/gpu'
Verify that storage classes are available:

kubectl get storageclass
Test the registry login:

helm registry login registry.scilo.tools
Verify that the ingress controller is running (adjust the namespace for your controller):

kubectl get pods -n ingress-nginx
When all of the checks have passed, continue to Step 1.

Step 1: Obtain Registry Credentials

Navigate to the registry at https://registry.scilo.tools/. The Harbor landing page appears.
Click Login via OIDC Provider.
Click Customer Login.
Enter your Salesforce credentials provided by ScienceLogic.
Click your name in the top right corner and select User Profile.
Click the copy icon in the CLI Secret field for use in the next steps.

Step 2: Sign into Helm Registry (Required)

You must authenticate with the Helm registry before proceeding with any chart operations. You authenticate with the registry using your Harbor CLI credentials (from step 6 in the previous procedure):

helm registry login registry.scilo.tools --username <your-harbor-username>

Step 3: Install the OpenTelemetry Operator

Install the OpenTelemetry Operator with custom image:

# Install OpenTelemetry Operator with custom image
helm registry login 
helm install opentelemetry-operator oci://registry.scilo.tools/skylar/opentelemetry-helm-charts/opentelemetry-operator \
  --version 0.114.0 \
  --namespace opentelemetry-operator \
  --create-namespace \
  --set "manager.collectorImage.repository=registry.scilo.tools/skylar/sl-otelcol" \
  --set "manager.collectorImage.tag=0.16" \
  --set admissionWebhooks.certManager.enabled=false \
  --set admissionWebhooks.autoGenerateCert.enabled=true

If you are using Cert-Manager for your certificate management, you can remove the following lines:

--set admissionWebhooks.certManager.enabled=false \
--set admissionWebhooks.autoGenerateCert.enabled=true

Step 4: Download the Skylar AI Chart

Use Helm to pull the Skylar AI Chart.yaml file from the registry, using the registry login from Step 2:

helm pull oci://registry.scilo.tools/skylar/skylar-charts

Decompress the charts:

tar -xvf skylar-charts-x.x.x.tgz

Scaling Profiles

Choose the appropriate scaling profile based on your environment. These scaling profiles can be located within the downloaded Helm chart inside the sizing directory. Please work with your ScienceLogic Sales Engineer to understand which scaling profile you will need to deploy. Most Proof of Concepts (PoC) systems can be deployed with a small profile.

This document will use small.yaml in the following steps, though larger deployments will need other files such as medium.yaml or large.yaml.

Availability Profiles

Starting with version 2.4.0, the Skylar AI platform is deployed by default in a High Availability (HA) pattern that requires each replica of a HA service to be scheduled on a distinct worker node and affinity zone (hard HA). As a result, the cluster must have at least three worker nodes.

If you do not want to use the default availability settings, you can choose an availability profile from the availability directory.

Use no-ha.yaml to deploy the bare minimum number of replicas for eligible services. This option reduces the resources required by the cluster in exchange for availability guarantees.
Use soft-ha.yaml to relax hard affinity zone (AZ) and worker node scheduling requirement to preferred for most services. HA services will attempt to distribute replicas across worker nodes and affinity zones, but will allow double scheduling if necessary. Soft HA requires at least two worker nodes, as the OTel Collector must schedule its two replicas on separate worker nodes, guaranteeing telemetry can be received from tenants if a worker node fails.

The following table provides additional details about the different types of High Availability (HA):

Profile	File	Minimum Nodes	Minimum Affinity Zones *	Replica Scheduling	Affinity	Best for
Hard HA (default)	N/A	3	3	Each replica on distinct node and AZ (required)	Hard constraints	Production, fault tolerance priority
Soft HA	soft-ha.yaml	2	2	Distributed across nodes and AZs, allows double scheduling if needed	Preferred	Balanced HA & flexibility
No HA	no-ha.yaml	1	1	Minimal replicas, no distribution	None	Dev/test, resource-constrained clusters

* Minimum Affinity Zones applies only to AWS deployments. On-premises deployments should ensure sufficient physical and logical separation across theinfrastructure.

Installation Process

Unless noted otherwise, the information in this section is relevant for both Skylar Analytics and Skylar Advisor deployments.

Step 1: Environment Configuration

You will need to prepare the Helm override files with environment-specific details. Inside the decompressed skylar-charts you will find an example directory.

If you are deploying Skylar Analytics, populate the examples/override-analytics.yaml by creating a copy locally:

cp skylar-charts/examples/override-analytics.yaml .

cp skylar-charts/sizing/analytics-small.yaml .
Open the override-analytics.yaml file and add the values specific to the Kubernetes deployment. The file includes comments with helpful guidelines.

vi override-analytics.yaml
If you are using a proxy for egress Internet traffic, be sure to un-comment the sections that set the HTTP_PROXY/HTTPS_PROXY/NO_PROXY environment variables in the override-analytics.yaml file.
If you are also deploying Skylar Advisor with Skylar Analytics, you will need to complete the examples/override-advisor.yaml file. For Skylar Advisor you will need a fourth override that specifies the GPU-specific tolerations and resources. You will need to work directly with your sales engineer to come up with this configuration. You can find examples of this file inside the sizing directory, such as sizing/l40s-rtx6000-combined.yaml or sizing/h100-resources.yaml:

cp skylar-charts/examples/override-advisor.yaml .

cp skylar-charts/sizing/l40s-rtx6000-combined.yaml .
Finally, you need to choose a High Availability (HA) profile.

If you are using the default HA configuration explained in Availability Profiles, then you do not need to do anything else.
If you want to choose soft-ha.yaml or no-ha.yaml, you need to apply the correct file. Depending on which configuration you want to use, run one of the following commands to copy the correct file out of the skylar-helm-charts into your local directory:

cp skylar-charts/availability/no-ha.yaml .

or

cp skylar-charts/availability/soft-ha.yaml .

Ingress Controller-Specific Annotations

Inside the override-analytics.yaml file, you will need to add annotations specific to your ingress controller. Skylar AI recommends the following ingress annotations. Each controller will be different, so the ingress nginx annotations below are only an example. If you are using Traefik or some other ingress controller, you will need to work with a ScienceLogic Sales Engineer to understand the comparable annotations.

Also, in the global section, set the protocol to https, as using HTTP will result in a redirect loop.

#ingress-nginx

global:
  ingress:
    className: external
    annotations:
      nginx.ingress.kubernetes.io/proxy-body-size: 1024m
      nginx.ingress.kubernetes.io/proxy-buffer-size: 512k
      nginx.ingress.kubernetes.io/proxy-buffering: "on"
      nginx.ingress.kubernetes.io/proxy-buffers-number: "4"
      nginx.ingress.kubernetes.io/proxy-max-temp-file-size: 1024m
      nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
    protocol: 'https'
    hostname: 'skylar.<yourdomain>.com'

skylar-auth:
  ui:
    ingress:
      className: external
      sl1embed:
        annotations:
          nginx.ingress.kubernetes.io/rewrite-target: "/skylar-auth/sl1embed"
          nginx.ingress.kubernetes.io/configuration-snippet: |
            more_clear_headers "Content-Security-Policy";
            more_set_headers "Content-Security-Policy: frame-ancestors: *;"; 


 #If deploying advisor add in the below
 skylar-advisor-ui:
  ingress:
    annotations:
      # enables launching content residing outside our domain (eg: storylane)
      nginx.ingress.kubernetes.io/configuration-snippet: 'more_clear_headers "Content-Security-Policy"; more_set_headers "Content-Security-Policy: frame-ancestors: *;";'

High Availability Settings

Skylar AI version 2.4.0 improved the stability of Skylar AI and allowed for zero-downtime upgrades by deploying Postgres and PgPool in High Availability (HA) mode by default, each with three replicas. Please note that you will need to account for the increase in resources required in Kubernetes before you upgrade to this release, as this update could triple the size of the Postgres and PgPool footprint, depending on the current deployment.

If you do not want to use HA, you will need to turn it off in an override.yaml file supplied to the Skylar Helm chart at deploy time, with the following values:

postgresql-ha:
  postgresql:
    replicaCount: 1
    pdb:
      create: false
  pgpool:
    replicaCount: 1
    pdb:
      create: false

Step 2: Deploy the Skylar AI Platform

Run the following steps to deploy the Skylar AI platform with the scaling profile and customer overrides:

helm upgrade --install skylar-production \
  oci://registry.scilo.tools/skylar/skylar-charts \
  --namespace skylar-production \
  --create-namespace \
  --values analytics-small.yaml \
  --values override-analytics.yaml

If you are enabling Skylar Advisor, add the following lines:

--value override-advisor.yaml
--value l40s-rtx6000-combined.yaml

The second .yaml file name above is just an example. You can name this file based on the GPU being deployed.

If you are enabling soft or no High Availability (HA) options, add the following line:

--value no-ha.yaml

When running the Helm install command, you can open a second and third terminal to monitor pod creation and events in parallel while the Helm install runs:

Open a second terminal, full pod state, that refreshes every two seconds:

watch kubectl get pods -n skylar-production

The watch command is preferred over kubectl get pods -w because -w only shows incremental changes, while watch shows the full current state.
Open a third terminal and watch for error events:

kubectl get events -n skylar-production --sort-by='.lastTimestamp' -w

Step 3: Configure TLS Certificate (Optional)

You can skip this step if you have automated certificate management (such as cert-manager) or deployment processes that handle TLS configuration.

Create a TLS secret with your provided certificate and private key:

kubectl create secret tls skylar-tls-secret \
  --cert=path/to/your/certificate.crt \
  --key=path/to/your/private.key \
  --namespace=skylar-production

Validation and Access

The information in this section is relevant for both Skylar Analytics and Skylar Advisor deployments.

Verify Deployment Status

Run the following commands to verify your deployment was successful:

Check to see if all pods are running:

kubectl get pods -n skylar-production
Verify the ingress configuration:

kubectl get ingress -n skylar-production
Check to see if the TLS certificate is properly configured:

kubectl describe ingress -n skylar-production
Check to see if image pull secrets are working:

kubectl describe pod <any-skylar-pod> -n skylar-production | grep -A5 "Events:"

Access the Platform

To access the Skylar AI platform for the first time:

Verify DNS Resolution:

nslookup skylar.<yourdomain>.com
Navigate to the Skylar AI user interface using your provided FQDN, such as https://skylar.<yourdomain>.com.

Skylar AI requires HTTPS for access, and using HTTP will result in a redirect loop. Also, you will need to accept the self-signed certificate warning if you need to configure TLS.
Log in for the first time with the default email of skylar@sciencelogic.com.
Set a password for your first login. The password must be at least 15 characters, with at least one special character required.
To set up a new Skylar AI account, see Provisioning a New Account.
For more information about other Skylar AI settings, see Configuring Skylar AI System Settings.

Please note that multi-factor authentication (MFA) is enabled by default when you first log in. Complete the following steps if you need to disable MFA (for test or development environments, for example):

To disable MFA at the account level, go to Skylar Settings, navigate to the Authentication page ( > Account Settings > Account Access > Authentication), and click the Not Required button for MFA Status.
To reset MFA at the user level, navigate to the Users page ( > Account Settings > Account Access > Users). Click the Actions icon () and select Reset MFA.

If your first login results in a timeout loop after password change:

Press Ctrl+Shift+Delete and clear the browser cache and cookies.
If the issue persists, try a different browser.
Wait 2-3 minutes and retry the login.

The timeout issues might differ between NGINX and Traefik ingress controllers. Report occurrences to ScienceLogic support for investigation.

Monitoring the Integration (Optional)

If you have an existing Prometheus setup, you can configure it to scrape metrics from Skylar AI services.

In the override.yaml file, enable the metrics endpoints:

global:
  enableMonitoring: true

This setting exposes Prometheus metrics endpoints on Skylar AI services that can be scraped by your existing Prometheus deployment. Configure your Prometheus to discover and scrape these endpoints based on your service discovery method, such as Kubernetes service discovery or static configurations.

You can ingest your scraped metrics from Prometheus into Skylar One leveraging the "SL1 Prometheus" PowerPack.