Skylar AI On-Premises Installation Guide

This chapter details the Kubernetes cluster requirements for Skylar Analytics and Skylar Advisor. It also includes requirements for only Skylar Analytics if you are currently not using Skylar Advisor. The second half of this chapter covers how to install Skylar AI in an on-premises environment.

Skylar Analytics Cluster Requirements

Before beginning the installation process, verify that your environment has the required versions of Kubernetes and the container runtime (where relevant) listed in the following sections.

Skylar Analytics Cluster Requirements

• Kubernetes: Version 1.32 or later (Skylar Analytics only). Any container runtime will work.

  You can run the following command to verify your environment:

  kubectl version

  Kubernetes upgrades must be performed one minor version at a time. If you are planning a cluster upgrade, see the Kubernetes upgrade documentation.

• Storage: Configured EBS StorageClass capable of dynamic PV/PVC provisioning. Storage cannot be NFS-backed, as this is not supported by ScienceLogic databases. The amount of storage varies by environment; for more information see the Sizing Guide.

• Networking: Currently, only IPv4 is supported. IPv6 is not supported at this time.

• CPU: OS must have full x86-64-v2 support. If using Hypervisor, ScienceLogic recommends that you have CPU Host Passthrough enabled to achieve this.

• External Networking:

  • The Skylar AI Kubernetes cluster will need Internet access to https://registry.scilo.tools/. This connection can be through a proxy if needed (additional configuration required).

  • 443 connectivity from the Skylar One instance to the Skylar AI Kubernetes cluster.

• Deployment Options:

  • Self-hosted Kubernetes clusters, such as bare metal or VMware

  • Cloud-managed Kubernetes services (EKS, GKE, AKS)

• Availability Options:

  • By default, Skylar Analytics will deploy the Postgres, Clickhouse-Keeper, OpenTelemetry Collector, and Tenant services in a high-availability pattern.

  • For the smallest resource footprint possible, Skylar AI deployments can be configured to use the minimum replicas required for each service.

  • Additionally affinity zone and node scheduling requirements for the Postgres and Tenant services may be relaxed from hard to preferred. For more information, see Availability Profiles.

• CLI Packages: Customer will need kubectl access to the cluster along with Helm and Oras packages installed into the command-line interface (CLI).

Skylar Advisor Cluster Requirements

If you are installing only Skylar Analytics, you can skip this topic.

• Kubernetes: Version 1.35 or later with a container runtime that supports OCI image volume, such as ContainerD 2.1 or cri-o 1.32 or later runtime.

• Storage: Read Write Many (RWX) storage class.

• GPU driver version: 580.105.08 with CUDA 13.x

Required Infrastructure Components

Ingress or Gateway API controller: All ingress controllers supported, such as Traefik and nginx-ingress. If using Gateway API, your controller must support listenersets.

DNS and TLS Requirements:

• FQDN: A fully qualified domain name pointing to the load balancer or node IPs. FQDN must be resolvable by the cluster nodes DNS servers.
• TLS Certificate: Valid (can be self-signed) TLS certificate for the FQDN, provided as a Kubernetes secret.

Optional Cluster Items

Load Balancing: A load balancer solution to distribute traffic across worker nodes. This can be a physical load balanced ie: F5) or something virtual like HAProxy with floating IP, MetalLB etc.

Prometheus stack: Skylar AI components emit prometheus metrics which can be used to determine application health. The metrics can be exported to internal prometheus endpoint then consumed my Skylar One with Prometheus PowerPack.

Backup Solution: Backups of Kubernetes manifests and volumes should be available. This is helpful not only for Disaster Recovery (DR) situations but also for version upgrades.

Registry Access Setup

Pre-flight Validation

Perform all of the following validations before continuing to Step 1:

1. Verify the Kubernetes version; the minimum is version 1.32 (Skylar Analytics only) or 1.3.5 (Skylar Advisor and Skylar Analytics):

   kubectl version

2. Verify the container runtime version; ContainerD 2.1 or later is required:

   containerd --version

3. For Skylar Advisor, verify that the GPU Operator is running:

   kubectl get pods -n gpu-operator

4. For Skylar Advisor, verify that GPUs are visible to Kubernetes:

   kubectl describe nodes | grep -A5 'nvidia.com/gpu'

5. Verify that storage classes are available:

   kubectl get storageclass

6. Test the registry login:

   helm registry login registry.scilo.tools

7. Verify that the ingress controller is running (adjust the namespace for your controller):

   kubectl get pods -n ingress-nginx

8. When all of the checks have passed, continue to Step 1, below.

Step 1: Get Harbor CLI Secret

Use the Harbor command line interface (CLI) secret to authenticate to Harbor.

To get the Harbor CLI secret:

1. Go to https://registry.scilo.tools.
2. Click Login with OIDC Provider.
3. On the next page, click Customer Login.
4. Enter the Salesforce credentials that ScienceLogic provided.
5. After you log in, click your name in the upper right corner and select User Profile.
6. In the User Profile section, copy the CLI secret. You will use the secret in the following steps.

Step 2: Sign In to the Helm Registry

Use the CLI secret you copied in Step 1 to sign in to the Helm registry.

To sign in to the Helm registry:

1. Run the following command:

   helm registry login registry.scilo.tools --username <your-harbor-username>

Step 3: Install the OpenTelemetry Operator

To install the OpenTelemetry operator:

1. Run the following commands to install the OpenTelemetry operator with a custom image:

   # Install OpenTelemetry Operator with custom image
   helm install opentelemetry-operator oci://registry.scilo.tools/skylar/opentelemetry-helm-charts/opentelemetry-operator \
     --version 0.114.0 \
     --namespace opentelemetry-operator \
     --create-namespace \
     --set "manager.collectorImage.repository=registry.scilo.tools/skylar/sl-otelcol" \
     --set "manager.collectorImage.tag=0.16" \
     --set admissionWebhooks.certManager.enabled=false \
     --set admissionWebhooks.autoGenerateCert.enabled=true

If you want to use cert-manager for certificate management, omit the following lines from the command:

--set admissionWebhooks.certManager.enabled=false \

--set admissionWebhooks.autoGenerateCert.enabled=true

Step 4: Download the Skylar AI Chart

Use Helm to pull the Skylar AI chart from the Open Container Initiative (OCI) registry.

To download the Skylar AI chart:

1. Pull the chart. You must be logged in to the registry from Step 2.

   helm pull oci://registry.scilo.tools/skylar/skylar-charts

2. Decompress the chart. Replace x.x.x with the version you downloaded in the previous step. Use ls to find the correct .tgz file name.

   tar -xvf skylar-charts-x.x.x.tgz

3. Create a directory for your configuration files:

   mkdir skylar-ai-configs

Step 5: Determine Scaling Profiles

ScienceLogic sizes deployments based on the Skylar One datapoints-per-minute (DPM) rate, which depends on the number of devices. Work with ScienceLogic to determine the size you need to deploy.

This guide uses the Micro sizing file, which supports up to 5,000 DPM and requires few resources.

To copy the sizing file, run the following command:

cp skylar-charts/sizing/analytics-micro.yaml skylar-ai-configs/

Availability Profiles

Skylar AI can be configured for high availability (HA). The default deployment requires three nodes so that pods can run on distinct hosts. You can split these nodes across data centers if latency is low.

For a proof of concept (POC), ScienceLogic recommends the no-ha.yaml profile to keep resource requirements low.

Profile	Override File	Minimum Nodes	Scheduling
HA (default)	None. This profile is the default.	3	3 replicas on distinct nodes
Soft HA	soft-ha.yaml	1	3 replicas, which can run on a single node
No HA	no-ha.yaml	1	1 replica

If you choose soft-ha.yaml or no-ha.yaml, copy the file into the configuration directory. If you use the default profile, you do not need to copy a file.

To copy an availability profile:

1. Run the command for the profile you want to use:

   cp skylar-charts/availability/no-ha.yaml skylar-ai-configs/

   cp skylar-charts/availability/soft-ha.yaml skylar-ai-configs/

Installation Process

Analytics Configuration File Setup

Prepare the Helm override files with details specific to your environment. The decompressed skylar-charts directory contains an examples directory.

If you are deploying Skylar Analytics, copy examples/override-analytics.yaml and populate the copy.

To set up the Skylar Analytics override file:

1. Copy the example file:

   cp skylar-charts/examples/override-analytics.yaml skylar-ai-configs/

2. If you want to use the Gateway API instead of ingress, copy the Gateway API example:

   cp skylar-charts/examples/override-analytics-gateway-api.yaml skylar-ai-configs/

3. Edit the override file:

   vi skylar-ai-configs/override-analytics.yaml

Ingress Controller Annotations and Known Issues

The override-analytics.yaml file contains ingress-nginx annotations. Annotations differ by controller, so the file includes only the ingress-nginx annotations as an example. Use them as a guide to implement the equivalent annotations for your ingress controller.

Traefik: ScienceLogic has done limited testing with Traefik and found that the nginx ingress translation layer works for this application. ScienceLogic suggests this approach if you use Traefik. For more information, see the Traefik ingress-nginx documentation. Otherwise, contact ScienceLogic for help implementing middleware for Traefik.

nginx-ingress (F5): nginx-ingress requires a master/minion configuration. If you use nginx-ingress, set mergeableIngress: true.

ingress-nginx: The controller must allow snippet annotations. Use the following Helm chart overrides:

--set controller.allowSnippetAnnotations=true

--set controller.config.annotations-risk-level=Critical

Proxy Configuration

If you use a proxy for outbound internet traffic, uncomment the sections in override-analytics.yaml that set the HTTP_PROXY, HTTPS_PROXY, and NO_PROXY environment variables.

Adding Streaming Telemetry

To enable the Streaming Telemetry product, set the following flags in override-analytics.yaml:

global:

enableOtelUi: true

enableStreamingTelemetry: true

alertBuilderEnabled: true

otel:

collector:

featureFlag: true

Adding Prometheus Monitoring

To enable Prometheus monitoring, set the following flag in override-analytics.yaml. This flag exposes Prometheus metrics endpoints on Skylar AI services that your existing Prometheus deployment can scrape.

global:

enableMonitoring: true

Skylar Advisor Configuration (If Applicable)

If you are also deploying Skylar Advisor, populate examples/override-advisor.yaml as well. Skylar Advisor also requires a fourth override file that specifies the GPU-specific tolerations and resources. Work directly with ScienceLogic to create this configuration. The sizing directory contains example files. For example, sizing/l40s-rtx6000-combined.yaml and sizing/h100-resources.yaml are good starting points.

To set up the Skylar Advisor override files:

1. Copy the example files:

   cp skylar-charts/examples/override-advisor.yaml skylar-ai-configs/

   cp skylar-charts/sizing/l40s-rtx6000-combined.yaml skylar-ai-configs/

2. Edit override-advisor.yaml with the details for your cluster:

   vi skylar-ai-configs/override-advisor.yaml

Deploying the Skylar AI Platform

To deploy the Skylar AI platform:

1. Deploy Skylar AI with the scaling profile and your custom overrides:

   # Deploy skylar with scaling profile and custom overrides
   helm upgrade --install skylar-production \
     oci://registry.scilo.tools/skylar/skylar-charts \
     --namespace skylar-production \
     --create-namespace \
     --values skylar-ai-configs/analytics-micro.yaml \
     --values skylar-ai-configs/override-analytics.yaml
   

2. If you enable Skylar Advisor, add the following options to the command. The GPU file name is a sample. You can name the file to match the GPU you deploy.

   --values skylar-ai-configs/override-advisor.yaml

   --values skylar-ai-configs/l40s-rtx6000-combined.yaml

3. If you enable the soft HA or no HA option, add the following option to the command:

   --values skylar-ai-configs/no-ha.yaml

4. Save the command that you used. You will use the same command for future upgrades.

   vi skylar-ai-configs/upgrade.txt

5. Paste the Helm upgrade command into the text file.

Configure the TLS Certificate

Create a TLS secret with your certificate and private key.

Skip this step if you have automated certificate management, such as cert-manager, or deployment processes that handle TLS configuration.

Create the TLS secret with your certificate:

kubectl create secret tls skylar-tls-secret \

--cert=path/to/your/certificate.crt \

--key=path/to/your/private.key \

--namespace=skylar-production

Validation and Access

Verify Deployment Status

To verify the deployment status:

1. Check that all pods are running:

   kubectl get pods -n skylar-production

2. Check the volumes:

   kubectl get pvc -n skylar-production

Check Ingress and User Interface

The following commands show the configured domain name. Use that domain name to access the user interface (UI).

To check the ingress configuration:

1. Verify the ingress configuration:

   kubectl get ingress -n skylar-production

2. If you use the Gateway API, verify that the HTTP routes and listener sets appear:

   kubectl get httproutes

   kubectl get listenerset

Useful Commands

Use the following commands to check the deployment status and troubleshoot:

1. Check the deployment status:

   helm status skylar-prod -n skylar-production

2. View pod logs for troubleshooting:

   kubectl logs <pod-name> -n skylar-production

3. Describe a pod to view detailed information:

   kubectl describe pod <pod-name> -n skylar-production

Troubleshooting

Registry Authentication Issues

If you encounter image pull errors, try the following:

• Verify CLI Secret. Ensure that the CLI secret has not expired in Harbor.
• Test Registry Login. Re-authenticate with helm registry login.
• Check Secret Format. Verify that you created the Kubernetes secret correctly.
• Namespace Access. Ensure that the secret exists in the correct namespace.