Skylar AI On-Premises Installation Guide

This chapter details the Kubernetes cluster requirements for Skylar Analytics and Skylar Advisor. It also includes requirements for only Skylar Analytics if you are currently not using Skylar Advisor. The second half of this chapter covers how to install Skylar AI in an on-premises environment.

For detailed sizing information, see the Skylar AI System Requirements.

Skylar Analytics Cluster Requirements

Before beginning the installation process, verify that your environment has the required versions of Kubernetes and the container runtime (where relevant) listed in the following sections.

Skylar Analytics Cluster Requirements

Kubernetes: Version 1.32 or later (Skylar Analytics only). Any container runtime will work.

You can run the following command to verify your environment:

kubectl version

Kubernetes upgrades must be performed one minor version at a time. If you are planning a cluster upgrade, see the Kubernetes upgrade documentation.
Storage: Configured EBS StorageClass capable of dynamic PV/PVC provisioning. Storage cannot be NFS-backed, as this is not supported by ScienceLogic databases. The amount of storage varies by environment; for more information, see the Skylar AI System Requirements.
Networking: Currently, only IPv4 is supported. IPv6 is not supported at this time.
CPU: OS must have full x86-64-v2 support. If using Hypervisor, ScienceLogic recommends that you have CPU Host Passthrough enabled to achieve this.
External Networking:
- The Skylar AI Kubernetes cluster will need Internet access to https://registry.scilo.tools/. This connection can be through a proxy if needed (additional configuration required).
- 443 connectivity from the Skylar One instance to the Skylar AI Kubernetes cluster.
Deployment Options:
- Self-hosted Kubernetes clusters, such as bare metal or VMware
- Cloud-managed Kubernetes services (EKS, GKE, AKS)
Availability Options:
- By default, Skylar Analytics will deploy the Postgres, Clickhouse-Keeper, OpenTelemetry Collector, and Tenant services in a high-availability pattern.
- For the smallest resource footprint possible, Skylar AI deployments can be configured to use the minimum replicas required for each service.
- Additionally affinity zone and node scheduling requirements for the Postgres and Tenant services may be relaxed from hard to preferred. For more information, see Availability Profiles.
CLI Packages: Customer will need kubectl access to the cluster along with Helm and Oras packages installed into the command-line interface (CLI).
Licensing: To pull application images, you need an active Skylar AI license. Please work with ScienceLogic to ensure that you are properly licensed for Skylar AI. Also, make sure that you have access to the skylar Harbor project before beginning deployment.

Skylar Advisor Cluster Requirements

If you are installing only Skylar Analytics, you can skip this topic.

Kubernetes: Version 1.35 or later with a container runtime that supports OCI image volume, such as ContainerD 2.1 or cri-o 1.32 or later runtime.
Storage: Read Write Many (RWX) storage class.
GPU driver version: 580.105.08 with CUDA 13.x

Required Infrastructure Components

Ingress or Gateway API controller: All ingress controllers supported, such as Traefik and nginx-ingress. If using Gateway API, your controller must support listenersets.

DNS and TLS Requirements:

FQDN: A fully qualified domain name pointing to the load balancer or node IPs. FQDN must be resolvable by the cluster nodes DNS servers. Skylar One appliances must be able to resolve the Skylar AI FQDN for telemetry and data streaming functionality.
TLS Certificate: Valid (can be self-signed) TLS certificate for the FQDN, provided as a Kubernetes secret.

Optional Cluster Items

Load Balancing: A load balancer solution to distribute traffic across worker nodes. This can be a physical load balancer like F5 or something virtual like HAProxy with floating IP or MetalLB.

Prometheus stack: Skylar AI components emit Prometheus metrics that can be used to determine application health. The metrics can be exported to an internal Prometheus endpoint then consumed my Skylar One with Prometheus PowerPack.

Backup Solution: Backups of Kubernetes manifests and volumes should be available. This is helpful for Disaster Recovery (DR) situations and for version upgrades.

Pre-installation Steps

Run Pre-installation checks

Check to see if Helm version 3 or 4 is installed:

helm version
version.BuildInfo{Version:"v3.18.6",

Check to see if kubectl is installed:

kubectl get pods

Check to see if oras installed:

oras version
Version:    1.3.0+Homebrew
Go version: go1.25.5
OS/Arch:    darwin/arm64

Check Access to Harbor

Use the Harbor command line interface (CLI) secret to authenticate to Harbor.

To get the Harbor CLI secret:

Go to https://registry.scilo.tools.
Click Login with OIDC Provider.
On the next page, click Customer Login.
Enter the Salesforce credentials that ScienceLogic provided.
After you log in, click your name in the upper right corner and select User Profile.
In the User Profile section, copy the CLI secret.
Use the secret to sign in to the Helm registry:

helm registry login registry.scilo.tools --username <your-harbor-username>
helm pull oci://registry.scilo.tools/skylar/skylar-charts

Install the OpenTelemetry Operator

To install the OpenTelemetry operator:

Run the following commands to install the OpenTelemetry operator with a custom image:

# Install OpenTelemetry Operator with custom image
helm install opentelemetry-operator oci://registry.scilo.tools/skylar/opentelemetry-helm-charts/opentelemetry-operator \
  --version 0.114.0 \
  --namespace opentelemetry-operator \
  --create-namespace \
  --set "manager.collectorImage.repository=registry.scilo.tools/skylar/sl-otelcol" \
  --set "manager.collectorImage.tag=0.16" \
  --set admissionWebhooks.certManager.enabled=false \
  --set admissionWebhooks.autoGenerateCert.enabled=true

If you want to use cert-manager for certificate management, omit the following lines from the command:

--set admissionWebhooks.certManager.enabled=false \

--set admissionWebhooks.autoGenerateCert.enabled=true

Download the Skylar AI Chart

Use Helm to pull the Skylar AI chart from the Open Container Initiative (OCI) registry.

To download the Skylar AI chart:

Pull the chart. You must be logged in to the registry from Step 2.

helm pull oci://registry.scilo.tools/skylar/skylar-charts
Decompress the chart. Replace x.x.x with the version you downloaded in the previous step. Use ls to find the correct .tgz file name.

tar -xvf skylar-charts-x.x.x.tgz
Create a directory for your configuration files:

mkdir skylar-ai-configs

Determine Scaling Profiles

ScienceLogic sizes deployments based on the Skylar One datapoints-per-minute (DPM) rate, which depends on the number of devices. Work with ScienceLogic to determine the size you need to deploy.

This guide uses the Micro sizing file, which supports up to 5,000 DPM and requires few resources.

To copy the sizing file, run the following command:

cp skylar-charts/sizing/analytics-micro.yaml skylar-ai-configs/

Availability Profiles

Skylar AI can be configured for high availability (HA). The default deployment requires three nodes so that pods can run on distinct hosts. You can split these nodes across data centers if latency is low.

For a proof of concept (POC), ScienceLogic recommends the no-ha.yaml profile to keep resource requirements low.

Profile	Override File	Minimum Nodes	Scheduling
HA (default)	None. This profile is the default.	3	3 replicas on distinct nodes
Soft HA	soft-ha.yaml	1	3 replicas, which can run on a single node
No HA	no-ha.yaml	1	1 replica

If you choose soft-ha.yaml or no-ha.yaml, copy the file into the configuration directory. If you use the default profile, you do not need to copy a file.

To copy an availability profile:

Run the command for the profile you want to use:

cp skylar-charts/availability/no-ha.yaml skylar-ai-configs/

cp skylar-charts/availability/soft-ha.yaml skylar-ai-configs/

Installation Process

Step 1: Analytics Configuration File Setup

Prepare the Helm override files with details specific to your environment. The decompressed skylar-charts directory contains an examples directory.

If you are deploying Skylar Analytics, copy examples/override-analytics.yaml and populate the copy.

To set up the Skylar Analytics override file:

Copy the example file:

cp skylar-charts/examples/override-analytics.yaml skylar-ai-configs/
If you want to use the Gateway API instead of ingress, copy the Gateway API example:

cp skylar-charts/examples/override-analytics-gateway-api.yaml skylar-ai-configs/
Edit the override file:

vi skylar-ai-configs/override-analytics.yaml

Ingress Controller Annotations and Known Issues

The override-analytics.yaml file contains ingress-nginx annotations. Annotations differ by controller, so the file includes only the ingress-nginx annotations as an example. Use them as a guide to implement the equivalent annotations for your ingress controller.

Traefik: ScienceLogic has done limited testing with Traefik and found that the nginx ingress translation layer works for this application. ScienceLogic suggests this approach if you use Traefik. For more information, see the Traefik ingress-nginx documentation.

If using middleware, apply this manifest for URL re-writes:

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: skylar-auth-sl1embed-rewrite
  namespace: skylar-production
spec:
  replacePath:
    path: /skylar-auth/sl1embed
---
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: skylar-auth-sl1embed-headers
  namespace: skylar-production
spec:
  headers:
    customResponseHeaders:
      Content-Security-Policy: "frame-ancestors *;"

nginx-ingress (F5): nginx-ingress requires a master/minion configuration. If you use nginx-ingress, set mergeableIngress: true.

ingress-nginx: The controller must allow snippet annotations. Use the following Helm chart overrides:

--set controller.allowSnippetAnnotations=true

--set controller.config.annotations-risk-level=Critical

Proxy Configuration

If you use a proxy for outbound internet traffic, uncomment the sections in override-analytics.yaml that set the HTTP_PROXY, HTTPS_PROXY, and NO_PROXY environment variables.

Adding Prometheus Monitoring

To enable Prometheus monitoring, set the following flag in override-analytics.yaml. This flag exposes Prometheus metrics endpoints on Skylar AI services that your existing Prometheus deployment can scrape.

global:
  enableMonitoring: true

Step 2: Skylar Advisor Configuration (If Applicable)

If you are also deploying Skylar Advisor, populate examples/override-advisor.yaml as well. Skylar Advisor also requires a fourth override file that specifies the GPU-specific tolerations and resources. Work directly with ScienceLogic to create this configuration.

The sizing directory contains example files. For example, sizing/l40s-rtx6000-combined.yaml and sizing/h100-resources.yaml are good starting points.

To set up the Skylar Advisor override files:

Copy the example files:

cp skylar-charts/examples/override-advisor.yaml skylar-ai-configs/

cp skylar-charts/sizing/l40s-rtx6000-combined.yaml skylar-ai-configs/
Edit override-advisor.yaml with the details for your cluster:

vi skylar-ai-configs/override-advisor.yaml

Step 3: Configure the Harbor Password

Use the commands below to create the secret used for ImagePulls. Concepts like ExternalSecretsOperator and Vault are fully supported. Just configure your provider to create the following secrets instead:

kubectl create namespace skylar-production

kubectl -n skylar-production create secret docker-registry harbor-creds \
  --docker-server=registry.scilo.tools \
  --docker-username='your-email@sciencelogic.com' \
  --docker-password='HarborCLISecret'

kubectl -n skylar-production create secret generic harbor-creds-mountable \
  --from-literal=HARBOR_USERNAME='your-email@sciencelogic.com' \
  --from-literal=HARBOR_PASSWORD='HarborCLISecret' \
  --from-literal=HARBOR_REGISTRY=registry.scilo.tools

Step 4: Configure the TLS Certificate

You can skip this step if you have automated certificate management, such as cert-manage, or deployment processes that handle TLS configuration.

Create a TLS secret with certificate and key provided by your Certificate Authority of choice. You may use self-signed if you prefer. Place certificates in the skylar-ai-configs/certs directory.

mkdir skylar-ai-configs/certs
cd skylar-ai-configs/certs

# Below OpenSSL command is just an example of self signed cert. Recommended to use real 
cert signed by Certificate Authority 
openssl req -x509 -newkey rsa:2048 -nodes -sha256 -days 825 \
  -keyout tls.key -out tls.crt \
  -subj "/CN=on-prem-advisor.ai.sciencelogic.biz" \
  -addext "subjectAltName=DNS:example.com" \
  -addext "basicConstraints=critical,CA:TRUE"

chmod 600 tls.key

kubectl create secret tls tls-certificate \
  --cert=tls.crt --key=tls.key \
  --namespace=sky \
  --dry-run=client -o yaml | kubectl apply -f -

Step 5: Deploying the Skylar AI Platform

To deploy the Skylar AI platform:

Deploy Skylar AI with the scaling profile and your custom overrides:

helm upgrade --install skylar-production \
  oci://registry.scilo.tools/skylar/skylar-charts \
  --namespace skylar-production \
  --values skylar-ai-configs/analytics-micro.yaml \
  --values skylar-ai-configs/override-analytics.yaml

If you are enabling Skylar Advisor, add the following options to the command. The GPU file name is a sample. You can name the file to match the GPU you deploy.
```
  --values skylar-ai-configs/override-advisor.yaml 
  --values skylar-ai-configs/l40s-rtx6000-combined.yaml 
```
If you enable the soft HA or no HA option, add the following option to the command:

--values skylar-ai-configs/no-ha.yaml
Save the command that you used. You will use the same command for future upgrades.

vi skylar-ai-configs/upgrade.txt
Paste the Helm upgrade command into the text file.

Validation and Access

Verify Deployment Status

To verify the deployment status:

Check that all pods are running:

kubectl get pods -n skylar-production
Check the volumes:

kubectl get pvc -n skylar-production

Check Ingress and User Interface

The following commands show the configured domain name. Use that domain name to access the user interface (UI).

To check the ingress configuration:

Verify the ingress configuration:

kubectl get ingress -n skylar-production
If you use the Gateway API, verify that the HTTP routes and listener sets appear:

kubectl get httproutes

kubectl get listenerset

Troubleshooting

Registry Authentication Issues

If you encounter image pull errors, try the following:

Verify CLI Secret. Ensure that the CLI secret has not expired in Harbor.
Test Registry Login. Re-authenticate with helm registry login.
Check Secret Format. Verify that you created the Kubernetes secret correctly.
Namespace Access. Ensure that the secret exists in the correct namespace.