Kubernetes Collector

The zlog-collector is the Skylar Automated RCA log collector for Kubernetes. 

Installing the Helm Chart

To install the Helm chart with the release name zebrium, run the following commands:

helm repo add zebrium http://charts.zebrium.com

helm upgrade -i zlog-collector zebrium/zlog-collector --namespace zebrium --create-namespace --set
zebrium.collectorUrl=<YOUR_ZE_API_URL>,zebrium.authToken=YOUR_ZE_API_AUTH_TOKEN,zebrium.deployment=
<YOUR_DEPLOYMENT_NAME>,zebrium.timezone=<KUBERNETES_HOST_TIMEZONE>

where <KUBERNETES_HOST_TIMEZONE> is the time zone setting on Kubernetes host, such as UTC or America/Los_Angeles. If this option is not provided, the default value of UTC will be used.

Uninstalling the Helm Chart

To uninstall the Helm chart with the release name zebrium, run the following command:

helm delete zlog-collector -n zebrium

Additional Information

Log Path Mapping

Log path mapping is the process of detecting semantic items in log file paths (IDs, configurations, and tags) and then including them in the Skylar Automated RCA log data. You can enable this by providing a JSON mapping file to the log collector, as described in the repository at https://www.github.com/zebrium/ze-fluentd-plugin.

To use this functionality with the supplied Helm chart, you will need to complete a customValues.yaml file and supply that file to the Helm install command line with the following command:

helm install ... -f customValues.yaml ...

A prototype example_logPathMappings.yaml file is provided in the repository under the example directory, with the following format:

overridePMFConfig: true
zebrium:
  pathMapFile: "pathMapFile.json"
customPMFConfig: {
"mappings": {
  "patterns":["/var/log/remote_logs/(?<host>[^/]+)/.*"],
    "tags": [],
    "ids" : [
    "host"],
    "configs": []
  }
}

Custom Namespace to Service Group Mapping

Matching a Custom Namespace to a Service Group is the process of dynamically assigning a service group to a log stream based on the resources namespace. This is enabled by providing a JSON mapping file to the log collector.

To use this functionality with the supplied Helm chart, complete a customValues.yaml file and supply that file to the Helm install command line with the following command:

helm install ... -f customValues.yaml ...

A prototype example_ns_svcgrp.yaml file is provided in the repository under the example directory, with the following format:

overrideSVCGRPConfig: true
zebrium:
  svcgrpMapFile: "svcgrpMapFile.json"
customSVCGRPConfig: {
  "mynamespace1" : "svcgrp1",
  "mynamespace2" : "svcgrp1",
  "mynamespace3" : "svcgrp3"
}

Values

Key	Type	Default	Description
daemonset.dnsPolicy	string	"ClusterFirst"
daemonset.nodeSelector	object	{}
daemonset.priorityClassName	string	""
daemonset.tolerateAllTaints	bool	true
daemonset.tolerations	list	[]	set ‘daemonset.tolerations [0].operator=Equal,daemonset.tolerations [0].effect=NoSchedule,daemonset.tolerations [0].key=node-role.kubernetes.io/master’
extraEnv	list	[]
image.name	string	"zebrium/zlog-collector"
image.pullPolicy	string	"Always"
image.tag	string	"latest"
resources.limits.cpu	string	"1000m"
resources.limits.memory	string	"1Gi"
resources.requests.cpu	string	"20m"
resources.requests.memory	string	"500Mi"
ruby.gcHeapOldObjectLimitFactor	float	1.2
secret.enabled	bool	true
services.automountServiceAccountToken	bool	true
services.automountServiceAccountTokenSupported	bool	false
updateStrategy	string	"OnDelete"
zebrium.authToken	string	""
zebrium.autoupdate	string	"1"
zebrium.bufferChunkLimitRecords	int	40000
zebrium.bufferChunkLimitSize	string	"8MB"
zebrium.bufferRetryMaxTimes	int	360
zebrium.bufferRetryTimeout	string	"1h"
zebrium.bufferRetryWait	string	"10s"
zebrium.bufferTotalLimitSize	string	"64GB"
zebrium.clusterName	string	""	Name of the Kubernetes Cluster that the zlog-collector is deployed into
zebrium.collectorUrl	string	""
zebrium.deployment	string	"default"
zebrium.disableEc2MetaData	string	"true"
zebrium.ec2ApiClientTimeoutSecs	string	"1"
zebrium.excludeNamespaceRegex	string	""	Regex String to Exclude Namespaces, such as: ^(?!.*(bar foo)) would exclude all namespaces except foo and bar
zebrium.excludePath	string	"[\"/var/log/boot.log\", \"/var/log/lastlog\"]"
zebrium.excludePodRegex	string	""	Regex String to exclude pods, such as: ^fluentbit.* would exclude all fluentbit pods from collection
zebrium.flushInterval	string	"30s"
zebrium.flushThreadCount	string	"4"
zebrium.handleHostAsConfig	bool	false
zebrium.k8sApiSecretName	string	""
zebrium.logFile	string	""
zebrium.logLevel	string	"info"
zebrium.name	string	"zlog-collector"
zebrium.nodeLogsPath	string	"/var/log/*.log,/var/log/syslog, /var/log/messages,/var/log/secure"
zebrium.pathMapFile	string	""
zebrium.svcgrpMapFile	string	""
zebrium.tailFromHead	string	"true"
zebrium.timezone	string	"UTC"
zebrium.useHostEtcHostnameFile	bool	false
zebrium.verifyK8sApiSSL	bool	true
zebrium.verifySSL	string	"true"