Setting Up Effective Log Management in Kubernetes

Installing and configuring an effective logging system for SecureAuth stack in any environment.

Logging at SecureAuth

At SecureAuth, we believe that logging is essential for security audits and incident investigations. By analyzing logs, security teams can identify and respond to suspicious activities, detect data breaches, and comply with regulatory standards that require data access and modification tracking.

The article describes how to configure logging in SecureAuth for both on-premises installation and SaaS solution.

Customer Deployed Installation

For on-premises installations, we recommend configuring Elasticsearch in Kubernetes (using Elastic Cloud on Kubernetes - ECK) to collect and analyze logs and tracing (OpenTelemetry - OTEL). This section includes instructions on configuring and deploying ECK, setting up OpenTelemetry to ingest logs, and visualizing and analyzing the logs collected on Elasticsearch.

Prerequisites

1. Kubernetes cluster installed

2. SecureAuth deployed on the cluster

3. ECK with Elasticsearch and Kibana installed

4. Helm installed and configured with the cluster

Configuration

In order to collect SecureAuth logs install Elastic Filebeat, a lightweight agent for forwarding and centralizing log data.

Each Kubernetes node should have an instance of Filebeat, so they should be backed by a DaemonSet object. Please follow these steps to install Filebeat using the Helm chart:

1. Add the Elastic Helm charts repo:

   helm repo add elastic https://helm.elastic.co                  

2. Update Helm repositories:

   helm repo update                  

3. Prepare values.yaml configuration file. Below is the minimal version of the file based on our experience:

   daemonset:
     filebeatConfig:
       filebeat.yml: |
         filebeat.inputs:
         - type: container
           paths:
             - /var/log/containers/acp*.log
           processors:
           - decode_json_fields:
               fields: ["message"]
               target: json
               max_depth: 1
               add_error_key: true
         processors:
         - add_host_metadata:
         - add_kubernetes_metadata:
             host: $${NODE_NAME}
             matchers:
             - logs_path:
                 logs_path: "/var/log/containers/"
         - copy_fields:
             fields:
               - from: kubernetes.container.name
                 to: event.dataset
               - from: kubernetes.container.name
                 to: app
             fail_on_error: false
             ignore_missing: true
         - rename:
             fields:
             - from: input.type
               to: host.type
             - from: json.cause
               to: error.type
             - from: json.code
               to: event.code
             - from: json.description
               to: event.type:info
             - from: json.details
               to: event.reason
             - from: json.duration
               to: event.duration
             - from: json.error
               to: error.message
             - from: json.hint
               to: event.kind:enrichment
             - from: json.host
               to: host.container.ip
             - from: json.ip
               to: client.ip
             - from: json.level
               to: log.level
             - from: json.method
               to: http.request.method
             - from: json.msg
               to: event.action
             - from: json.name
               to: service.name
             - from: json.path
               to: url.path
             - from: json.size
               to: http.response.bytes
             - from: json.stack
               to: error.stack_trace
             - from: json.status
               to: http.response.status_code
             - from: json.sub
               to: user.id
             - from: json.tenantID
               to: tenant.id
             - from: json.traceID
               to: trace.id
             - from: json.userAgent
               to: user_agent.original
             ignore_missing: true
             fail_on_error: false
         - convert:
             fields:
             - from: event.duration
               type: long
             - from: http.request.bytes
               type: long
             - from: http.response.body.bytes
               type: long
             - from: http.response.status_code
               type: long
             - from: error.code
               type: long
             ignore_missing: true
             fail_on_error: false
         output.elasticsearch:
           hosts: ["<elasticearch svc address>:9200"]
           protocol: "https"
           username: '<elasticsearch username>'
           password: '<elasticsearch password>'
           ssl:
             certificate_authorities: ["/usr/share/filebeat/certs/ca.crt"]                  

   note

   Refer to the official filebeat.yml reference page to get know more possible options for this file.

4. Install Filebeat in a dedicated logging namespace:

   helm install filebeat-release --values values.yaml --namespace logging --create-namespace elastic/filebeat                  

5. Verify installation:

   helm list --all
   kubectl get pods --namespace logging                  

   The output of the above commands should present that the Helm chart is installed and all Filebeat pods are up and running.

Hardening

In the production environment, the Elasticsearch credentials defined in the values.yaml file should be stored in a Secret entity and referred to in that file. Also, the SSL verification should be enabled, and the CA certificate of Elastisearch should be provided.