Integrating External Logging Solutions with Cloudentity Audit Events

Configure HTTP endpoint, which listens for requests with audit logs from Cloudentity. For example, if you use Elasticsearch, you can use Logstash HTTP Input plugin. Here is the basic configuration of it which processes Cloudentity audit logs and sends them to Elastic Cloud:

input {
    http {
        host => "0.0.0.0"
        port => "<port number>"
        codec => json
    }
}
filter {
    split {
        field => "audit_events"
    }
    mutate {
        remove_field => [ "audit_events" ]
        add_field => {
            "message" => "%{[audit_events]}"
        }
        remove_field => [ "audit_events" ]
    }
    json {
        source => "message"
    }
}
output {
    elasticsearch {
        cloud_id => "<cloud id>"
        api_key => "<api key>"
    }
}         

The input section contains the configuration of the endpoint dedicated to listening for incoming audit logs. Click here for more information about the http input plugin.

The filter section splits the audit_events array of the JSON payload into separate log entries. It also removes the original JSON payload from each particular log entry to reduce data duplication. Click here for more details about filter plugins.

The output parameter points to your Elastic Cloud. Click here for more information about elasticsearch output. Optionally, it can point to a self-hosted Elasticsearch installation, as described in the first chapter of this article.

Whatever solution you choose, it must be available for Cloudentity public IPs at some URL (logging endpoint URL) so that Cloudentity can reach it.

Cloudentity sends audit events in JSON batches, i.e.:

{
    "audit_events": [
        {
            "tenant_id": "example",
            "server_id": "demo",
            "event_id": "a6ea34a8ef24cfdb74da4ff9f3cb693e",
            ...
        },
        {
            "tenant_id": "example",
            "server_id": "demo",
            "event_id": "474cf3eeaf2a4f34a8e2dbf9c69b7ddf",
            ...
        },
        ...
    ]
}         

which imposes the need to split the payload into individual log entries. That is why the filter part of the above example is crucial.

Integrate Cloudentity with the logging solution configured in the previous section by creating a webhook.

This section provides a set of best practices for visualizing and exploring collected logs using Kibana.

Example Visualization

Follow these steps to create an example visualization presenting failed login causes over time:

1. In Kibana, go to Analytics > Visualize Library > Create visualization button

2. Select the Lens type.

3. Choose Bar vertical stacked in the Visualization type dropdown.

4. On the left side, add two fields to the Selected fields set:

   • @timestamp

   • json.fields.msg.audit_events.payload.login_failed.cause

5. Select Save

6. Type Failed login causes in the name field and select Save

The visualization should look like this:

A dashboard is a collection of visualizations, searches, and maps that you can use to analyze and present your data. You can create dashboards that provide an overview of your log data and allow you to drill down into specific areas as needed.

Here are some best practices for dashboards:

The Discover tool in Kibana allows you to perform deep dives on your data. You can search your data using Elasticsearch's powerful search capabilities, view detailed information about each document, and explore your data by field.

Some best practices with Discover include:

Kibana provides several built-in field formatters. Field formatters change how the data in a field is displayed in Kibana without altering the underlying data stored in Elasticsearch.

For example, you can use field formatters to display timestamps in a human-readable format or to turn URLs into clickable links. This can make your data much easier to understand at a glance.

Example usage

1. In Kibana, select Management > Stack Management > Data Views.

2. Select the Cloudentity log stream

3. Search for http.request.bytes and select the edit icon on the right side.

4. Toogle the Set format option and select the Bytes format.

5. Save.

From this point forward, whenever tje http.request.bytes is displayed, it will appear in a human-readable format:

Elasticsearch and Kibana support alerting and anomaly detection, which can be used to monitor your data for changes or unusual behavior. You can create alerts based on your log data and receive notifications when certain conditions are met.

Elasticsearch is a powerful, versatile, and scalable search and analytics engine, but it does come with some limitations and challenges. Here are a few key ones:

Elasticsearch, when combined with Kibana, provides powerful tools for visualizing and analyzing log data. By following these best practices, you can make the most of these tools, allowing you to gain deep insights into your logs and understand the behavior and performance of your systems.