Demonstration of Vault audit log parsing and indexing using Open Distro variant of Elasticsearch and Kibana. This is because it provides important features like LDAP, role based access, and index management out of the box. The stack also includes OSS version of Logstash and Filebeat (i.e Apache2 license).
This setup is built on OpenStack cluster https://github.com/Jibinxavier/Openstack-Homelab
Elasticsearch cluster is a master-master setup (elk01 and elk02), where elk01 was initially configured to be the master node. This is not a highly available cluster as its prone to single node failure and underlying Openstack cluster is not at all fault tolerant
Simple single node Vault and Consul cluster. Where Consul is the storage backend for Vault.
There are two pipelines logstash.conf and deadletter.conf.
logstash.conf is the main pipeline. It contains parsing logic to send a log line to an index depending on identified type. Types could be vault operational log, vault audit log, caonsul agent log and more.
deadletter.conf leverages DLQ to keep track of log lines that failed parsing. And can be used to improve the logstash.conf.
The VMs and network configurations are created using Terraform. Ansible playbooks are used to install and configure Vault and Elasticsearch components.
Main Ansible roles are:
- certs-gen - to generate root and client certficates
- deploy-consul
- deploy-elasticsearch
- deploy-filebeat
- deploy-kibana
- deploy-logstash
- deploy-vault
- Dashboards
- Improve separation of Vault audit logs and vault operational logs
- Better access control - not using admin