This is fork and a significant update from the initial publication on the ISC Storm Center website by Scott Jensen as a BACS paper and the scripts published in Github.
https://github.com/fkadriver/Dshield-ELK
https://isc.sans.edu/diary/DShield+Sensor+Monitoring+with+a+Docker+ELK+Stack+Guest+Diary/30118

This is a good reference on howto use DShield-SIEM for analysis: https://isc.sans.edu/diary/30962/

This provides an overview how the log collection with the DShield sensor is done.
https://github.com/bruneaug/DShield-SIEM/blob/main/Troubleshooting/DShield-SIEM-Flow.png

This docker is custom built to be used with the DShield Honeypot to collect and parse the logs and collect its data in a visual and easy to search for research purposes. The suggested installation is to install the DShield sensor in a Rasperry using PI Raspbian OS or a system running Ubuntu 20.04 LTS either in your network or in the cloud of your choice.

• This was tested on Ubuntu 20.04 LTS
  
• Step 1 build Ubuntu
  
• Step 2 install docker and ELK
  
• Step 3 install and configure Filebeat on DShield Sensor(s)
  

• Ubuntu 20.04 LTS Live Server 64-Bit
  
• Minimum 8+ GB RAM
  
  • If the amount of RAM assigned to each containers (see below) is more than 2GB, consider increasing the server RAM capacity.
    
• 4-8 Cores
  
• Minimum 300 GB partition assigned to /var/lib/docker
  
• Adding a 300 GB to a VM: https://github.com/bruneaug/DShield-SIEM/blob/main/AddOn/Build_a_Docker_Partition.md

• Kibana
• Elasticsearch
• Logstash
• Elastic-Agent

Note: To update ELK server components, follow these steps:
https://github.com/bruneaug/DShield-SIEM/blob/main/README.md#download-github-update

$ sudo apt-get install ca-certificates curl gnupg network-manager txt2html
$ sudo install -m 0755 -d /etc/apt/keyrings
$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
$ sudo chmod a+r /etc/apt/keyrings/docker.gpg
$ echo "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
"$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
$ sudo apt update && sudo apt upgrade
$ sudo reboot (if update were applied)
$ sudo apt-get install -y jq docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin pip
$ sudo systemctl enable docker

$ git clone https://github.com/bruneaug/DShield-SIEM.git
$ chmod 754 ~/DShield-SIEM/scripts/cowrie-setup.sh
$ mkdir scripts
$ mv DShield-SIEM/AddOnScripts/parsing_tty.sh scripts
$ mv DShield-SIEM/AddOnScripts/rename_arkime_pcap.sh scripts
$ chmod 754 scripts/*.sh

The parsing_tty.sh script will be configured later in another document.
$ cd ~/DShield-SIEM

Note: Before installation, you can edit the .env file to make any derided changes.
- Current default password for elastic is student
Memory Limits in .env are the most memory that docker will allocate for each of the ELK containers.
Default to 2147483648 (2GB) but can be expanded if you have the resources

• Update the following variables in Logstash to match your ELK server DNS information, hostname, IP and default elastic password if you want to change it:
  
  • HOSTNAME="ubuntu"
  • DNS_SERVER="9.9.9.9"
  • IPADDRESS="192.168.25.231"
  • ELASTIC_PASSWORD=student
• If you want to change the default nameserver(s), go to the following directory for the files
  • $ cd logstash/pipeline
  • logstash-200-filter-cowrie.conf
    
  • logstash-201-filter-iptables.conf
    
  • logstash-202-filter-cowrie-webhoneypot.conf
    You can keep these default or edit each files and change them.

Now execute docker compose to build the ELK server applications.
This will build: Kibana, Elasticsearch, elastic-agent, Logstash and load the Cowrie parsers, configuration files and dashboard.

$ sudo docker compose up -d

Enable and start the docker service. This will restart DShield-SIEM when the server is rebooted.

$ sudo systemctl enable docker.service
$ sudo systemctl start docker.service

Confirm the docker service is running
$ sudo systemctl status docker.service

Installation Completed

This command shows the list of docker directories in used after all of Elasticsearch components have been installed.
As data gets added to Elasticsearch, you can also monitor either with the command below or within ELK the amount of disk is available for storage.
$ sudo du --human-readable --max-depth 1 --no-dereference --one-file-system /var/lib/docker

Using netstat, these 4 services should now be listening.

$ netstat -an | grep '9200\|8200\|5601\|5044'
tcp        0      0 0.0.0.0:5601            0.0.0.0:*               LISTEN  ---> Kibana
tcp        0      0 0.0.0.0:8220            0.0.0.0:*               LISTEN  ---> elastic-agent
tcp        0      0 0.0.0.0:9200            0.0.0.0:*               LISTEN  ---> Elasticsearch
tcp        0      0 0.0.0.0:5044            0.0.0.0:*               LISTEN  ---> Logstash
tcp6       0      0 :::5601                 :::*                    LISTEN
tcp6       0      0 :::8220                 :::*                    LISTEN
tcp6       0      0 :::9200                 :::*                    LISTEN
tcp6       0      0 :::5044                 :::*                    LISTEN

After docker finish installing all the ELK docker components, now it is time to login the ELK stack using your Ubuntu server IP.

• Login Kibana with username: elastic and default password if it hasn't been changed: student
  Web Access: https://serverIP:5601

In order to see the ELK stack health, it is necessary to configure the stack monitoring by navigating to: Management -> Stack Monitoring

• Select "Or, set up with self monitoring"
  
• Monitoring is currently off -> Select: Turn on monitoring
  

The elastic-agent will be used to ingest threat intelligence. It can also be used to do other things that won’t be covered here.
This is an example for the format to setup the fleet-server and the elastic-agent:
https://github.com/bruneaug/DShield-SIEM/blob/main/Troubleshooting/fleet-server-examples.txt

• From the dropdown menu, select Management → Fleet →Settings → Edit Outputs (Actions)
  

• Login server via SSH
  

• Copy ca.crt certificate to /tmp
  $ sudo cp /var/lib/docker/volumes/dshield-elk_certs/_data/ca/ca.crt /tmp

• Get a copy of Elasticsearch CA trusted fingerprint
  $ sudo openssl x509 -fingerprint -sha256 -noout -in /tmp/ca.crt | awk -F"=" {' print $2 '} | sed s/://g

• The output will look like this:
  673FB617E15CCCE73F9B647EF99449642A19CFC1D75BF5772047DA99DB950844

• Get Content of Elasticsearch CA Certificate to Apply to Advanced YAML configuration. Type the command because it doesn't copy well
  $  sudo cat /tmp/ca.crt | sed -r 's/(.*)/ \1/g'

Follow the example from the Troubleshooting fleet-server-examples guide URL above for the correct format.
sed will add the 4 spaces with the previous command against the CA certificate

• Change the Hosts to: https://es01:9200

After adding the certificate information, save and apply these settings.
Followed by Save and deploy

Under Settings, configure the Fleet server hosts by either edit or Add Fleet Server configured as follows:

Next phase is to Select Agent Policy → Add Agent → Enroll in Fleet → Add Fleet Server

• Provide a Name: es01
• Provide URL: https://fleet-server:8220
• Last: Generate Fleet Server policy
• Select: RPM Refer to this page for an example of Fleet-Server SSL Configuration:
  https://github.com/bruneaug/DShield-SIEM/blob/main/Troubleshooting/fleet-server-examples.txt

We are going to need this information to setup our fleet server.
Login via SSH to the fleet-server and make sure the fleet-server is running before setting up our agent:

$ sudo docker start fleet-server
$ sudo docker exec -ti fleet-server bash
$ ./elastic-agent status (check it is running)
$ ./elastic-agent restart (if it doesn't appear to be running, force a restart, and recheck the status)

This is an example of what need to be copied to the fleet server. Ensure the fleet server es is: https://es01:9200
Add the bold section after port=8220 because are certificates are self-generated. This will ensure the agent takes the update.

The token and fingerprint will be different than this example but what is in italic and bolded must be added for the certificat to load:

elastic-agent enroll \
--url=https://fleet-server:8220 \
--fleet-server-es=https://es01:9200 \
--fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE3MDU0NDg3MDMwNTI6NkNxcWlCeTRRVmlhYW0yeldhN3pGZw \
--fleet-server-policy=fleet-server-policy \
--fleet-server-es-ca-trusted-fingerprint=76DA77DAE186F8CFBA9E87D450D5419B68E2555A9BD57795611C0545ED0BF03F \
--fleet-server-port=8220 \
--certificate-authorities=/certs/ca/ca.crt \
--fleet-server-es-ca=/certs/es01/es01.crt \
--insecure

This will replace your current settings. Do you want to continue? [Y/n]: Y

Successfully enrolled the Elastic Agent.

From your current location, verify it installed correctly
$ ./elastic-agent status
$ ./elastic-agent restart (if you got what appears to be errors, force a restart and check the status)

Now that the Fleet Server is connected, close this Windows and lets proceed to the next step.

In Elastic Management → Fleet, refresh Agents and this is what shows up:

The server is now ready to install Threat Intel Agents to be used in Security (SIEM portion) against the honeypot logs.
The next step is to select Agent policies → Fleet Server Policy → Add integration:
Use this Dashboard to view the metrics collected by elastic-agent:

• Select and Add AlienVault OTX (need an API key)
  
• Select AbuseCH (no API key needed)
  
• Select Threat Intelligence Utilities
  
• Elasticsearch
  
• Select Kibana
  
• Select Docker
  

• In Elastic Management → Installed Integration
  
• Select each of the installed integration, then select Settings and enable the tab to keep the policy up to date:
  

• Select Rules → Detection rules (SIEM) → Add Elastic rules
• Under Search Tags: Rule Type: Indicator Match (add at the minimum those 4 rules)
• Install and enable those 4 rules
• You can look through the rules and enable those other rules that you want to try against your honeypot data.
  

If some of the threat intel above were added to the elastic-agent, and cowrie* was added to the Management -> Advanced Settings during the initial installation, it can now track rules that match in the alert trends of the SIEM part of the ELK stack.
This is the output from Management → Stack Management → Advanced Settings

If any threat intel match, the SIEM will display the following activity that can now be investigated:

After adding the webhoneypot.sh script, add the Filebeat package to the DShield Sensor to send the logs the Elasticsearch.

Use the following steps to install Filebeat using the following commands taken from this reference [3]:

$ wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
$ sudo apt-get install apt-transport-https
$ echo "deb https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-8.x.list
$ echo "deb https://artifacts.elastic.co/packages/oss-8.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-8.x.list
$ sudo apt-get update && sudo apt-get install filebeat elastic-agent softflowd

Download the custom filebeat.yml file that will forward the logs the Elasticsearch:

$ sudo curl https://raw.githubusercontent.com/bruneaug/DShield-SIEM/main/filebeat.yml -o /etc/filebeat/filebeat.yml

• Edit the filebeat.yml and change the IP address to the logstash parser (192.168.25.23) to match the IP used by Logstash:
  $ sudo vi /etc/filebeat/filebeat.yml
  

  output.logstash:
  hosts: ["192.168.25.23:5044"]
  

$ sudo systemctl enable filebeat
$ sudo systemctl start filebeat
$ sudo systemctl status filebeat
$ sudo systemctl enable elastic-agent
$ sudo systemctl start elastic-agent
$ sudo systemctl enable softflowd
$ sudo systemctl start softflowd

Filebeat tracks the events it has processed with a file located called log.json, if deleted, all the events that were previous sent to Elasticsearch will be reprocessed when filebeat is restarted.
The location of this file:
$ cd /var/lib/filebeat/registry/filebeat

Follow this step-by-step documents to install the elastic-agent to the DShield sensor
https://github.com/bruneaug/DShield-SIEM/blob/main/Configure-Elastic-Agent.pdf

To access the Dashboard select Analytics -> Dashboard -> [Logs DShield Sensor] Overview

This script will start all the ELK components and if installed, the Arkime services.

Install startelk.tgz tarball to ELK server as follow:
$ cd ~/DShield-SIEM/AddOn
$ sudo tar zxvf startelk.tgz -C /

$ sudo vi /etc/init.d/startelk.sh

The script is configured with the default password to sudo: training
You need to change it to the account's password in use.
You need to change the account under which location DShield-SIEM is located.

PASSWORD="training" ELK="**/home/guy/**DShield-SIEM"

$ sudo docker compose rm -f -v (clear setup but need to run up -d again)
$ sudo docker compose up -d (reload container with changes)
$ sudo docker compose up --build -d
$ sudo docker compose up --build --force-recreate -d (force a rebuild if the container hasn't changed)
$ sudo docker compose start/stop
$ sudo docker compose ps (list running containers)
$ sudo docker stats (shows status of container)
$ sudo docker container ls/ps
$ sudo docker network ls (network listing)
$ sudo docker stats (shows status of container)
$ sudo docker system prune -a (Remove everything)
$ sudo docker logs kibana (troubleshooting docker)
$ sudo docker compose logs --follow (debugging)
$ sudo docker compose down --remove-orphans && sudo docker compose up --build -d (Removed or renamed orphan container)
$ sudo docker rm -f cowrie (remove a container)
$ sudo docker system df (Check docker usage)
$ sudo du --human-readable --max-depth 1 --no-dereference --one-file-system /var/lib/docker (Shows a breakdown of docker filesystem)
$ sudo docker builder prune (Clear the build cache if > 0)

$ sudo docker stop fleet-server
$ sudo docker restart logstash (restart logstash service)
$ sudo docker logs logstash (looking at debugging information)
$ sudo docker inspect logstash

$ cd DShield-SIEM
$ sudo docker compose stop
Edit the .env and reset your your hostname & IP address variables. The other option is to manually update to the new version and skip git pull
. $ git pull (Update the code from Github)
$ sudo docker compose rm -f -v
$ sudo docker compose up --build -d

$ sudo docker inspect logstash

$ sudo docker exec -ti es01 bash
$ sudo docker exec -ti logstash bash
$ sudo docker exec -ti kibana bash
$ sudo docker exec -ti fleet-server bash
$ sudo docker exec -ti cowrie bash

$ sudo docker cp server:/usr/share/elastic-agent/elastic-agent.yml .
$ sudo docker cp elastic-agent.yml fleet-server:/usr/share/elastic-agent/

These command are run from the Dev Tool -> Console

GET _nodes/http?pretty (Get a list and information of all the nodes)
GET _security/_authenticate

[1] https://isc.sans.edu/tools/honeypot/
[2] https://www.elastic.co/downloads/beats/filebeat
[3] https://www.elastic.co/guide/en/beats/filebeat/8.8/setup-repositories.html#_apt
[4] https://isc.sans.edu/diary/DShield+Honeypot+Activity+for+May+2023/29932
[5] https://isc.sans.edu/diary/DShield+Sensor+JSON+Log+to+Elasticsearch/29458
[6] https://isc.sans.edu/diary/DShield+Sensor+JSON+Log+Analysis/29412
[7] https://github.com/jslagrew/cowrieprocessor/blob/main/submit_vtfiles.py
[8] https://handlers.sans.edu/gbruneau/elastic.htm
[9] https://www.elastic.co/guide/en/fleet/current/secure-connections.html
[10] https://www.docker.elastic.co/