Introduction

Splunk Boss of the SOC (BOTS) is an awesome capture the flag event that I have had the privilege of attending. Splunk have now released everything required to run your own BOTS v3.0, more information can be found here: https://www.splunk.com/en_us/blog/security/botsv3-dataset-released.html. Splunk have opensourced everything required to run a Splunk BOTS v3.0 event including the dataset; the questions and answers; and the scoring server.

Motivation for this article: While installing and configuring the BOTS v3.0 “toolkit” I ran into a few issues and noticed there is no end-to-end guide to get this thing working. So I have put together this article along with a handy script to make the install easier and a lot faster!

The BOTS v3.0 toolkit consists of:
BOTS v3.0 dataset
Questions CSV
Answers CSV
Hints file CSV
Scoreboard Admin app
Scoreboard app

Prerequisites

A Splunk Enterprise license is required to run BOTS. You can utilise the a free 60 day trial of Splunk Enterprise, described here: https://www.splunk.com/en_us/download/get-started-with-your-free-trial.html or you can request the Splunk Developer license from here (assuming you meet the requirements): https://dev.splunk.com/enterprise/dev_license/

Machine Setup

In this article I am using the following versions and VM setup:

Ubuntu Server 20.04.2 LTS
Splunk 7.3.0
100GB HDD dynamically allocated
4 GB RAM
4 CPU cores

I am using SSH to configure and install everything. It is assumed you have already prepped a Ubuntu Server VM with the above specs.

Splunk Enterprise Installation

We’re not going to cover the Splunk install in great detail here. There are plenty of guides out there already on how to install and fully configure Splunk Enterprise, the goal here is just to cover the basics and minimum requirements to get BOTS running. Navigate to https://www.splunk.com/page/previous_releases#x86_64linux, select your desired OS version of Splunk. Note: this is tested using 7.3.0. BOTS v3 is not compatible with Splunk 8.x.

Click the “Download Now” button for 7.3.0, then on the next screen click the (as shown above) “Command Line (wget)” link and copy the wget command. Connect to your Ubuntu server via SSH and copy this command in and hit enter.

Move the .deb file to your /tmp folder

$ mv splunk-7.3.0-657388c7a488-linux-2.6-amd64.deb /tmp && cd /tmp

Install Splunk 7.3

$ sudo dpkg -i splunk-7.3.0-657388c7a488-linux-2.6-amd64.deb

Start Splunk at boot, enter admin user and password. Hit space to go through the software license agreement, and press Y to accept. “Enable boot-start” will set the splunk service to auto start on boot.

$ sudo /opt/splunk/bin/splunk enable boot-start

Start the Splunk service

$ sudo service splunk start

Install Splunk Enterprise License:

Add new license via Settings > System: Licensing

Enable HTTPS (optional):

Navigate to Settings > System: Server settings > General Settings.

Ensure the Enable SSL (HTTPS) in Splunk Web is set to Yes

Restart Splunk, run:

$ sudo /opt/splunk/bin/splunk restart

Now use: https: //[IP address]:8000 to access Splunk Web

Ref: https://docs.splunk.com/Documentation/Splunk/7.0.3/Security/TurnonbasicencryptionwithSplunkWeb

BOTS Install

We have created a handy script that carries out the following steps and allows for a much smoother install:

Installs all the required Splunk ad-ons to run BOTS
Installs the CTF Scoreboard app and configures it
Downloads and extracts the BOTS v3.0 Dataset
Creates the Answers service account
Creates the BOTS CTF admin account
Creates the BOTS Competitor account

Clone the repository from here: https://github.com/runasroot/BOTSv3.git to your Ubuntu server:

$ git clone https://github.com/runasroot/BOTSv3.git

Navigate to the BOTSv3 folder:

$ cd BOTSv3

There a 2 main components, the install script itself and the “botsapps” folder which contains all the Splunks add-ons:

IMPORTANT NOTE: There are 2 apps not included in the Git repo , as the file size exceeds GitHubs limit of 25MB. Download the following app via the Splunk Download Link and copy to the “botsapps” folder before running the install script.

Splunk Stream*	7.2.0	https://splunkbase.splunk.com/app/1809/	splunk-stream_720.tgz
Splunk Security Essentials*	3.0.6	https://splunkbase.splunk.com/app/3435/	splunk-security-essentials_306.tgz

Apps not included in the Git repo due to file size limits

Once you have all the required apps in the “botsapps” (including the 2 manual downloads mentioned above) we are ready to run the install script. Your “botsapps” folder should look like this:

Run the install script from the BOTSv3 folder, first we have to make the script executable:

$ chmod +x install-botsv3_dev.sh

Then run it:

sudo ./install-botsv3_dev.sh

You will be asked for the following inputs:

enter your Splunk admin credentials to allow the install of the add-ons
CTF Answers Service account credentials – this will be created based on the details you enter. You should never need to use this account, it is used in the backend
CTF Admin account credentials – this will be created based on the details you enter. This account is used to run a BOTS event, including importing questions and answers
CTF Competitor account credentials – the script creates one competitor account to get you going based on the input you enter

Import the Splunk BOTS 3.0 Questions and Answers

Send a request to bots@splunk.com for BOTS 3.0 questions and answers. Once you receive a response, you should have 3 CSV files, a questions file, answers file, and a hints file.

Import Questions, Answers, and Hints

Questions: From Capture the Flag Admin app > Edit > Edit Questions

Then Import

Select file to import > select the ctf_questions.csv file

Same process for the Answers and Hints.

Answers: From Capture the Flag Admin app > Edit > Edit Answers > Import > Select file to import > select the ctf_answers.csv file
Hints: From Capture the Flag Admin app > Edit > Edit Hints > Import > Select file to import > select the ctf_hints.csv file

EULA Definition

A key step to allow user to submit answers is to define the EULA – End User License Agreement.

From Capture the Flag Admin app > Edit > Edit User License Agreements – the below settings worked for me. Edit according to your requirements:
_key =[auto generated] EulaContent = BOTS EULA, YOU MUST AGREE TO PLAY! EulaDefault = 1 Eulald = 001 EulaName = BOTS Training EULA

User/Team Management

Create Additional Users in Splunk

Now we have all the data, apps, questions, answers, and hints ready to go it’s time to create some extra users and teams. Create a new user in Splunk with roles: user; ctf_competitor. It’s also a good idea to set their default app as SA-ctf_scoreboard (Capture the Flag)

Setup Complete

Login with one of the competitor accounts you created either in the above step, or the account you created when installing BOTS. Accept the EULA and you’re ready to get going! The search index can be found here:

index=botsv3 earliest=0

Happy Splunking!

Troubleshooting

Check Splunk Web for Messages

Log into Splunk web and check the Messages drop down for any errors or notables:

Check the certs are in date

$ cd /opt/splunk/etc/auth/

$ for i in *.pem; do echo $i && openssl x509 -in $i -text | grep -A 2 Validity ; done

Check Logs

This will check the splunkkd log for the word “fail” – this could indicate a failure to start and should be inspected to determine if there is a problem.

cat /opt/splunk/var/log/splunk/splunkd.log | grep fail

Use this to check the mongo logs for the word “shutting” – this could be an indication an error is causing the KV store to shutdown with error code 100

cat /opt/splunk/var/log/splunk/mongod.log | grep shutting*

You can find a full list of logs here: https://docs.splunk.com/Documentation/Splunk/8.0.2/Troubleshooting/WhatSplunklogsaboutitself

Check the status of the KV Store:

/opt/splunk/bin/splunk show kvstore-status

Splunk Web

Using the Searching and Reporting App, use the following SPL to search for error messages:

index=_internal log_level=ERROR

KV store errors: https://splunkonbigdata.com/2019/07/03/failed-to-start-kv-store-process-see-mongod-log-and-splunkd-log-for-details/

Issue Tracking and Improvements

Any issues, improvements or feature requests for will be tracked in the associated Github Repo, found here: https://github.com/runasroot/BOTSv3_install/issues

References

Splunk Enterprise Free Trial: https://www.splunk.com/en_us/download/get-started-with-your-free-trial.html
Splunk BOTS v3: https://www.splunk.com/en_us/blog/security/botsv3-dataset-released.html
BOTS v3.0 dataset, questions, answers: https://github.com/splunk/botsv3
Scoreboard App: https://github.com/splunk/SA-ctf_scoreboard
Scoreboard Admin App: https://github.com/splunk/SA-ctf_scoreboard_admin
Slunk download: https://www.splunk.com/en_us/download.html
Splunk app install: https://docs.splunk.com/Documentation/AddOns/released/Overview/Singleserverinstall

Splunk BOTSv3 Install and Configuration

Introduction

Prerequisites

Machine Setup