Splunk Boss of the SOC (BOTS) is an awesome capture the flag event that I have had the privilege of attending. Splunk have now released everything required to run your own BOTS v3.0, more information can be found here: https://www.splunk.com/en_us/blog/security/botsv3-dataset-released.html. Splunk have opensourced everything required to run a Splunk BOTS v3.0 event including the dataset; the questions and answers; and the scoring server.
Motivation for this article: While installing and configuring the BOTS v3.0 “toolkit” I ran into a few issues and noticed there is no end-to-end guide to get this thing working. So I have put together this article along with a handy script to make the install easier and a lot faster!
- The BOTS v3.0 toolkit consists of:
- BOTS v3.0 dataset
- Questions CSV
- Answers CSV
- Hints file CSV
- Scoreboard Admin app
- Scoreboard app
A Splunk Enterprise license is required to run BOTS. You can utilise the a free 60 day trial of Splunk Enterprise, described here: https://www.splunk.com/en_us/download/get-started-with-your-free-trial.html or you can request the Splunk Developer license from here (assuming you meet the requirements): https://dev.splunk.com/enterprise/dev_license/
In this article I am using the following versions and VM setup:
- Ubuntu Server 20.04.2 LTS
- Splunk 7.3.0
- 100GB HDD dynamically allocated
- 4 GB RAM
- 4 CPU cores
I am using SSH to configure and install everything. It is assumed you have already prepped a Ubuntu Server VM with the above specs.
Splunk Enterprise Installation
We’re not going to cover the Splunk install in great detail here. There are plenty of guides out there already on how to install and fully configure Splunk Enterprise, the goal here is just to cover the basics and minimum requirements to get BOTS running. Navigate to https://www.splunk.com/page/previous_releases#x86_64linux, select your desired OS version of Splunk. Note: this is tested using 7.3.0. BOTS v3 is not compatible with Splunk 8.x.
Click the “Download Now” button for 7.3.0, then on the next screen click the (as shown above) “Command Line (wget)” link and copy the wget command. Connect to your Ubuntu server via SSH and copy this command in and hit enter.
Move the .deb file to your /tmp folder
$ mv splunk-7.3.0-657388c7a488-linux-2.6-amd64.deb /tmp && cd /tmp
Install Splunk 7.3
$ sudo dpkg -i splunk-7.3.0-657388c7a488-linux-2.6-amd64.deb
Start Splunk at boot, enter admin user and password. Hit space to go through the software license agreement, and press Y to accept. “Enable boot-start” will set the splunk service to auto start on boot.
$ sudo /opt/splunk/bin/splunk enable boot-start
Start the Splunk service
$ sudo service splunk start
Install Splunk Enterprise License:
Add new license via Settings > System: Licensing
Enable HTTPS (optional):
Login to your Splunk instance via http: //[IP address]:8000
Navigate to Settings > System: Server settings > General Settings.
Ensure the Enable SSL (HTTPS) in Splunk Web is set to Yes
Restart Splunk, run:
$ sudo /opt/splunk/bin/splunk restart
Now use: https: //[IP address]:8000 to access Splunk Web
We have created a handy script that carries out the following steps and allows for a much smoother install:
- Installs all the required Splunk ad-ons to run BOTS
- Installs the CTF Scoreboard app and configures it
- Downloads and extracts the BOTS v3.0 Dataset
- Creates the Answers service account
- Creates the BOTS CTF admin account
- Creates the BOTS Competitor account
Clone the repository from here: https://github.com/runasroot/BOTSv3.git to your Ubuntu server:
$ git clone https://github.com/runasroot/BOTSv3.git
Navigate to the BOTSv3 folder:
$ cd BOTSv3
There a 2 main components, the install script itself and the “botsapps” folder which contains all the Splunks add-ons:
IMPORTANT NOTE: There are 2 apps not included in the Git repo , as the file size exceeds GitHubs limit of 25MB. Download the following app via the Splunk Download Link and copy to the “botsapps” folder before running the install script.
|Splunk Security Essentials*||3.0.6||https://splunkbase.splunk.com/app/3435/||splunk-security-essentials_306.tgz|
Once you have all the required apps in the “botsapps” (including the 2 manual downloads mentioned above) we are ready to run the install script. Your “botsapps” folder should look like this:
Run the install script from the BOTSv3 folder, first we have to make the script executable:
$ chmod +x install-botsv3_dev.sh
Then run it:
You will be asked for the following inputs:
- enter your Splunk admin credentials to allow the install of the add-ons
- CTF Answers Service account credentials – this will be created based on the details you enter. You should never need to use this account, it is used in the backend
- CTF Admin account credentials – this will be created based on the details you enter. This account is used to run a BOTS event, including importing questions and answers
- CTF Competitor account credentials – the script creates one competitor account to get you going based on the input you enter
Import the Splunk BOTS 3.0 Questions and Answers
Send a request to email@example.com for BOTS 3.0 questions and answers. Once you receive a response, you should have 3 CSV files, a questions file, answers file, and a hints file.
Login in as the CTF admin account (as configured above) and select the “Capture the Flag Admin” app:
Import Questions, Answers, and Hints
Questions: From Capture the Flag Admin app > Edit > Edit Questions
Select file to import > select the ctf_questions.csv file
Same process for the Answers and Hints.
- Answers: From Capture the Flag Admin app > Edit > Edit Answers > Import > Select file to import > select the ctf_answers.csv file
- Hints: From Capture the Flag Admin app > Edit > Edit Hints > Import > Select file to import > select the ctf_hints.csv file
A key step to allow user to submit answers is to define the EULA – End User License Agreement.
From Capture the Flag Admin app > Edit > Edit User License Agreements – the below settings worked for me. Edit according to your requirements:
_key =[auto generated]
EulaContent = BOTS EULA, YOU MUST AGREE TO PLAY!
EulaDefault = 1
Eulald = 001
EulaName = BOTS Training EULA
Create Additional Users in Splunk
Now we have all the data, apps, questions, answers, and hints ready to go it’s time to create some extra users and teams. Create a new user in Splunk with roles: user; ctf_competitor. It’s also a good idea to set their default app as SA-ctf_scoreboard (Capture the Flag)
Login with one of the competitor accounts you created either in the above step, or the account you created when installing BOTS. Accept the EULA and you’re ready to get going! The search index can be found here:
Check Splunk Web for Messages
Log into Splunk web and check the Messages drop down for any errors or notables:
Check the certs are in date
$ cd /opt/splunk/etc/auth/
$ for i in *.pem; do echo $i && openssl x509 -in $i -text | grep -A 2 Validity ; done
This will check the splunkkd log for the word “fail” – this could indicate a failure to start and should be inspected to determine if there is a problem.
cat /opt/splunk/var/log/splunk/splunkd.log | grep fail
Use this to check the mongo logs for the word “shutting” – this could be an indication an error is causing the KV store to shutdown with error code 100
cat /opt/splunk/var/log/splunk/mongod.log | grep shutting*
You can find a full list of logs here: https://docs.splunk.com/Documentation/Splunk/8.0.2/Troubleshooting/WhatSplunklogsaboutitself
Check the status of the KV Store:
/opt/splunk/bin/splunk show kvstore-status
Login with your splunk credentials and you should see the reported status: ready, along with other information.
Using the Searching and Reporting App, use the following SPL to search for error messages:
KV store errors: https://splunkonbigdata.com/2019/07/03/failed-to-start-kv-store-process-see-mongod-log-and-splunkd-log-for-details/
Issue Tracking and Improvements
Any issues, improvements or feature requests for will be tracked in the associated Github Repo, found here: https://github.com/runasroot/BOTSv3_install/issues
- Splunk Enterprise Free Trial: https://www.splunk.com/en_us/download/get-started-with-your-free-trial.html
- Splunk BOTS v3: https://www.splunk.com/en_us/blog/security/botsv3-dataset-released.html
- BOTS v3.0 dataset, questions, answers: https://github.com/splunk/botsv3
- Scoreboard App: https://github.com/splunk/SA-ctf_scoreboard
- Scoreboard Admin App: https://github.com/splunk/SA-ctf_scoreboard_admin
- Slunk download: https://www.splunk.com/en_us/download.html
- Splunk app install: https://docs.splunk.com/Documentation/AddOns/released/Overview/Singleserverinstall