Setting up Lookyloo

If you’re looking for an alternative to public URL scanners like urlscan.io or rescan.pro, then you’ve come to the right place. Hosting your own tool allows you to keep a personal record of the URLs you’ve scanned, keep the results from the scans for as long as you need, and by using a cloud Virtual Private Server (VPS) like the one’s I’ve mentioned in previous posts, it can minimise the risk of attribution from the adversary.

The tool I’m writing about is Lookyloo

This is an amazing and free URL scanner, which gives you the option to scan down to a depth of 9! It also gives you a nice graphical output. Here’s an example scan, with the target being my first blog post.

It’s important to note that I have largely used the guide in the README.md section of the github page. However, there were some aspects that I had issues with, so I decided to do a write up with a bit more detail. So, lets get started!

For this I recommend a minimum of 4 GB RAM on your VPS. This is due to the amount of memory that’s used when scanning the URLs. The 2 GB RAM will work, but you’ll find the memory fills up quite quickly and then crashes.

Initial Setup

Lets start with some general housekeeping and installing Python 3 (this will not work on anything earlier than Python 3.6+). Lookyloo requires the installation of pipenv.

$ sudo apt update && sudo apt upgrade
$ sudo apt install python3
$ sudo apt install python3-pip
$ pip3 install pipenv

To ensure pipenv runs correctly, append the below to the bottom of your .bashrc.

$  nano ~/.bashrc 
export PYTHON_BIN_PATH="$(python3 -m site --user-base)/bin"
export PATH="$PATH:$PYTHON_BIN_PATH"

Installation of Splash

You need a running splash instance, preferably on docker

$ sudo apt install docker.io
$ sudo docker pull scrapinghub/splash
$ sudo docker run -p 8050:8050 -p 5023:5023 scrapinghub/splash --disable-ui --disable-lua --disable-browser-caches
# On a server with a decent abount of RAM, you may want to run it this way:
$ sudo docker run -p 8050:8050 -p 5023:5023 scrapinghub/splash --disable-ui -s 100 --disable-lua -m 50000 --disable-browser-caches

Once you’ve confirmed this is working, I’d recommend running this in the background and on boot using the following commands. Running Splash in the background prevents it from outputting to standard out.

$ sudo systemctl enable docker.service
$ sudo docker run --restart unless-stopped  -d -p 8050:8050 -p 5023:5023 scrapinghub/splash --disable-ui --disable-lua --disable-browser-caches

You can confirm that Splash is running in the background by running the below, I will put example output in there too.

$ docker ps
CONTAINER ID        IMAGE                COMMAND                  CREATED             STATUS              PORTS                                            NAMES
0effdd8ea400        scrapinghub/splash   "python3 /app/bin/sp…"   7 hours ago         Up 6 hours          0.0.0.0:5023->5023/tcp, 0.0.0.0:8050->8050/tcp   xenodochial_khayyam

Install redis

$ cd /opt
$ git clone https://github.com/antirez/redis.git
$ cd redis
$ git checkout [version]
$ make

# change [version] to the current version, example: $ git checkout 5.0

Installation of Lookyloo

$ git clone https://github.com/CIRCL/lookyloo.git
$ cd lookyloo
$ pipenv install
$ pipenv shell
# take note of the path, for example /home/.local/share/virtualenvs/lookyloo-Q2m2QRHy/bin, this will be used when updating your environment etc/systemd/system/lookyloo.service when running app in production
$ exit
$ echo LOOKYLOO_HOME="'`pwd`'" > .env

Run the app

$ pipenv run start.py

Navigate to 0.0.0.0:5100 and confirm this is working, if so, you may want to run this as a service in the background. This is covered in the next section.

Run the app in production

With a reverse proxy (Nginx)

$ pip install uwsgi

Config files

You have to configure the two following files:

etc/nginx/sites-available/lookyloo
etc/systemd/system/lookyloo.service

Copy them to the appropriate directories using the below:

$ cp /opt/lookyloo/etc/systemd/system/lookyloo.service.sample etc/systemd/system/lookyloo.service
$ /opt/lookyloo/etc/nginx/sites-available/lookyloo etc/nginx/sites-available/lookyloo

Here’s the template of what you’re lookyloo.service will look like, I have filled it out with example text. Remember under “Installation of Lookyloo” when I mentioned taking note of the virtual environment path? This is where you need it. Also, please note that the “user” field needs to match the “home” path of where you setup your pipenv. i.e. the /home/… would need to be that of the user “www-data” in the below example.

[Unit]
Description=uWSGI instance to serve lookyloo
After=network.target

[Service]
User=www-data
Group=www-data
Type=forking
WorkingDirectory=/opt/lookyloo
Environment=LOOKYLOO_HOME=/opt/lookyloo
Environment=PATH=/home/.local/share/virtualenvs/lookyloo-Q2m2QRHy/bin:$PATH
ExecStart=/home/.local/share/virtualenvs/lookyloo-Q2m2QRHy/bin/start.py
ExecStop=/home/.local/share/virtualenvs/lookyloo-Q2m2QRHy/bin/stop.py


[Install]
WantedBy=multi-user.target

Next, create a symbolic link for nginx by running the following command:

$ sudo ln -s /etc/nginx/sites-available/lookyloo /etc/nginx/sites-enabled

If needed, remove the default site

$ sudo rm /etc/nginx/sites-enabled/default

Make sure everything is working:

$ sudo systemctl start lookyloo
$ sudo systemctl enable lookyloo
$ sudo nginx -t
# If it is cool:
$ sudo service nginx restart

You should now be able to navigate to Lookyloo by using http://<IP-or-domain>/

Follow @herringtonalex