If you’re looking for an alternative to public URL scanners like urlscan.io or rescan.pro, then you’ve come to the right place. Hosting your own tool allows you to keep a personal record of the URLs you’ve scanned, keep the results from the scans for as long as you need, and by using a cloud Virtual Private Server (VPS) like the one’s I’ve mentioned in previous posts, it can minimise the risk of attribution from the adversary.
The tool I’m writing about is Lookyloo
This is an amazing and free URL scanner, which gives you the option to scan down to a depth of 9! It also gives you a nice graphical output. Here’s an example scan, with the target being my first blog post.
It’s important to note that I have largely used the guide in the README.md section of the github page. However, there were some aspects that I had issues with, so I decided to do a write up with a bit more detail. So, lets get started!
For this I recommend a minimum of 4 GB RAM on your VPS. This is due to the amount of memory that’s used when scanning the URLs. The 2 GB RAM will work, but you’ll find the memory fills up quite quickly and then crashes.
Lets start with some general housekeeping and installing Python 3 (this will not work on anything earlier than Python 3.6+). Lookyloo requires the installation of pipenv.
$ sudo apt update && sudo apt upgrade $ sudo apt install python3 $ sudo apt install python3-pip $ pip3 install pipenv
To ensure pipenv runs correctly, append the below to the bottom of your .bashrc.
$ nano ~/.bashrc export PYTHON_BIN_PATH="$(python3 -m site --user-base)/bin" export PATH="$PATH:$PYTHON_BIN_PATH"
Installation of Splash
You need a running splash instance, preferably on docker
$ sudo apt install docker.io $ sudo docker pull scrapinghub/splash $ sudo docker run -p 8050:8050 -p 5023:5023 scrapinghub/splash --disable-ui --disable-lua --disable-browser-caches # On a server with a decent abount of RAM, you may want to run it this way: $ sudo docker run -p 8050:8050 -p 5023:5023 scrapinghub/splash --disable-ui -s 100 --disable-lua -m 50000 --disable-browser-caches
Once you’ve confirmed this is working, I’d recommend running this in the background and on boot using the following commands. Running Splash in the background prevents it from outputting to standard out.
$ sudo systemctl enable docker.service $ sudo docker run --restart unless-stopped -d -p 8050:8050 -p 5023:5023 scrapinghub/splash --disable-ui --disable-lua --disable-browser-caches
You can confirm that Splash is running in the background by running the below, I will put example output in there too.
$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 0effdd8ea400 scrapinghub/splash "python3 /app/bin/sp…" 7 hours ago Up 6 hours 0.0.0.0:5023->5023/tcp, 0.0.0.0:8050->8050/tcp xenodochial_khayyam
$ cd /opt $ git clone https://github.com/antirez/redis.git $ cd redis $ git checkout [version] $ make # change [version] to the current version, example: $ git checkout 5.0
Installation of Lookyloo
$ git clone https://github.com/CIRCL/lookyloo.git $ cd lookyloo $ pipenv install $ pipenv shell # take note of the path, for example /home/.local/share/virtualenvs/lookyloo-Q2m2QRHy/bin, this will be used when updating your environment etc/systemd/system/lookyloo.service when running app in production $ exit $ echo LOOKYLOO_HOME="'`pwd`'" > .env
Run the app
$ pipenv run start.py
Navigate to 0.0.0.0:5100 and confirm this is working, if so, you may want to run this as a service in the background. This is covered in the next section.
Run the app in production
With a reverse proxy (Nginx)
$ pip install uwsgi
You have to configure the two following files:
Copy them to the appropriate directories using the below:
$ cp /opt/lookyloo/etc/systemd/system/lookyloo.service.sample etc/systemd/system/lookyloo.service $ /opt/lookyloo/etc/nginx/sites-available/lookyloo etc/nginx/sites-available/lookyloo
Here’s the template of what you’re lookyloo.service will look like, I have filled it out with example text. Remember under “Installation of Lookyloo” when I mentioned taking note of the virtual environment path? This is where you need it. Also, please note that the “user” field needs to match the “home” path of where you setup your pipenv. i.e. the /home/… would need to be that of the user “www-data” in the below example.
[Unit] Description=uWSGI instance to serve lookyloo After=network.target [Service] User=www-data Group=www-data Type=forking WorkingDirectory=/opt/lookyloo Environment=LOOKYLOO_HOME=/opt/lookyloo Environment=PATH=/home/.local/share/virtualenvs/lookyloo-Q2m2QRHy/bin:$PATH ExecStart=/home/.local/share/virtualenvs/lookyloo-Q2m2QRHy/bin/start.py ExecStop=/home/.local/share/virtualenvs/lookyloo-Q2m2QRHy/bin/stop.py [Install] WantedBy=multi-user.target
Next, create a symbolic link for nginx by running the following command:
$ sudo ln -s /etc/nginx/sites-available/lookyloo /etc/nginx/sites-enabled
If needed, remove the default site
$ sudo rm /etc/nginx/sites-enabled/default
Make sure everything is working:
$ sudo systemctl start lookyloo $ sudo systemctl enable lookyloo $ sudo nginx -t # If it is cool: $ sudo service nginx restart
You should now be able to navigate to Lookyloo by using http://<IP-or-domain>/