DeepTrace Installation Guide

Back to Docs

Table of Contents

Overview

DeepTrace is a collection of deep convolutional neural networks that pick arrival times in seismic data; we refer to each neural network as a “model”. DeepTrace comes pre-installed with a few models, and enables users to train models further with their own picked seimic data.

DeepTrace’s interface to seismic data is through Front Range Geoscience’s Phoenix Refraction Statics software. Phoenix handles importing and serving seismic data to DeepTrace. Phoenix is typically installed on a CPU cluster to run physics simulations, while DeepTrace is installed on GPU compute units. DeepTrace exposes its functionality to Phoenix as a network service.

This guide will cover the installation, usage, and maintenance of DeepTrace.

Prerequisites

Because DeepTrace utilizes GPUs to accelerate neural network performance, it is necessary first to install the relevant GPU system libraries. These include:

Note: CentOS 8 installations must append –override when installing CUDA 10.0 like so:

> sudo bash cuda_10.0.130_410.48_linux.run --override

Finally, modify the following environment variables in your .bashrc file:

export PATH=/bin:/usr/local/cuda-10.0/bin:/usr/local/cuda/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda-10.0/lib64:/usr/local/cuda/lib64:$LD_LIBRARY_PATH

Installing DeepTrace

DeepTrace comes bundled with the remainder of its software requirements, as well as some models to get users started. Since users may train additional models, DeepTrace requires some disk space to save new models to.

Please decide on a networked directory with plenty of free disk that all GPU nodes can access, to act as the shared storage path. Set:

export DEEPTRACEPATH=/the/path/you/chose

in your current shell.

Note: this is the path new models will be saved to. This is not the path that the DeepTrace binary is installed to.

Packaged with DeepTrace you will find a setup script installdeeptrace.sh - run this setup script by typing sh installdeeptrace.sh. This script does a few things:

You will be asked for your root password - we need this to write to /etc/systemd/system and /usr/local. You may verify that nothing nefarious is going on by checking the shell script less installdeeptrace.sh.

Note: we install the deeptrace service unit files to /etc/systemd/system by default

Sysadmins may wish to install the software to another directory - they should then modify every instance of /usr/local in the following files to their preferred directory:

installdeeptrace.sh
deeptracehead.service
deeptracemodel.service

The following command will achieve this:

CUSTOMDEEPTRACEPATH='/your/custom/path'; sed -i "s|ExecStart=/usr/local|ExecStart=${CUSTOMDEEPTRACEPATH}|g" deeptracehead.service deeptracemodel.service; sed -i "s|/usr/local|${CUSTOMDEEPTRACEPATH}|g" installdeeptrace.sh

Now that DeepTrace and the system service unit files are installed, run

sudo systemctl daemon-reload

to get access to the DeepTrace service.

Understanding DeepTrace Services

DeepTrace exposes its functionality to Phoenix via two services, a head service which keeps track of GPU nodes and the models they have loaded, and a model service which performs the work.

Only one head service is needed per site, while each separate GPU node needs to start its own model service.

Note: only one model service is needed per GPU node. If a single node has multiple GPUs, the model service will handle that automatically.

Users/sysadmins can examine the deeptracehead.service and deeptracemodel.service files to understand how this is achieved. The head service listens on ports (5000-5002 by default) for requests from Phoenix and model services. The model service needs to be told where to find the head service, and which ports to open itself to listen to Phoenix requests.

The default assumption is that all GPUs exist on a single node, and that node runs both the head and model service. Users should change the deeptracemodel.service file’s HEADADDRESS environment variable from localhost to the IP address running the head service if the model service is installed on another node.

Users can set the deeptrace services to start on boot by:

sudo systemctl enable deeptracehead.service
sudo systemctl enable deeptracemodel.service

Users manually start and stop the deeptrace services with

sudo systemctl {start/stop/restart} deeptrace{head/model}.service

Note: curly braces “{option1/option2}” denote that a single option should be chosen

Checking DeepTrace Logs

Because DeepTrace is installed as a system service, it utilizes systemd’s journaling to log usage.

To check the DeepTrace logs, access

journalctl -fu deeptracehead
journalctl -fu deeptracemodel

Connecting to Phoenix

When the DeepTrace head and model services have been started, users connect to DeepTrace from Phoenix using the head services IP and port.

HEADIPADDRESS:HEADPORT

The DeepTrace head service listens for Phoenix requests on the HEADPORT passed (default=5000). The head service listens for DeepTrace model requests on HEADPORT+1. To modify this value, change the variable in your /etc/systemd/system/deeptracehead.service:

#Change 5000 to whatever port you wish
Environment=HEADPORT=5000

Correspondingly, you will need to change the /etc/systemd/system/deeptracemodel.service:

#Change the follow lines to reflect new values
Environment=HEADPORT=5001
Environment=HEADADDRESS=localhost #localhost should change to HEADIP if head service is run on a separate node
Environment=MODELSTARTPORT=5005 #this is the beginning port the model processes start on

Phoenix defaults to 10.1.10.196:5000 as this is the value used internally at FRG. Change this value in Phoenix to your local GPU node’s values.

Ensure ports 5000-50XX are open in firewall settings:

firewall-cmd --zone=public --add-port=50XX/tcp --permanent

where 50XX runs through the number of available CPU cores on the GPU node, plus five. E.g. if there are 12 CPU threads available on the GPU machine, ports 5000-5017 should be opened.

Finally, reload the firewall settings:

firewall-cmd --reload

Troubleshooting

Monitor GPU utilization on the GPU nodes with

watch -n0.1 nvidia-smi

If the %utilization is not consistently hitting around 100%, try increasing the threads Phoenix is assigning to DeepTrace tasks.

DeepTrace reserves all available GPU memory after the first request to a GPU from Phoenix. If you do not wish to use a GPU to serve DeepTrace models, simply do not select it in Phoenix. Use sudo systemctl stop deeptracemodel to kill the deeptrace model service.

Please contact demoss@frontrangegeosciences.com with any questions, or for assistance with installing and running DeepTrace.