Looking at Logs#
Logs are extremely useful in piecing together what went wrong when things go wrong. They contain a forensic record of what individual pieces of software were doing before things went bad, and can help us understand the problem so we can fix it.
TLJH collects logs from JupyterHub, Traefik Proxy, & from each individual user’s notebook server. All the logs are accessible via journalctl. The installer also writes logs to disk, to help with cases where the installer did not succeed.
If you are providing a snippet from the logs to someone else to help debug a problem you might have, be careful to redact any private information (such as usernames) from the snippet first!
The JupyterHub installer writes log messages to
This is very useful if the installation fails for any reason.
JupyterHub is responsible for user authentication, & starting / stopping user notebook servers. When there is a general systemic issue with JupyterHub (rather than a specific issue with a particular user’s notebook), looking at the JupyterHub logs is a great first step.
sudo journalctl -u jupyterhub
This command displays logs from JupyterHub itself. See journalctl tips for tips on navigating the logs.
Traefik Proxy Logs#
traefik redirects traffic to JupyterHub / user notebook servers as necessary & handles HTTPS. Look at this if all you can see in your browser is one line cryptic error messages, or if you are having trouble with HTTPS.
sudo journalctl -u traefik
This command displays logs from Traefik. See journalctl tips for tips on navigating the logs.
User Server Logs#
Each user gets their own notebook server, and this server also produces logs. Looking at these can be useful when a user can launch their server but run into problems after that.
sudo journalctl -u jupyter-<name-of-user>
This command displays logs from the given user’s notebook server. You can get a list of all users from the “users” button at the top-right of the Admin page. See journalctl tips for tips on navigating the logs.
journalctl has a lot of options to make your life as an administrator
easier. Here are some very basic tips on effective
When looking at full logs (via
sudo journalctl -u <some-name>), the output usually does not fit into one screen. Hence, it is paginated with less. This allows you to scroll up / down, search for specific words, etc. Some common keyboard shortcuts are:
Arrow keys to move up / down / left / right
Gto navigate to the end of the logs
gto navigate to the start of the logs
/followed by a string to search for &
enterkey to search the logs from current position on screen to the end of the logs. If there are multiple results, you can use
nkey to jump to the next search result. Use
/to search backwards from current position
Ctrl + Cto exit
There are plenty of other commands & options to explore if you wish.
journalctlcommand to view live logging output that updates as new log lines are written. This is extremely useful when actively debugging an issue.
For example, to watch live logs of JupyterHub, you can run:
sudo journalctl -u jupyterhub -f