Across the Double Negative product line there are a number of vitally important 'behind the scenes' processes which allow our applications to function smoothly.

For example:

  • This blog is powered by the Ghost blogging platform. Ghost is a node application and as such behind the scenes it is run using npm: npm start.

  • Across our Android and iOS application lineup we utilise constantly running scripts to connect and process the sending of notifications to your Android and iOS devices.

  • There are a number of background scripts which validate and maintain data integrity across our network sites.

  • Revision.net is powered by a number of scripts which constantly poll and submit your scheduled posts to Twitter, and Facebook.

So what..?

It is very rare that I have a need to reboot our servers. Likewise the scripts running behind the scenes are fairly well tested and tend not to stop without reason. This knowledge has led to a false sense of security such that when I did need to reboot our servers it completely passed me by that I would also be taking this blog down, and stopping all of our behind the scenes processes.

Having made the mistake once I did not want to do it again, and as such I investigated how one could automatically restart all of our background processes on server reboot.

This is what I found.

Ubuntu

All of our servers are Ubuntu servers. I recently upgraded them to 16.04.1 LTS (Xenial Xerus).

One tool that I utilise heavily is screen. It allows me to have multiple terminal tabs in which I can execute different commands. For example, some tabs tail error logs, whilst others simply execute shell scripts.

This allows me to quickly switch to what I need when I need it, and allows me a good general overview of the vital server processes that are executing.

A good 'quick reference' of screen commands can be found here.

The above mentioned 'behind the scenes' processes run in tabs of a detached screen session, and as such when my server reboots what I really want to do is restart the screen session and all its contained tabs.

screen with a .sh script

The first thing that I did was I wrote a shell script that launched screen, created my tabs, and executed the appropriate commands in each.

That looked something like this:

#!/bin/sh

echo "uid is ${UID}"
echo "user is ${USER}"
echo "username is ${USERNAME}"

killall screen

screen -AdmS myshell
screen -S myshell -X screen -t 'iOS Notification Sender' sh /site/scripts/ios-notifications-sender.sh
screen -S myshell -X screen -t 'Android Notification Sender' sh /site/scripts/android-notifications-sender.sh

Executing that script from the command line did exactly as expected.

Credit where due.. I gained a lot of insight into working with screen from this answer on Stack Overflow. Give it a read.

It is also important to note that you need to make sure that any scripts that you are executing/logs that you are tailing are owned by the appropriate user/group, and have the correct permissions.

Now I just needed to execute the script on startup..

Upstart? No.. systemd

I made the mistake of rushing in without due consideration. I googled 'Ubuntu startup scripts' or equivalent and stumbled upon Upstart. I dived in without noting that in the latest versions of Ubuntu systemd is the default init daemon.

I don't want to confuse any potential readers by posting links to things that did not work. OK.. I will anyway :)

I did try a number of things that yielded less than ideal results. For example I tried this, and whilst it did execute my shell script, for some reason it was completely inconsistent in the screen tabs that it actually opened. That is to say that each reboot it opened a different (but incomplete) combination of tabs. It also seemed a little more complex (read low level) than it surely needed to be? Not ideal.

What did work

Eventually I discovered systemd and investigated its usage.

I found this answer which gave a general insight into configuration of systemd startup items.

I then worked backwards and read the documentation :)

I managed to get systemd to execute my shell script, but again I was getting inconsistent results - not every tab was being opened every time. I can only assume that this has something to do with permissions or timing (if anyone could share any insight, that would be appreciated), but the fact I got things working (see below) makes this somewhat bemusing.

I tried getting rid of the shell script entirely and executing my commands directly within the ExecStart configuration. This however.. did not work, and felt somewhat dirty.

The cleanest and best solution came to me in the end:

.screenrc

.screenrc is a configuration file for the initiation of a screen instance. If I hadn't rushed in and had taken the time to think about my approach in advance I would have surely used this methodology from the get go.

I wrote a .screenrc file and in it placed the tabs and code that I wanted executed in each. I then tested that it worked by instantiating a screen instance using the configuration file: screen -c ~/.screenrc.

This worked as expected.

Now it was simply a case of setting the ExecStart configuration command in my systemd service configuration to execute the above command. This 'native' approach executed without issue and creates all of the appropriate tabs each time.

Don't forget ghost

I rebooted my server a few times to test my setup. Everything worked as expected.

I had however completely forgotten about ghost. I had once again taken all our blogs down, and I had not put them back up. This is in itself an example use case of this type of startup script - to prevent downtime caused by oversight. Server configuration is complex, and it is certainly the case that you will forget something at somepoint. Make it so that it doesn't matter when you do.

Whilst researching some ghost specific queries I stumbled upon the crontab @reboot property. This might be of interest to you - it seems like another way of executing commands on reboot. I did not investigate this too much because:

  • It doesn't seem like a semantically logical place to restart background tasks.

  • It seems to have numerous issues, bugs, etc.

In the end I went down the same line as above - systemd.

This post by Chris Ebert outlines the specifics of setting up systemd for a ghost blog. It is well worth a read.