Submitting long running jobs

Author

These exercises where authored and tested by Kasper Thystrup Karstensen

Goal

Sometimes, when not using queing and job management systems, bioinformaticians may encounter submitting a job which runs for hours maybe even days.

In this exercise participants learns about how run long lasting jobs while being able to close the current active session.

We will use the sleep command, which locks the current session for X amount of seconds, to simulate long running tasks.

Level

Advanced

Part 1 – Send to background

Introduction

When executing commands, the terminal is (normally) occupied untill the command is finished with failure or success. While running, the command is running in the active session, this is now reffered to as a job. Lets explore whether it’s possible to do the job in the background and free up the terminal session.

First steps - How to send a job to the background

Commands can be send to the background by adding a & to the end of the command.

First lets try to run a ordinary slow command, and confirm that the active session is in fact occupied while running. Run sleep 15 and immidiately after run a new commands like e.g. echo hello, don’t wait for sleep to finish!

• Will it be executed?
  • If yes, when?

Lets try again with sleep 30, but this time sending the job to the background by adding the & sign to the end of the command. Once again try typing another command before it finishes.

• Will the commands be executed
  • If yes, when?

• How will we be notified when the job has finished?
  • Hint if you haven’t touched the terminal 30 seconds after running the command, press enter in the terminal

Part 2 – Active session

Introduction

We have tried to send jobs to the background and have confirmed it to finish up. Lets investegate what happens when we close down the active session while running longer scripts.

In order to check what happens if we terminate during a job being in the background, we need to get an output from the terminal after the sleep period is over, we achieve this by generating an empty file with the echo command.

This can be achieved by adding a && between two commands. && basically means “Run a command and once successfully finished, then run another”. NOTE: Don’t get confused by the & command it is not related to the && command!

Checking whether the job succeeds

Set a sleep timer to a few second (X) and generate a quick file to ensure that the job has completed using following command:

sleep X && touch yay.success

• Was the file created?

Terminating the active session during a job

Now that we have a way to check whether a job ends successfully, lets try sending it to the background and terminate the active session by cloing the terminal.

First, remove the yay.success file and run the command again (with ~15 seconds), but this time make sure to send it to the background (using & at the end) and then immediately close the terminal.

• Was the command successfull and the file created?
  • if Yes, Remove the yay.success

• Did sleep or touch fail or succeed?
  • Do you think either or both programs are dependent on an active session?
    • Do you have a guess of why/ why not?

How terminating the active session can affect certain jobs

Some jobs will finish wihtout requiring an active session, while others will stop dead in its track, if no active session is available.

echo works by printing to the active session lets try to see if it fails if the active session is interrupted. Run sleep for 10 seconds, send the entire command to the background and immediately close down the terminal:

sleep X && echo [something] && touch yay.success &

• Was the file created?
  • if yes, remove the yay.success

• Did the command succeed or fail?
  • Do you think echo is dependent on the active session?
    • Do you have a guess of why/ why not?

Part 3 – Long term jobs

Introduction

At this point we have learned that we can send jobs to the background and that some jobs does require an active session, while others may finish without. To esnure consistency in our long term runs, we would be interested in ensuring that jobs can finish successfully irrespectivelly of being dependent or independent of an active session.

Fortunenately for us, there is an option to redirecting the active session into a file. This can be achieved by using nohup in front of your command. Nohup redirects the session to a output file (which by default is called nohup.out).

Running our first long term job

Run the command once again for e.g. 120 seconds, by adding nohup in front of the entire command and make sure to send the job to the background. e.g. nohup [do stuff] &. Close - and restart the terminal, play around if you like. After two minutes has passed, check the following.

• Was the yay.success file generated?

• What is the contents of the nohup.out file?
  • What do you think the nohup.outis used for?

Final test

• Can you shut down the computer, while you have long term jobs running in the background, and still expect the job to succeed?;-)

Congratulations!!!

You have now learned how to send job to the background, gained a grain of insight into job assignment in the active session, as well as learned how to redirect information sent to the active session, into a file. Combining these things will aid you in setting up huge jobs, and stille be able to shut down your active session. When connecting to a server and setting up huge jobs, you can now avoid getting failed runs due to e.g. interrupted connection issues.