Shell scripts to delete old Jenkins builds

Requirements

A Jenkins server used by several teams frequently requires some periodic clean up. Old builds which are not used need to be deleted. Such task can easily be automated and set up to be periodically executed using crontab. This blog post explains several examples of shell scripts serving the purpose.

Deleting builds older than X days

This is a basic example of Jenkins jobs builds deletion script. We will delete all jobs builds modified more than 3 days ago. If the jobs hierarchy is flat (there are no folders), the script is just a one-line, elegant, but complex find command:

find /var/lib/jenkins/jobs/*/builds -maxdepth 1 -type d -mtime +3 -name "[0-9]*" -exec rm -rf {} \;

Explanation of the command elements:

• /var/lib/jenkins/jobs/*/builds - we are searching for all builds of all jobs located in the Jenking home directory, for example /var/lib/jenkins. The asterisk * replaces all job folder names,
• -maxdepth 1 means that we are searching for direct children of builds directory,
• -type d means that we are searching for directories,
• -mtime +3 selects items modified more than 3 days ago,
• -name "[0-9]*" means that we are searching for items with names beginning with a number. This part is added in order not to delete the actual builds directory,
• -exec rm -rf {} \; force deletes all found items.

Deleting builds older than X days with size greater than Y kilobytes

As the build directory usually contains more than one file, and in order to calculate the directory total size we need to sum up the sizes of all files and directories inside, in this example we will not be able to use only the find command. We will be using a for-loop.

We will reuse a part of requirements from the basic example. We will delete all jobs builds modified more than 3 days ago.

In order to make the example more interesting, we will assume that the jobs hierarchy has two levels, meaning that at the root level we may have jobs as well as folders containing jobs. Also, we will take into account jobs containing spaces in their names.

Additionally, we will log each deletion, specifying the name and the size of each deleted folder.

#!/bin/bash
#Yaroslav Grebnov, grebnov@gmail.com

log_file=delete_jenkins_jobs-$(date +%Y-%m-%d)
old_IFS=$IFS
IFS=$'\n'

#Delete all jobs builds older than 3 days, with size greater than 500000 kilobytes
for f in $(find /var/lib/jenkins/jobs/*/builds -maxdepth 1 -type d -mtime +3 -name "[0-9]*" 2>/dev/null) $(find /var/lib/jenkins/jobs/*/*/*/builds -maxdepth 1 -type d -mtime +3 -name "[0-9]*" 2>/dev/null); do
  if [[ $(du -sk -- $f | awk '{print $1}') -gt 500000 ]]; then
    echo "Deleted: $(du -sk -- $f)" >> /var/log/$log_file
    rm -rf $f
  fi
done

IFS=$old_IFS

Explanation of the script elements:

log file

log_file=delete_jenkins_jobs-$(date +%Y-%m-%d) creates a variable containing the log file name. The name is based on the formatted timestamp

IFS

Three lines concerning IFS address the requirement of possible space characters in job names. In for-loop we are using an array created from the output of the find command. By default, the IFS variable contains the space character, so the found directories containing spaces in their names can be incorrectly split into two entries. For example, the directory /var/lib/jenkins/jobs/some job/builds/1 with the default IFS will be represented as two elements in the array in for-loop: /var/lib/jenkins/jobs/some and job/builds/1. In order to represent a job with space characters in its name as one element in the array in for-loop, we modify the IFS, so that only new line character is considered as a separator. This change is revereted back after the loop

for-loop

We are using a for-loop of the following format:

for f in <array>; do
  <loop body>
done

builds array used in for-loop

The array is created from the concatenated output of the two find commands.

We have two commands because we search for builds in paths using two patterns:

• /var/lib/jenkins/jobs/*/builds - for jobs defined at root level. Path to a root-level job named someJob will be /var/lib/jenkins/jobs/someJob,
• /var/lib/jenkins/jobs/*/*/*/builds - for jobs located in folders. Path to the job someJob located in the someFolder folder will be /var/lib/jenkins/jobs/someFolder/jobs/someJob. As you can see, the jobs part of the path is repeated twice, which is how it is implemented in Jenkins. This repetion is the reason why we have three asterisks in the path pattern.

An imported note about the find commands output concatenation. We need to have a space character between them. In case we do not put that space, the first element of the second output will be added to the last element of the first output and this pair will be considered as one element of the array, like this: “firstLastsecondFirst” instead of: “fisrtLast”, “secondFirst”.

find command

Compared to the basic exmaple, we have one more modification. There is 2>/dev/null at the end. This part serves for ignoring errors in examining the directories structure. An example of such error may be absense of permission for the current user to access some directory. Without the 2>/dev/null part, error messages will be added to the for-loop array, which is incorrect, because we want only paths to the directories there.

compare directory size with threshold value

According to the requirements, we delete directories with size greater than 500000 kilobytes. In order to compare a given directory size with the threshold value, we use an if condition:

if [[ <directory_size> -gt 500000 ]]; then
    <deletion>
fi

where -gt means greater than.

get directory size

In order to get an f directory size for further comparison, we use two piped commands:

• du -sk -- $f - a du command with two flags: 1) s meaning summarize, that is take total size of all elements inside and 2) k meaning 1 in output is equal to 1 kilobyte,
• awk '{print $1}' - an awk command which prints the first word from the du command output. The first word is the actual size in kilobytes.

log deleted directory

echo "Deleted: $(du -sk -- $f)" >> /var/log/$log_file adds a record with the deleted directory name and size to the end of the log file. >> means append. $log_file was defined at the beginning of the script.

delete directory

rm -rf $f deletes a directory with name from the f variable.

Deleting builds older than X days but skip the ones marked to be kept forever

Jenkins has functionality to keep no more than N builds for some job. Also, we can mark a build to be ‘kept forever’ by checking the corresponding checkbox in UI. In this way, Jenkins will skip such build during the deletion process.

In the automatic jobs builds deletion script, we would like to take this functionality into consideration. So, the third example demonstrates how to exclude the builds marked to be kept forever from the deletion process.

In order to keep the things simple, we will remove the requirements of non-flat jobs hierarchy and the build directory size.

#!/bin/bash
#Yaroslav Grebnov, grebnov@gmail.com

log_file=delete_jenkins_jobs-$(date +%Y-%m-%d)
old_IFS=$IFS
IFS=$'\n'

#Delete all jobs builds older than 3 days except the ones marked to be kept forever
for f in $(find /var/lib/jenkins/jobs/*/builds -maxdepth 1 -type d -mtime +3 -name "[0-9]*" 2>/dev/null); do
  if [[ $(grep "<keepLog>true</keepLog>" $f/build.xml 2>/dev/null | wc -l) -eq 0 ]]; then
    echo "Deleted: $(du -sk -- $f)" >> /var/log/$log_file
    rm -rf $f
  fi
done

IFS=$old_IFS

In explanation we will address only the script new elements compared to the previous example:

skip builds marked to be kept forever

This requirement is implemented by checking whether the result of piping two commands: grep and wc: $(grep "<keepLog>true</keepLog>" $f/build.xml 2>/dev/null | wc -l) is equal to zero.

Let’s examine the grep "<keepLog>true</keepLog>" $f/build.xml 2>/dev/null part. A job build configuration in Jenkins is stored internally in a file build.xml located in that build directory. The builds which are marked as the ones to be kept forever have keepLog element equal to true. In other words, such build build.xml file contains text <keepLog>true</keepLog>. So, we are using grep command to search for that text in the build.xml file. By default, we are considering the builds as not marked to be kept forever. That is why, we ignore all grep command execution errors using this part: 2>/dev/null.

grep command execution output serves as input for the wc -l command which counts the number of lines in its input.

So, the command grep "<keepLog>true</keepLog>" $f/build.xml 2>/dev/null | wc -l outputs the number of occurences of <keepLog>true</keepLog> text in build.xml file. Next, we check whether this number is equal to zero. In other words, we consider a build as the one marked to be kept forever if there is a text <keepLog>true</keepLog> in build.xml file located in that build directory.

In case you have a question or a comment concerning this post, please send them to: