Pipeline HeavyJob Plugin

This plugin allows defining job weights for pipeline projects like HeavyJob Plugin does for freestyle projects.

По-русски

Usage

Pipeline:

nodeWithWeight(label: 'nodejs', weight: 2) {
    echo 'Run heavy tests'
}

Pipeline Declarative Syntax:

pipeline {
    agent {
        nodeWithWeight {
            label 'nodejs'
            weight 2
        }
    }
    stages {
        stage('Build') {
            steps {
                echo 'Run heavy build'
            }
        }
    }
}

Rationale

A typical big Jenkins setup consists of the main server with several worker nodes. Each node has several slots. By default one build occupies one slot. Some builds will wait in a queue if there are no more free slots available.

Each node has a limited amount of resources like CPU, RAM, or network bandwidth. Different builds require different amounts of these resources but each build occupies only one slot. That is why in some cases one node can be overloaded and another one can be almost idle.

There are several ways to solve this problem.

Suppose we have two types of builds: light and heavy. A heavy build requires twice as much resources as a light one. Suppose we have two worker nodes. Each node has enough resources to run two parallel light builds or one heavy build.

We want to run as many builds as possible but do it in such a way that none of the nodes will be overloaded.

Separate worker for heavy builds

The idea is to configure Jenkins setup this way:

• worker 1 should have 2 slots and perform only light builds;
• worker 2 should have 1 slot and perform light and heavy builds.

Pros: workers will never be overloaded. Cons: in some cases worker 2 will utilize only half of the available resources.

lockable-resources

Configure 4 resources named “slot” using lockable-resources. For a light build lock one “slot” resource:

lock(label: 'slot', quantity: 1) {
    echo 'Perform light build'
}

For a heavy build — lock 2 resources:

lock(label: 'slot', quantity: 2) {
    echo 'Perform heavy build'
}

Pros: workers will run as many builds as possible. Cons: in some cases workers can be overloaded.

Freestyle projects

It's possible to use freestyle projects with HeavyJob Plugin which solves the problem completely.

Pros: workers will run as many builds as possible, workers won't be overloaded. Cons: we can't use configuration as a code.

Conclusion

Each described solution has pros and cons and it will be very useful to have something like HeavyJob Plugin but for pipeline projects. As I've found in Jenkins Jira there are no such plugins. That is why Pipeline HeavyJob Plugin was developed.

Implementation details

This plugin implements nodeWithWeight step and nodeWithWeight agent based on it. The code of the plugin is based on the code of workflow-durable-task-step-plugin.