Jenkins LTS 2.528.3 Release Checklist

by Alex Johnson 38 views

Welcome, Jenkins enthusiasts and administrators! Preparing a new Long-Term Support (LTS) release of Jenkins is a significant undertaking, ensuring stability and reliability for countless users worldwide. This guide dives deep into the LTS 2.528.3 release checklist, providing a clear, step-by-step process for a smooth and successful launch. We'll cover everything from the initial prep work to the final release announcements, ensuring you have all the information needed to contribute to or understand this critical process. The Jenkins LTS release cycle is designed to provide a stable, predictable, and well-tested version of Jenkins, making it the go-to choice for production environments. Each LTS release is supported for an extended period, receiving critical bug fixes and security patches, which is why adhering to a rigorous checklist is paramount. This detailed walkthrough is for anyone involved in the Jenkins release engineering process, from release leads to contributors, and even curious users who want to understand the mechanics behind their favorite automation server.

Release Lead

The Release Lead is the linchpin of the entire LTS release process. This person isn't necessarily the one performing every single task but is responsible for ensuring that all steps are completed on time and to the required standard. Think of them as the conductor of an orchestra, ensuring every section plays its part harmoniously. To ensure a fresh perspective and shared responsibility, this role typically rotates between LTS releases. This prevents any single individual from being overburdened and fosters broader community involvement in the release engineering effort. The release lead acts as the primary point of contact for the release, coordinating efforts across different teams and individuals, and making critical decisions when necessary. Their main goal is to shepherd the release candidate through testing, address any emerging issues, and ultimately oversee the final public release. The success of an LTS release hinges significantly on the dedication and organization of the Release Lead. They must have a clear understanding of the Jenkins release process, be adept at communication, and be prepared to dedicate the necessary time and attention to guide the release to fruition. This role is crucial for maintaining the integrity and timely delivery of Jenkins LTS.

Preparation is Key: The Pre-Release Workflow

Before we can even think about a release candidate, a substantial amount of prep work needs to be meticulously completed. This phase is all about setting the stage for a stable and secure LTS release. It begins with establishing the LTS baseline, a foundational decision made after thorough discussion on the Jenkins developers mailing list. This baseline determines the starting point for the new LTS line, and sometimes, if the last release of the preceding LTS line was a security release, the matching weekly release might be chosen as the new LTS baseline for simplicity and consistency. For instance, if 2.462.3 LTS and 2.479 were security releases, using 2.479 as the baseline instead of 2.477 simplifies the process. Following this, several critical repositories need to be updated. The jenkinsci/jenkins repository requires a new release branch (e.g., stable-2.387) to be created or updated, often using the helpful init-lts-line script. Similarly, the jenkins-infra/release and jenkinsci/packaging repositories need their respective release branches (stable-2.387) created or updated. This involves modifying environment files (profile.d/stable) and packaging scripts (Jenkinsfile.d/core/package) to align with the new release version, as detailed in the stable documentation. The jenkinsci/bom (Bill of Materials) also needs an update, ensuring its bom-weekly version number is testing the release line's base or newer. A crucial step involves reviewing recent security advisories to identify fixes that need to be backported to the LTS line, ensuring corresponding Jira issues exist. All identified lts-candidate issues in Jira and GitHub pull requests are reviewed, and tickets must be resolved. An announcement email is then sent to the jenkinsci-dev mailing list, using a default template but customized with the specific LTS version, release date, and Jira URLs. Jira labels are updated for lts-candidate issues, marking them as fixed or rejected. The actual backporting of changes occurs next, using git cherry-pick -x $commit after locating commits via Jira IDs (sometimes requiring manual effort). Once backported, a pull request is opened with the into-lts label, summarizing the changes and listing relevant Jira candidates and potential candidates from the release and packaging repositories. To catch regressions, a pull request is opened against the acceptance test harness and plugin compatibility test (jenkinsci/bom) to ensure the backported changes don't introduce new problems. Finally, the LTS changelog and LTS upgrade guide are prepared, usually by the docs team, with requests made in the Gitter channel. This extensive preparation phase is vital for a stable and secure release.

The Release Candidate (RC) Phase: Testing and Validation

With the preparatory work complete, we move into the RC creation phase. This is where we generate a Release Candidate, a near-final version of the LTS release, allowing for final testing and validation before the official public announcement. The first major milestone here is merging the backporting PR into the jenkinci/jenkins repository. It's crucial to use a merge commit and not squash during this process. This preserves the commit history, which is important for tracking changes. Once merged, the next step is to retrieve the URL for the Release Candidate build. This is typically found in the commit status of the last build on the stable branch within Jenkins Incrementals Publisher. A passing build is essential for this. You'll navigate to the jenkins-war URL to get the specific .war file location, which looks something like https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/main/jenkins-war/2.387.1-rc32701.b_06d9cef554c/jenkins-war-2.387.1-rc32701.b_06d9cef554c.war. If the incrementals are broken, a manual build and deploy using mvn -e clean deploy -DskipTests=true might be necessary. Following the successful retrieval of the RC build, a pre-release GitHub release is published for the jenkinsci/jenkins repository, mirroring the structure of previous RC tags (e.g., jenkins-2.387.1-rc). Currently, RCs don't have a dedicated changelog, but this is a point to be aware of. An important check is to confirm that the automatic announcement of the RC has been sent to the jenkinsci-dev mailing list and the community forums. If it hasn't, the Release Lead must manually compose and send this announcement. Close coordination with the security team is vital during this phase. It must be confirmed that no security update is planned concurrently. If a security update is planned, the checklist and timeline might need revision, especially after any public pre-announcement to the jenkinsci-advisories mailing list. For a new LTS baseline's first point release (the .1 release), there's an extra layer of verification if recent security advisories required backporting. In such cases, these advisories need to be updated to include the new .1 LTS release as a fix version, and the warnings.json metadata needs updating to exclude the .1 release. The Jenkins security team must also be informed to update CVE metadata accordingly. This RC phase is critical for catching any last-minute issues before the official release, ensuring the final product is as robust as possible.

The Final Countdown: LTS Release Day

Release day is here! This is the culmination of all the hard work, and the LTS release process kicks into high gear. The day typically starts with the publication of the changelog. This is done one day prior to the release, especially if a security update is involved, allowing users to prepare. The official announcement of the LTS release process start is made in the #jenkins-release:matrix.org channel. The core of the release process involves launching a job on release.ci.jenkins.io. This automated job orchestrates the building and distribution of the new LTS version. However, if a security release for Jenkins is planned concurrently, this job is not launched. This distinction is crucial. Once the release job is launched, the Release Lead must manually review and approve the child release job. This involves carefully checking the 'Plan' stage, often comparing it with the configurations of previous stable line releases to ensure consistency and correctness. A special consideration arises for the first release of a new LTS line: the packaging job might fail on its initial run. This is expected and can be handled by either running the packaging job once and canceling it before the primary release job, or by accepting that the packaging job will need a second run after its initial failure. Approximately three to four hours after the release job begins, the Release Lead must again manually review and approve the child packaging job. This meticulous manual oversight at critical junctures ensures that the automated processes align with the release goals and quality standards. The entire release process, from job launch to completion, typically takes about 3 to 4 hours for the release job and around 30 minutes for the packaging job. Once these jobs complete successfully, the validation begins. First, check that the LTS changelog is visible on the Jenkins downloads site. Following this, a GitHub release is published for the jenkinsci/jenkins repository, explicitly referencing the LTS changelog and version (e.g., jenkins-2.387.1). Verification extends to ensuring all packages are available on the Datadog page. Crucially, the Debian installer acceptance test and the Red Hat installer acceptance test must be passing. During these tests, it's important to check the console logs to confirm that the correct release package (e.g., containing 2.387) was used; if not, the tests might need to be relaunched. All resolved Jira issues included in the release are then updated to reflect their final state and Released As version, cross-referencing with the changelog. Post-release tasks include updating the jenkins.version in the plugin BOM to the newly released version via a pull request. A tag matching the LTS release is created in the jenkinsci/docker repository, and a corresponding GitHub release is published, followed by confirmation that the Docker images are available on Docker Hub. The jenkins-dependency-updater bot's PR in the jenkinsci/helm-charts repository needs to be merged. A helpdesk ticket is created to update various ci.jenkins.io related services to the new LTS release. Finally, an email is sent out to solicit the next release lead, providing dates for future releases found on the Jenkins calendar. This comprehensive final stage ensures the LTS release is not only built but also correctly deployed, documented, and communicated to the community.

Important Resources

For further details and continuous learning about the Jenkins release process, please refer to these trusted resources: