= Apache Camel Quarkus release process
:page-aliases: release-guide.adoc

The process is _mutatis mutandis_ the same as for the main Apache Camel repository - see the
xref:manual::release-guide.adoc[Release guide] page of the Camel documentation.

The process is oriented toward releasing the main branch, some adjustments might be needed when it comes to releasing maintenance branches.

Here, just a sketch of the repeating part, after you have performed the initial setup following the
xref:manual::release-guide.adoc[Camel Release guide].

== Pre-release tasks

For new major / minor version releases:

* When releasing after a recent Quarkus release, check for new releases of third party Quarkus extensions we depend on.
  They are listed in the https://github.com/apache/camel-quarkus/blob/main/pom.xml#L48-L61[Primary dependencies] section of the top level `pom.xml`.
  If there are new releases, upgrade the versions accordingly and run `mvn cq:sync-versions -N` from the root directory of the source tree.
  Note that some of the versions are mandated by quarkus, so camel-quarkus should use the same values as the quarkus-platform bom.
* Close the GitHub release https://github.com/apache/camel-quarkus/milestones[milestone] and assign any remaining issues that aren't covered by the release to the next milestone

For patch version releases:

* Create and immediately close an empty milestone for the patch version you plan to release. This is required for the Camel Website release metadata.

== Set up a release branch

For new major / minor version releases from the `main` branch:

[source,shell]
----
export VERSION=$(mvn help:evaluate -Dexpression=project.version -q -DforceStdout | sed 's|-SNAPSHOT||') # the version you are releasing, e.g. 0.1.0
export NEXT_VERSION=$(echo ${VERSION} | awk -F. -v OFS=. '{$NF++;print}')-SNAPSHOT # the next development iteration, e.g. 0.1.1-SNAPSHOT
export BRANCH=$(echo $VERSION | sed 's|.[0-9][0-9]*$|.x|') # the release branch, e.g. 0.1.x
git fetch upstream # upstream is git@github.com:apache/camel-quarkus.git
git checkout main # main is the branch from which you want to release
git reset --hard upstream/main # make sure you are in sync with upstream
git checkout -b $BRANCH
----

For patch releases where a release maintenance branch already exists:

[source,shell]
----
export BRANCH=3.20.x # The maintenance branch you want to release from
git checkout $BRANCH
git reset --hard upstream/$BRANCH
export VERSION=$(mvn help:evaluate -Dexpression=project.version -q -DforceStdout | sed 's|-SNAPSHOT||') # the version you are releasing, e.g. 0.1.0
export NEXT_VERSION=$(echo ${VERSION} | awk -F. -v OFS=. '{$NF++;print}')-SNAPSHOT # the next development iteration, e.g. 0.1.1-SNAPSHOT
----

== `release:prepare` and `release:perform`

With Camel Quarkus, adding `-Prelease` to `release:prepare` and `release:perform` is not necessary,
because all necessary profiles are set in the configuration of the `maven-release-plugin`.

The Maven command to release is as follows:

[source,shell]
----
mvn clean release:clean release:prepare -DreleaseVersion=$VERSION -DdevelopmentVersion=$NEXT_VERSION -B release:perform
----

In case `release:prepare` is successful but an issue occurs during `release:perform`, you can resume from the failing artifact to save some time:

* First find out from the console output, in which module the `release:perform` task failed.
  Say that there was an upload timeout in `camel-quarkus-openapi-java` module.
* Then locate the directory `mvn release:perform` used for checking out the tag and running the `deploy` goal.
  It is usually `<root_of_the_project>/target/checkout`:
+
[source,shell]
----
$ cd target/checkout
----
+
* And finally resume the `deploy` build as follows:
+
[source,shell]
----
mvn deploy -Dapache-release -DskipTests -Denforcer.skip -Dquarkus.build.skip -Dformatter.skip -Dimpsort.skip -rf :camel-quarkus-openapi-java
----

== Close the Apache staging repository

* Login to the https://repository.apache.org[Apache staging repository] using your Apache LDAP credentials.
* Click on `Staging Repositories`
* Then select `orgapachecamel-xxxx` in the list of repositories, where `xxxx` represents the staging repository number
* Click `Close` on the toolbar above
* This will close the repository from future deployments and make it available for others to view
* After that, a popup window is created, enter the name and version of the artifact being released in the `Description` field and finally click `Close`
* Make a note of the staging repository id as it'll be needed to upload sources to ASF dist/dev

== Update the Apache Camel Quarkus development subversion repository

Camel Quarkus provides some release scripts to push the required bits in development subversion repository.

In a first step, make sure your public key is present in the `KEYS` file. Add it, if it is not there yet using `gpg` command:

[source,shell]
----
export EMAIL= # the e-mail address in your key
gpg -k $EMAIL >> KEYS
gpg --armor --export $EMAIL >> KEYS
----

Then execute the `upload-source.sh` script, like below. `$NEXUS_STAGING_REPO_ID` is the Nexus staging repository id that you noted down from the previous step.

[source,shell]
----
cd release-utils/scripts/
./upload-source.sh $VERSION $NEXUS_STAGING_REPO_ID
----

This will take care of fetching the release source zip and SBOMs from https://repository.apache.org, creating SHA512 checksums and pushing the content to ASF `dist/dev/`

After the changes have been committed. You can check them by browsing to https://dist.apache.org/repos/dist/dev/camel/camel-quarkus/.

== Collect the performance regression report

Once the staging repository has been closed, run the `perf-regression` tool in order to compare the current staging against the previous released version.

It's important that the machine running the `perf-regression` tool be equally loaded during the run.
The simplest way would be to take a 1-hour slot at the end of the day, launch the tool on your main station and then lock the screen.

At this stage, one is probably working from the release branch.
The performance regression tool might be present in the `target/checkout` folder.
Otherwise, a full `mvn clean install -Dquickly` rebuild is needed prior to running the performance regression tool.
Another option is to switch to the main branch, where no rebuild should be needed.

For instance, running the tool to compare the `2.10.0` staging release against the `2.9.0` previous release would look as below:

[source,shell]
----
git checkout main
cd ../..
mvn clean install -Dquickly
cd tooling/perf-regression
java -jar target/quarkus-app/quarkus-run.jar -an 2.9.0 2.10.0
----

After approximately 1 hour, it should generate a report like below:

[source,shell]
----
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
 ---------------------------------------------------------------------------------------------------------
                 2.9.0  |       10m  |  16127.95 req/s [+0.00%]  |    9421.92 req/s [+0.00%]  |      OK  |
                2.10.0  |       10m  |  15334.57 req/s [-4.92%]  |    9044.55 req/s [-4.01%]  |      OK  |
----

Copy/paste the generated report into issue https://github.com/apache/camel-quarkus/issues/4012[CAMEL-QUARKUS-4012].

The `perf-regression` tool has some options that could come in handy if one needs to:

* Specify the camel-quarkus staging repository url, e.g. when doing the release from a machine and checking regression from another
* Specify the camel staging repository url, e.g. when the camel release is still under vote
* Perform regression tests in JVM mode only, e.g. on machine where `quarkus.native.container-build` could not be used
* Run longer tests, e.g. to confirm that a potential regression detected with 10 minutes long scenarios remains with 1h long scenarios
* Run shorter tests, e.g. to have quick feedback when trying to tune the `perf-regression` tool options

For more details about options, please run:

[source,shell]
----
java -jar target/quarkus-app/quarkus-run.jar --help
----

== Start an upstream vote

Send a mail to start the upstream vote.
There are plenty of mails on the dev mailing list to get https://lists.apache.org/thread/ntjrw585l5mjtrsvd6sn530boh43ccqo[inspiration] from.
For instance, one could search mail with subject below:

[source,shell]
----
[VOTE] Release Camel Quarkus 3.0.0
----

== Next version in Camel Quarkus main branch

The following steps apply only after a new major / minor release. They can be skipped for patch version releases.

If there are no substantial commits in the release branch, which need to get merged/cherry-picked to `main`, you can
perform this step right after creating the release branch. Otherwise, e.g. if there is a Camel upgrade in the release
branch which is not available on Maven Central yet, it is better to perform this step after the new Camel release is
available on the Central and after all relevant commits were merged/cherry-picked to `main`.

While in the release branch we set the `$NEXT_VERSION` to the next micro SNAPSHOT (e.g. when releasing `0.1.0`,
`$NEXT_VERSION` would be `0.1.1-SNAPSHOT`), in `main`, we typically set the next version to the next minor
SNAPSHOT (the next minor SNAPSHOT of `0.1.0` is `0.2.0-SNAPSHOT`).

[source,shell]
----
NEXT_RELEASE=... # e.g. 0.2.0
NEXT_SNAPSHOT="${NEXT_RELEASE}-SNAPSHOT"
git checkout "main"
git reset upstream/main
mvn release:update-versions -DautoVersionSubmodules=true -DdevelopmentVersion=$NEXT_SNAPSHOT -B
sed -i "s|<camel.quarkus.jvmSince>[^<]*</camel.quarkus.jvmSince>|<camel.quarkus.jvmSince>$NEXT_RELEASE</camel.quarkus.jvmSince>|" tooling/create-extension-templates/runtime-pom.xml
sed -i "s|<camel.quarkus.nativeSince>[^<]*</camel.quarkus.nativeSince>|<camel.quarkus.nativeSince>$NEXT_RELEASE</camel.quarkus.nativeSince>|" tooling/create-extension-templates/runtime-pom.xml
mvn cq:sync-versions -N
mvn clean install -DskipTests -Dquarkus.build.skip # to regen the Qute Camel component metadata + flattened boms
git commit -am "Next is $NEXT_SNAPSHOT"
# Send a pull request
----

== Close the upstream vote

After at least 72 hours has past, close the vote.
There are plenty of mails on the dev mailing list to get https://lists.apache.org/thread/shyfw2ltrxknnzb26nf71bc406lxvsp4[inspiration] from.
For instance, one could search mail with subject:

[source,shell]
----
[Result][VOTE] Release Apache Camel Quarkus 3.0.0
----

== Release the Apache staging repository after the vote

* Login to the https://repository.apache.org[Apache staging repository] using your Apache LDAP credentials.
* Click on `Staging Repositories`
* Then select `orgapachecamel-xxxx` in the list of repositories, where `xxxx` represents the staging repository number
* Click `Release` on the toolbar above
* This will release the repository and make artifacts generally available
* After that, a popup window is created, click `Close`

== Update the Apache Camel Quarkus distribution subversion repository

Once the staging repository has been released, the Apache Camel Quarkus distribution subversion repository should be updated.

In a first step, one would simply need to promote the deliverables from the development to the distribution svn repository by invoking the `promote-release.sh` script and passing the release version as an argument.

[source,shell]
----
export VERSION=... # The release version to promote e.g 3.18.0
cd release-utils/scripts/
./promote-release.sh $VERSION
----

In a second step, it is needed to check that useless versions are deleted from the svn repository.
If you are doing this for the first time you need to checkout the Apache Camel Quarkus distribution subversion repository:

[source,shell]
----
svn checkout 'https://dist.apache.org/repos/dist/release/camel' camel-releases-dist
----

In case you have performed the above step during some release in the past, you need to update your working copy:

[source,shell]
----
cd camel-releases-dist
svn update .
cd "camel-quarkus/$VERSION"
----

Then, ensure that only the latest release from each maintained branches is present (including active LTS versions).
Typically, when releasing e.g. 3.12.0, you may need to delete e.g. 3.11.0:

[source,shell]
----
cd ..
ls
svn remove 3.11.0
----

Review and commit the changes:

[source,shell]
----
cd ..
svn diff
svn commit -m "Cleaning inactive Camel Quarkus versions after $VERSION release"
----


[[release-announcement]]
== Write the release announcement blog post

Taking inspiration from previous release announcement blog posts, we need to write one for the release at hand.

Ideally, the release announcement should be written before submitting the xref:contributor-guide/release-guide.adoc#quarkus-platform[Platform pull request]
so that Quarkus team can mention our release in their announcement.
In that way the information will reach the whole Quarkus audience.

Please, keep in mind a few things when preparing the pull request for the camel-website:

* A release marker file should be added in https://github.com/apache/camel-website/tree/main/content/releases/q
* Marker files for LTS versions should include the end of life date, ala  'eol: date'
* The end of life date is the minimum date between camel LTS eol date and quarkus LTS eol date
* Documentation branches for LTS releases should be updated in https://github.com/apache/camel-website/blob/main/antora-playbook-snippets/antora-playbook.yml[antora-playbook.yml]
* We maintain documentation branches only for LTS versions and main


[[quarkus-platform]]
== Upgrade Camel Quarkus in Quarkus Platform

You can proceed with upgrading Camel Quarkus in Quarkus Platform
once the newly released artifacts are available on https://repo1.maven.org/maven2/org/apache/camel/quarkus/camel-quarkus-bom/[Maven Central].

[TIP]
====
Synchronization between Apache Maven repository and Maven Central may take hours.
You may find the `await-release` mojo of `cq-maven-plugin` handy if you need to upgrade Camel Quarkus in the Platform as soon as possible:

[source,shell]
----
cd camel-quarkus
mvn cq:await-release -Dcq.version=$(git describe --tags `git rev-list --tags --max-count=1`)
----

The mojo first lists the artifacts having `groupId` `org.apache.camel.quarkus` and the given `$VERSION`
from the local Maven repository and then checks that they are available in Maven Central.
As long as there are unavailable artifacts, the requests are re-tried with a (configurable) delay of 60 seconds.
====

NOTE: https://github.com/quarkusio/quarkus-platform[Quarkus Platform] hosts the metadata and Maven BOMs necessary for
https://{link-quarkus-code-generator}/[{link-quarkus-code-generator}] as well as for https://quarkus.io/guides/tooling[Quarkus tools].

* Clone Quarkus Platform unless you have done it in the past
+
[source,shell]
----
git clone git@github.com:quarkusio/quarkus-platform.git
----
* For patch releases `git checkout` the appropriate platform maintenance branch (e.g 3.20).
+
* Change `camel-quarkus.version` property in the Quarkus platform top level `https://github.com/quarkusio/quarkus-platform/blob/main/pom.xml#L54[pom.xml]` to the newly released version:
+
[source,shell]
----
cd quarkus-platform
export NEW_VERSION=... # the version you just released, e.g. 0.1.0
sed -i "s|<camel-quarkus.version>[^<]*</camel-quarkus.version>|<camel-quarkus.version>$NEW_VERSION</camel-quarkus.version>|" pom.xml
# Make sure that it worked
git status
----
+
* Review the list of skipped tests configured on the Camel member in `pom.xml` (E.g ones that have `<skip>true</skip>`). If any are known to have been fixed in the latest release, remove the entries so that they can be enabled again.
+
* Re-generate the BOMs
+
[source,shell]
----
./mvnw clean install -DskipTests
# ^ This will take a couple of minutes because it resolves
# every single dependency of every single extension included
# in the platform
#
# double check files are well formatted
./mvnw -Dsync


# Then commit the generated changes
git add -A
git commit -m "Upgrade to Camel Quarkus $NEW_VERSION"
----
+
* Run Camel Quarkus integration tests at least in JVM mode:
+
[source,shell]
----
cd generated-platform-project/quarkus-camel/integration-tests
mvn clean test
----
+
* If all tests are passing, prepare a pull request to the Platform
* Paste the URL of the xref:contributor-guide/release-guide.adoc#release-announcement[Camel Quarkus release announcement] to the description field of the pull request.
  Quarkus team will then be able to refer to our release announcement from their release announcement.
  The online availability of our release announcement should not block submitting the PR.
  You can paste the link even before the content is online, but make sure that it is available at the latest by the time when Quarkus team will publish their announcement.
* Optionally, if there are some extraordinary new features, fixes, etc.,
  you can tag the pull request with `release/noteworthy-feature` label and provide a short text about those features in the PR description that Quarkus team will pick into their announcement.


== Create a GitHub release

This will trigger sending a notification to folks watching the Camel Quarkus GitHub repository,
so it should ideally happen once the newly released artifacts are available on https://repo1.maven.org/maven2/org/apache/camel/quarkus/camel-quarkus-bom/[Maven Central].

The following needs to be done:

* Go to https://github.com/apache/camel-quarkus/releases[https://github.com/apache/camel-quarkus/releases].
* Click the tag you want to promote to a GitHub release
* Click "Create release from tag" button
* In the "New release" form:
  * Leave "Release title" empty
  * Choose the tag of the previous release from the "Previous tag" select field
  * Click the "Generate release notes" button. It should generate a summary all changes for the release.
  * Unless the release is genuinely the latest version (E.g. the highest major.minor.patch), uncheck option `Set as the latest release`
  * In addition, you can add links to blog posts if required. e.g. something like
+
[source,markdown]
----
Check the full [release announcement](https://camel.apache.org/blog/2021/06/camel-quarkus-release-2.0.0/)
----
+
  * Click the green "Publish release" button at the bottom

== Update the project changelog

In the Camel Quarkus main branch, create a new release heading in the `CHANGELOG.md` file. E.g `## 3.2.0`.
Under there, you can copy & paste the generated release notes Markdown (see previous step) from the GitHub release page into `CHANGELOG.md`.

You may also want to clean up the Markdown, for instance:

* Remove any excess sections like the `What's Changed` heading
* Fix up any abbreviated lines that end in `...`

== Upgrade and tag Examples

This section needs to be executed only when a Quarkus Platform release has been published.
It should be the case most of the time.
In this case, right after the newest Quarkus Platform becomes available on https://repo1.maven.org/maven2/io/quarkus/platform/quarkus-bom/[Maven Central]:

NOTE: For patch releases, the instructions that follow are mostly the same. Except that you must `git checkout` the desired maintenance branch (E.g 3.20.x), perform any updates on there and send PRs to the same upstream maintenance branch.
There is no need to interact with the `camel-quarkus-main` branch in this case.

* Make sure all https://github.com/apache/camel-quarkus-examples/pulls[PRs] against `camel-quarkus-main` branch are merged.
* The examples should use Quarkus Platform BOMs in the `main` branch. To set it do the following:
+
[source,shell]
----
NEW_PLATFORM_VERSION=... # E.g. 2.2.0.Final
git fetch upstream
git checkout camel-quarkus-main
git reset --hard upstream/camel-quarkus-main
mvn org.l2x6.cq:cq-maven-plugin:2.10.0:examples-set-platform -Dcq.quarkus.platform.version=$NEW_PLATFORM_VERSION
./refresh-dockerfiles.sh
git add -A
git commit -m "Upgrade to Quarkus Platform $NEW_PLATFORM_VERSION"
----
+
* Make sure that the tests are still passing:
+
[source,shell]
----
./mvnw-for-each.sh clean verify
----
+
* If everything works for you locally, open a PR to merge `camel-quarkus-main` to `main`
* Once the PR is merged, tag the `main` branch with the `$NEW_CQ_VERSION`:
+
[source,shell]
----
NEW_CQ_VERSION=... # the recent release of Camel Quarkus; e.g. 2.2.0
git checkout main
git fetch upstream
git reset --hard upstream/main
./mvnw-for-each.sh versions:set versions:update-child-modules -DnewVersion=$NEW_CQ_VERSION
# Update version labels in Kubernetes resources
./mvnw-for-each.sh process-sources
git add -A
git commit -m "Tag $NEW_CQ_VERSION"
git tag $NEW_CQ_VERSION
git push upstream main
git push upstream $NEW_CQ_VERSION
# Create a maintenance branch for the release unless this is a micro release
export BRANCH=$(echo $NEW_CQ_VERSION | sed 's|.[0-9][0-9]*$|.x|')
git checkout -b $BRANCH $NEW_CQ_VERSION
git push upstream $BRANCH
----
+
* Prepare the `camel-quarkus-main` branch for the next development iteration:
+
[source,shell]
----
NEXT_CQ_VERSION=... # The version used in the current Camel Quarkus main branch without the -SNAPSHOT suffix; e.g. 2.3.0
git checkout camel-quarkus-main
git reset --hard main
./mvnw org.l2x6.cq:cq-maven-plugin:2.10.0:examples-set-platform -Dcq.camel-quarkus.version=${NEXT_CQ_VERSION}-SNAPSHOT -Dcq.newVersion=${NEXT_CQ_VERSION}-SNAPSHOT
./mvnw-for-each.sh versions:update-child-modules -N
# Update version labels in Kubernetes resources
./mvnw-for-each.sh process-sources
git add -A
git commit -m "Next is $NEXT_CQ_VERSION"
git push upstream camel-quarkus-main --force-with-lease
----

== Further steps

In addition to the above, the following is needed:

* https://github.com/apache/camel-quarkus/issues/new[Create a ticket] asking a https://projects.apache.org/committee.html?camel[PMC member] to update the https://reporter.apache.org/addrelease.html?camel[Apache Committee Report Helper]. The ticket title could be as follows.
  Release: The Apache Committee Report Helper should be updated by a PMC member as camel-quarkus-X.Y.Z has been released on YYYY-MM-DD.

== Troubleshooting
=== Edge case involving SSH credential prompting
If you encounter an error related to `ksshaskpass`, such as:
----
[ERROR] The git-push command failed.
[ERROR] Command output:
[ERROR] fatal: cannot exec '/usr/bin/ksshaskpass': No such file or directory
[ERROR] fatal: could not read Username for 'https://gitbox.apache.org': terminal
----
In some environments, an SSH askpass helper may be invoked unexpectedly for
credential handling. If the helper is missing or incompatible with the
desktop/session configuration, Git operations can fail.
One way to prevent the use of an askpass helper is to disable it explicitly:
----
unset SSH_ASKPASS
export SSH_ASKPASS_REQUIRE=never
----
After applying these settings, the error may change to:
----
[ERROR] The git-push command failed.
[ERROR] Command output:
[ERROR] fatal: could not read Username for 'https://gitbox.apache.org': terminal prompts disabled
[ERROR] -> [Help 1]
----
At this point, Git is no longer attempting to use an askpass helper, but is also
unable to prompt for credentials interactively.
To restore normal credential prompting or authentication behavior, several
commands may be helpful, depending on the environment and Git configuration.
The following commands were executed during troubleshooting; the exact one that
resolved the issue is unclear, but they are documented here for reference:
----
ssh -T git@gitbox.apache.org
git config --global credential.helper store
ssh-add -l
mvn help:evaluate -Dexpression=env.SSH_AUTH_SOCK -q -DforceStdout
----