Tools

Managing p2 Update Sites

To simplify and streamline the management of p2 update sites, JustJ provides the org.eclipse.justj.p2.manager application for the following purposes:

To promote a p2 repository, along with select associated artifacts, from the continuous integration server to the download server.
To manage the lifecyle of a collection of p2 repositories on the download server.
To generate integrated update site documentation.

This infrastructure is used to generate the update sites for JustJ's JREs as well as the update site for JustJ's tools, including the JRE generator and the p2 manager tools. It is being exploited by a growing number of Eclipse projects 🔗. It can be directly used from within a pom.xml 🔗 to manage the updates sites as an integrated part of the build.

The Anatomy of a Managed Update Site

The following is the general folder structure of the set of p2 repositories managed by the org.eclipse.justj.p2.manager application.

2023-12 - Only present if -simrel-alias is used. It composes either nightly/latest, if that contains logical version 4.29.0, milestone/latest, if that is present and contains logical version 4.29.0, or release/4.29.0, if that is present. The logical version is determined by -version-iu or -version-iu-pattern

release - Present if there is at least one release. It composes all the releases.

latest - Always present. It composes the latest release.
4.29.0 - A release with a folder name based on the logical version as specified by -version-iu or -version-iu-pattern.

milestone - Present if there is at least one milestone. It composes all the milestones.

latest - Always present. It composes the latest milestone.
S20231111000 - A milestone with a folder name based on the -timestamp. This is automatically deleted if and when there is a milestone published with a higher logical version, as specified by -version-iu or -version-iu-pattern, than this one's logical version.

nightly - Present if there is at least one nightly. It composes all the nightlies.

latest - Always present. It composes the latest nightly.
N20231110000 - A nightly with a folder name based on the -timestamp. This is automatically deleted if and when this is older than the -retain number of other nightly builds.

Application Arguments for `org.eclipse.justj.p2.manager`

The org.eclipse.justj.p2.manager application recognizes the following command line arguments:

-quiet - Whether to avoid printing detailed logging information about the processing steps of the application.

-type - The type of the build.

<built-type> - A build type as specified below. Default nightly.
- nightly - This will promote the update site as a nightly build. The folder name will be of the form NyyyyMMddHHmm. If more than the -retain number of nightly repositories are present, the older ones will be deleted.
- milestone - This will promote the update site as a milestone build. The folder name will be of the form SyyyyMMddHHmm. Any existing milestone site that has a lower logical version than this newly promoted site will be deleted. Promotion will fail if there is not already at least one nightly build present in the update site.
- release - This will not promote the newly built site as a release build. Instead it will promote (mirror) the latest milestone build as a release build. I.e., the release update site will contain byte-for-byte the same artifacts as the latest milestone. The folder name will be of the form x.y.z where the -version-iu or -version-iu-pattern determines the version. Promotion will fail if there is not already at least one milestone build—the one being promoted—present in the update site.

-retain - The number of nightly builds to retain. Older nightly builds will automatically be deleted.

<integer> - A positive integer. Default 7.

-timestamp - The timestamp, i.e., the exact time, of the build.

<date> - A timestamp of the form yyyy-MM-dd'T'HH:mm:ss'Z' or yyyyMMddHHmm. Default current time.

-build-url - The URL of the build job that produces the update site.

<url> - An artibary URL. Optional, i.e., no generated build link.

-commit - The URL representing the state of the Git repository used by the build that produces the p2 repository. This is stored as a repository property named commit.

<url> - The URL of the commit. Optional, i.e., no generated commit link.

-root - The root location of the local mirror of the update site. The promoted update site will be mirrored to this location. In addition, the -remote update site will be partially mirrored using rsync to this location. The index generator processes this location. The final result will be promoted to the -remote host also using rsync.

<path> - A path in the local filesystem. Required.

-target-url - The URL at which the -root of site will exist once promoted to the -remote server. This is used for the display of links and also for the generation of the p2.mirrorsURL property in the artifact metadata.

<url> - A URL. Required.

-relative - The relative location below the root at which to target the promotion and generation.

<path> - A relative path. This path will be used in URLs on the remote site as well for the -root. Required.

-remote - A remote source/destination specifification as used by rsync.🔗

<rsync-path> - A specication of the host and the path on that host. E.g.,
genie.justj@projects-storage.eclipse.org:/home/data/httpd/archive.eclipse.org/justj
localhost:C:/Users/Account/justj
If a compatible/standard rsync is available on the system PATH, one can test using localhost. Optional, i.e, when not specified, only the -root will be populated locally which is useful for testing.

-promote - The local filesystem location of the simple p2 update site to promote, or a URL for the remote p2 update site to promote. The promotion is done as specified by the -build-type. Note in particular that release builds do not promote the specified site.

<path> - The filesystem path to the local simple repository. Required.

-baseline-url - The URL from which to check for baseline replacements for the artifacts of the -promotesite.

<url> - A URL. Optional.

-promote-products - The local filesystem folder location of the products, generally those produced by Tycho, to promote along with the update site. The product names are stored in the repository property products. This is generally used only for the update sites of a product. A download link for the each products will be available in the generated index.

<path> - The filesystem path to the local Tycho-built products. Optional, i.e, there are no products to promote.

-downloads - The local filesystem files of so-called downloads to promote and maintain along with the update site. These will be maintained in a subfolder called downloads nested within the update site. A download link for the each artfiact will be available in the generated index. The file names will be stored in the repository property downloads.

<paths> - A series of filesystem locations of arbitrary regular files. Optional, i.e., no downloads to promote.

-version-iu - An ID prefix used to select or one more installable units with an ID with that prefix as the installable units whose largest version determines the logical version of the repository as a whole. Typically this a feature, particularly an SDK feature. If neither this nor -version-iu-pattern are specified, all installable will be considered which is likely to be inappropriate.

<string> - An arbitrary string that will match the prefix of at least one installable unit's ID. Optional but then -version-iu-pattern really should be present.

-version-iu-pattern - A regular expression used to select or one more installable units with a matching ID as the installable units whose largest version determines the logical version of the repository as a whole. If neither this nor -version-iu are specified, all installable will be considered which is likely to be inappropriate.

<pattern> - A valid java.util.Pattern. Optional but then -version-iu really should be present.

-iu-filter-pattern - A regular expression used to filter down the installable units, by ID, for which generated index information is to be produced. E.g., often one does not wish to list 3rd party installable units in the index.

<pattern> - A valid java.util.Pattern. Optional, i.e., generate index details for all installable units.

-primary-iu-filter-pattern - A regular expression used to filter down the installable units, by ID, that are to be considered primary features, i.e., SDKs. This subset of primary features is presented on the root page and is shown in bold on all the other pages. If there are no primary features, then all features are presented on the root page. If all features are primary, then none of them are shown in bold on all the other pages.

<pattern> - A valid java.util.Pattern. Default .*\.sdk([_.-]feature)?\.feature\.group.

-excluded-categories-pattern - A regular expression used to filter away category installable units with matching ID. Such removal helps reduce the likelihood that a user will install something from that category.

<pattern> - A valid java.util.Pattern. Optional, i.e., don't remove any categories.

-maven-wrapped-mapping - Mappings for modifying or removing the maven-wrapped- (or maven-) installable unit properties, i.e., for mapping a maven coordinate groupId:artifactId:version to its replacement subsitution, or to nothing, for removal. E.g.,
org.eclipse.orbit:ant:(.*)
org.eclipse.orbit:(xyz:.*)->org.apache:$1

<pattern-substitution-pair> - A valid java.util.Pattern followed by an optional substituion string, separated by ->. Optional.

-label - The label used as the project name in the generated index pages.

<string> - An artibary string. Default Project.
If the value contains space, then in a command-line context, the value must be quoted.

-breadcrumb - An arbitrary label with optional associated URL, separated by a space from the label, for use in the breadcumbs of each generated index page. E.g.,
Justj https://eclipse.dev/justj
This argument may be repeated to specify multiple breadcumbs.

<string-url-pair> - An arbitrary label followed by an optional URL, separated by a space. Optional but this is an important branding detail about the project producing the content, so best to have at least one.
In a command-line context, the value must be quoted.

-archive - An arbitrary label with an associated URL, separated by a space from the label, for use in the archive navigation of each generated index page. E.g.,
0.0.1 - 0.10.12 https://example.org/downloads
This argument may be repeated to specify multiple archive links. The links are placed in a section labeled Archive immediately after the Release section in the navigation bar.

<string-url-pair> - An arbitrary label followed by a URL, separated by a space. Optional.
In a command-line context, the value must be quoted.

-favicon - A URL to an image that will be used as the favicon of each generated HTML index page.

<url> - The URL of an image suitable for use as a favicon. Optional, i.e., no favicon is generated.

-title-image - A URL to an image that will be used as the title image of each generated HTML index page.

<url> - The URL of an image suitable for use as a tile image. Optional, i.e., generate the default Eclipse logo title image.

-body-image - A URL to an image that will be used in the body of the generated HTML index page.

<url> - The URL of an image suitable for use as the body image. Optional, i.e., generate no body image.

-mapping - A name mapping to convert a lower case name to a title case name. E.g.,
justj->JustJ
This is used to help produce improved navigation labels in the generated index pages. Without such a mapping, justj would map to Justj. This arugment may be repeated to specify multiple mappings.

<string-pair> - Two arbitrary strings by ->. Optional.

-commit-mapping - A regular expresion mapping to convert a source commit URL to a target commit URL. E.g.,
(.*/)old-repo(.*)->$1new-repo$2
This argument may be repeated to specify multiple mappings.

<pattern-substitution-pair> - A valid java.util.Pattern followed by a substituion string, separated by ->. Optional.

--exclude - This argument and its associated value are passed directly as arguments to rsyn. This argument may be repeated to specify multiple folders to exclude from being transfered to or from the -remote host.

<file-name> - The name of a file. Optional.

-summary - The number of update site columns in the summary table. Each row of the tabular summary corresonds to an bundle symbolic name of an installable unit with an ID matching the -summary-iu-pattern. Each column corresponds to an update site, starting with nightly/latest, milestone/latest, and then the releases from the most recent to the oldest, where the specified value for -summary truncates that list. Each cells displays the version(s) of the installable unit(s) with the corresonding bundle symbolic name in the corresponding site.

<integer> - A positive integer. Optional, i.e., no generated summary table.

-summary-iu-pattern - A regular expression used to select the installable units with matching ID for display in the -summary table.

<pattern> - A valid java.util.Pattern. Default, .*(?<!\.source|\.feature\.group|\.feature\.jar), i.e., exclude all feature IUs and all source IUs.

-simrel-alias - Can be used with a repository where the logical version of the repository, via -version-iu or -version-iu-pattern, can be mapped onto an Eclipse SimRel version, e.g., 4.30 maps to 2023-12. This option is used by Orbit to produce its update sites. The first nightly build that produces new SimRel version will result in the creation of repository named according to that version, e.g., 2023-12, which then composes that first nightly build. Each subsequent nightly build will be composed as the one child of that version repository, until the first milestone build. Then each subsequent milestone build will be composed as the one child of that version repository, until the release build, at which point the lifecycle is complete, and the verion repository composes the release. This provides a permanent and stable URL at the start of each release cycle which contains the latest content and then will ultimately contain the release content at the end of the release cycle.

-bree - Can be used to generate details about the minimum execution environment of eaach bundle. The details are generated where the bundle's version is generated.

-super - This is used by JustJ and is not generally intended to be reused, so don't use it.

<path> - A relative path.

-latestVersionOnly - Whether to mirror only the latest version or all versions when mirroring an update site. This is highly unlikely to be used.

Managing p2 Update Sites in Maven

The org.eclipse.justj.p2.manager application can be invoked as follows directly from a pom.xml:

  <properties>
    <eclipse.repo>https://archive.eclipse.org/releases/latest</eclipse.repo>
    <justj.tools.repo>https://archive.eclipse.org/justj/tools/updates/nightly/latest</justj.tools.repo>
    <org.eclipse.storage.user>genie.justj</org.eclipse.storage.user>
    <org.eclipse.justj.p2.manager.args>
      -remote ${org.eclipse.storage.user}@projects-storage.eclipse.org:/home/data/httpd/archive.eclipse.org/justj
    </org.eclipse.justj.p2.manager.args>
    <org.eclipse.justj.p2.manager.relative>tools-test/updates</org.eclipse.justj.p2.manager.relative>
    <maven.build.timestamp.format>yyyyMMddHHmm</maven.build.timestamp.format>
    <org.eclipse.justj.p2.manager.build.url>http://www.example.com/</org.eclipse.justj.p2.manager.build.url>
    <build.type>nightly</build.type>
  </properties>

  <build>
    <plugins>
      <plugin>
        <groupId>org.eclipse.tycho</groupId>
        <artifactId>tycho-eclipse-plugin</artifactId>
        <version>${tycho-version}</version>
        <configuration>
          <executionEnvironment>JavaSE-21</executionEnvironment>
          <dependencies>
            <dependency>
              <artifactId>org.eclipse.justj.p2</artifactId>
              <type>eclipse-plugin</type>
            </dependency>
            <dependency>
              <artifactId>org.apache.felix.scr</artifactId>
              <type>eclipse-plugin</type>
            </dependency>
          </dependencies>
          <repositories>
            <repository>
              <id>eclipse.repo</id>
              <layout>p2</layout>
              <url>${eclipse.repo}</url>
            </repository>
            <repository>
              <id>justj.tools.repo</id>
              <layout>p2</layout>
              <url>${justj.tools.repo}</url>
            </repository>
          </repositories>
        </configuration>
        <executions>
          <execution>
            <id>promote</id>
            <goals>
              <goal>eclipse-run</goal>
            </goals>
            <phase>generate-sources</phase>
            <configuration>
              <argLine></argLine>
              <appArgLine>
              <![CDATA[ 
                -consoleLog
                -application org.eclipse.justj.p2.manager
                -data @None
                -nosplash
                ${org.eclipse.justj.p2.manager.args}
                -retain 5
                -label "JustJ Tools"
                -build-url ${org.eclipse.justj.p2.manager.build.url}
                -root ${project.build.directory}/justj-sync
                -relative ${org.eclipse.justj.p2.manager.relative}
                -target-url https://archive.eclipse.org/justj
                -promote ${project.basedir}/../../org.eclipse.justj.tools.site/target/repository
                -timestamp ${build.timestamp}
                -type ${build.type}
                -version-iu org.eclipse.justj.tools
                -commit https://github.com/eclipse-justj/justj.tools/commit/${git.commit}
                -breadcrumb "JustJ https://eclipse.dev/justj/?page=download"
                -favicon https://eclipse.dev/justj/justj_favicon.ico
                -title-image https://eclipse.dev/justj/justj_title.svg
                -body-image https://eclipse.dev/justj/justj.svg
              ]]>
              </appArgLine>
            </configuration>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>

Be aware that the procesing of the arguments does not properly handle tab characters as whitespace for separating arguments, so if you format the POM with tabs, it's best to use <![CDATA[ ]]> to ensure that the formatter does not insert tabs into your appArgLine.

Examples of Managed p2 Update Sites

The following is a sampling of update sites maintained by the org.eclipse.justj.p2.manager application.