Skip to content

feat(vagrant): publish the boxes in the AlmaLinux Vagrant box catalog on S3 - #379

Merged
andrewlukoshko merged 1 commit into
AlmaLinux:mainfrom
yuravk:vagrant-s3-catalog
Oct 19, 2026
Merged

andrewlukoshko merged 1 commit into
AlmaLinux:mainfrom
yuravk:vagrant-s3-catalog

Conversation

@yuravk

@yuravk yuravk commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

Why

HCP Vagrant stopped accepting new box versions on 2026-10-01 and shuts down on 2026-12-31 (#363). Since 2026-10-01, new almalinux/* box versions are built, tested and uploaded to S3, but nothing publishes them for vagrant box add / vagrant box update.

The replacement is a Vagrant box catalog on our S3 bucket, already live and verified with Vagrant:

  • one Vagrant metadata JSON per box at s3://almalinux-cloud/vagrant/almalinux/<box>[.json], public, served as application/json;
  • it points at the .box files the builds upload to S3;
  • it holds every version that was on HCP: the HCP-only boxes were copied to S3, and the missing .sha256sum files were generated. It also has the releases built since HCP closed (9.8.20261005, 10.2.20261012, the VirtualBox arm64 boxes).

Users switch with one variable:

export VAGRANT_SERVER_URL=https://almalinux-cloud.s3.us-east-1.amazonaws.com/vagrant
vagrant box add almalinux/9

This PR makes "Vagrant: Build, Test and Publish" add new versions to that catalog, instead of publishing to HCP.

What changes

vagrant-build-test-publish.yml

  • Each build leg records its box as a catalog-entry-<variant>-<leg> artifact (file name, S3 key, sha256). It does so only when the box is on S3 and passed its test: the vagrant up test, or the offline validation on the Hyper-V and aarch64 legs, which have no live test.
  • A new publish-catalog job runs after all builds and adds the recorded boxes at once with .github/scripts/vagrant-catalog-add.py:
    • downloads the current metadata of each affected box;
    • adds one version per catalog box with all its providers. A re-run replaces providers, never duplicates them;
    • sets default_architecture (amd64 where present);
    • keeps the previous file, private, under vagrant/.history/<time stamp>/;
    • uploads <box> (for VAGRANT_SERVER_URL) and <box>.json, tagged public=yes, with max-age=300;
    • checks both over HTTPS (content type, content);
    • writes the job summary and posts to Mattermost.
  • The vagrant-catalog concurrency group makes catalog updates wait for each other, across runs too, so two runs can't interleave download-change-upload on the same file. Users never see a partly published version.
  • release_to_catalog replaces release_to_hcp. It defaults to true, so the scheduled runs publish, and it needs upload_to_s3.
  • Publishing happens in AlmaLinux/cloud-images only. Forks and ci-dev use the same bucket, and must not change the production catalog.
  • The HCP publish steps and the publish-aarch64 job are removed. The arm64 legs no longer need an x86_64 runner, because the catalog needs no Vagrant CLI.

.github/scripts/vagrant-catalog-add.py (new): adds the recorded boxes to the catalog, as described above. It reuses the box-name parsing of tools/vagrant_catalog.py, e.g. AlmaLinux-10-Vagrant-vmware-10.2-20261012.0.aarch64.box -> almalinux/10 10.2.20261012 vmware_desktop/arm64.

tools/vagrant_catalog.py: the generator the live catalog was seeded with, from a snapshot of the HCP registry metadata plus the releases built after HCP closed (--extra). It is kept to rebuild the whole catalog from scratch, if ever needed.

Docs: VAGRANT_BUILD_TEST_PUBLISH.md gets a new "The Vagrant box catalog" section. SCHEDULED_BUILDS.md, BUILD_VAGRANT.md, README.md and the scheduled-builds.yml comment are updated.

The standalone vagrant-publish.yml and its vagrant-publish-steps composite still publish to HCP only, and are left as they are.

Requirements

The CI bot's almalinux_cloud_s3 policy allows s3:PutObject and s3:PutObjectTagging on almalinux-cloud/vagrant/* (already added).

@yuravk
yuravk requested a review from andrewlukoshko October 19, 2026 10:22
… on S3

HCP Vagrant stopped accepting new box versions on 2026-10-01 and shuts down
on 2026-12-31. The almalinux/* boxes are served from our own catalog on S3
instead: one Vagrant metadata JSON per box at
s3://<AWS_S3_BUCKET>/vagrant/almalinux/<box>[.json], public, served as
application/json, pointing at the .box files the build uploads to S3 -
users set VAGRANT_SERVER_URL to
https://<AWS_S3_BUCKET>.s3.us-east-1.amazonaws.com/vagrant.

"Vagrant: Build, Test and Publish" now publishes there instead of HCP:
- each build leg whose box is on S3 and passed its test (the vagrant up
  test, or the offline validation for hyperv and the aarch64 legs) records
  it (file name, S3 key, sha256) as a catalog-entry artifact;
- a new publish-catalog job, after all builds, adds them all at once with
  .github/scripts/vagrant-catalog-add.py: one new version per catalog box
  with all its providers (a re-run replaces, never duplicates), the previous
  file kept under vagrant/.history/<time stamp>/, the new <box> and
  <box>.json uploaded and checked over HTTPS. A vagrant-catalog concurrency
  group keeps runs from interleaving their updates;
- release_to_catalog (default true, so the scheduled runs publish) replaces
  release_to_hcp; publishing happens in almalinux/cloud-images only, as
  forks and ci-dev share the bucket;
- the HCP publish steps and the publish-aarch64 job are gone (the catalog
  needs no Vagrant CLI).

tools/vagrant_catalog.py, the generator the catalog was seeded with (from a
snapshot of the HCP registry metadata, plus the releases built after HCP
closed), is committed, to rebuild the catalog from scratch if ever needed.
Docs updated.
@yuravk
yuravk force-pushed the vagrant-s3-catalog branch from cb0ed6d to 8e49863 Compare October 19, 2026 10:54
@andrewlukoshko
andrewlukoshko merged commit 50c90e8 into AlmaLinux:main Oct 19, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants