Repository navigation
ci: fix baseline checks and release hooks - #1426
Merged
Merged
Conversation
sssciel
force-pushed
the
g.rybakov/tntp-11114-fix-base-ci
branch
8 times, most recently
from
October 30, 2026 21:39
fa491b7 to
7e2f729
Compare
Drop the mage PatchCC and GenerateGoCode hooks, which no longer exist, from the GoReleaser configurations. Discover Go modules instead of listing them, so mage unit also tests sdk/connect and sdk/dial, and lint every module together with the Aeon mock server. Add a lint job for Ruff formatting, spelling, go generate drift, tidy module files and go mod verify. Retry the license check on network errors only. Remove the pre-commit actions, configuration and instructions, which have no configuration since TNTP-9993, and reformat the Python tests with ruff format. Part of TNTP-11114
bigbes
force-pushed
the
g.rybakov/tntp-11114-fix-base-ci
branch
2 times, most recently
from
November 1, 2026 18:48
de01efd to
3d123b3
Compare
Run on Ubuntu 24.04 with Python 3.12, test CE 3.3 and 3.9 instead of 3.0 and the backup-only 3.8 job, and install the latest stable EE SDK through tt with customer-zone credentials. Run the Go integration tests on every push, add concurrency and read-only permissions to the workflows, and drop unused packages and the Mono server workaround from the test environments. Raise the go directive of the library modules to 1.27.0. Ubuntu 24.04 AppArmor blocks unshare -r -n before the offline rocks tests can start tt. Disable the unprivileged user namespace restriction on the test runners for the current boot, and check that unshare works during environment preparation. The image also lacks the libraries that tests building Tarantool from source need, so install them. Remove the tests and conditions for Tarantool 1.x and 2.x, for tuple formats before 3.2 and for TcS failover before 3.3, and adapt the fixtures to Tarantool 3. Search EE products for Tarantool 3 by the series of the newest release: --version takes a series such as 3.5. Keep the runtime edition in installed-binary test fixtures and start the backup TLS fixture with ssl:* so the four EE-only tests reach their assertions. Guard executable fixture writes with syscall.ForkLock so parallel forks cannot inherit a writable descriptor and cause ETXTBSY. Wait for replication in both directions before the initial election promotion and check its response. Wait for the sockets of a started script instance, which box.cfg opens after the start mark, and for the config.storage role of a TcS fixture, which older Enterprise releases define after the listener comes up. Part of TNTP-11114
Replace the legacy test workflows with fast and full tiers backed by reusable build and integration workflows. Reuse binary artifacts and start the tests once the default build is ready. Partition integration tests into four validated groups balanced on the EE run time; a group can take single files of a directory that another group owns. Keep the tested Tarantool versions in ci/tarantool-versions.env, so workflows refer to them as min and latest. Resolve the newest stable EE SDK of a series once per job and cache it by its exact version. Retry the CE installation once, as download.tarantool.org times out now and then. Run branch checks on push. Fork pull requests run CE checks without EE credentials, approvals or environments. Run EE only for trusted sources in the upstream repository with its existing credentials. Publish one aggregate status per tier, ci-ok or full-ok, on the exact source commit. A fork pull request runs CE checks with a read-only token, so a workflow_run workflow publishes its status from the latest CE run of the commit whenever such a run completes, approvals and re-runs included. It never checks out the fork and leaves a status set by a push run alone. Avoid pull_request_target: GitHub runs it from the default branch for any base and disables it by default for public repositories. Load status code only from the trusted workflow revision, serialize status updates, and ignore results from superseded runs. Add pytest reports with retry tracking and non-blocking commit, changelog and SDK API notes. Pin actions, run actionlint, zizmor and shellcheck, and remove the obsolete update workflow. Part of TNTP-11114
Build tt without cgo for Linux amd64/arm64 and macOS arm64, and build tt-openssl with static OpenSSL on native Linux runners. Check the GoReleaser configurations, and validate binaries and release artifacts before uploading them. The packages conflict, and tt-openssl provides tt. Only the deb packages also replace each other: nfpm writes Replaces to rpm as Obsoletes, and with mutual Obsoletes dnf installs tt when asked for tt-openssl. Smoke-test both variants through apt and dnf from a local repository: a plain install gets the requested variant, an upgrade keeps it, and the variants are only swapped on request. Run snapshot builds and the smoke checks for package-related PRs and full CI. Use snapshot binaries throughout full integration tests, and add an EE TLS job for the OpenSSL build. Publish both variants to draft GitHub releases and RWS, accept tags from v3 or master, and replace deprecated workflow outputs. Part of TNTP-11114
Run the latest CE integration tests on Linux arm64 and macOS arm64 using snapshot binaries. Require the Linux arm64 result in full CI and keep macOS informational while its tests are being stabilized. Prepare platform-specific dependencies, install Tarantool through tt on macOS, and separate setup-tarantool caches by architecture. Update Python dependencies for Apple Silicon and allow extra time for the macOS runtime source build. Part of TNTP-11114
Schedule full CI, the canary and the nightly analysis on the head of the default branch at 01:00 Moscow time. GitHub runs schedules only from the default branch, so they start once v3 becomes it. Support manual runs of individual suites and an artificial failure to check issue reporting. Move network builds of upstream master into the canary. Run race tests, unit and integration coverage, and govulncheck across project modules. Block ci-ok on vulnerabilities only when go.mod or go.sum differ from the default branch, retry network errors, and preserve coverage data and logs. Create or update labeled issues when scheduled checks fail. Keep issue write permissions in separate reporting jobs and leave the canary and nightly analysis outside the required commit statuses. Part of TNTP-11114
tt binary install tarantool --use-docker builds Tarantool in an Ubuntu image of the host architecture, but the image downloaded the x86_64 CMake archive, so the build failed on Linux arm64 with "cannot start cmake: exec format error". Take the archive uname -m names: CMake publishes linux-x86_64 and linux-aarch64 ones. Part of TNTP-11114
The test of installing Enterprise products from the customer zone had not run since v3 changed bundle naming, and full CI found it out of date. Development builds are listed with the dev_ prefix and keep the full name as their installation, and the installer names the installation it makes rather than the selector. These expectations follow the code, not a run, so skip the test until a run with credentials passes. tt binary search tt-ee finds no archives in the customer zone, so skip tt-ee on its own as well. Part of TNTP-11114
bigbes
force-pushed
the
g.rybakov/tntp-11114-fix-base-ci
branch
from
November 1, 2026 21:06
3d123b3 to
c208a84
Compare
Full CI on macOS failed for reasons of the platform, not of tt: - LibreSSL, the openssl of macOS, has no -noenc. Generate the aeon test certificates with -nodes, which OpenSSL 3 takes too. - gRPC under the etcd3 client handles fork in the test process and logs into the stderr of every tt command a test runs. Turn the handling off. - Homebrew keeps bash-completion under its prefix, and it needs bash 4 or later. Install both and look there too. - APFS ignores case by default, so My_App was the existing my_app. - A unix socket path holds 103 bytes, and the temporary directory under /var/folders, or the application name twice under /private/tmp, leaves too little. Keep the scope roots in /tmp and give the played application a short name. - os.UserCacheDir ignores XDG_CACHE_HOME on macOS. Make HOME hold the runtime cache of the packing tests there. Part of TNTP-11114
The aeon connection tests also listen on @aeon_unix_socket, a name in the abstract namespace of unix sockets, which only Linux has. On macOS "@" starts an ordinary file name, tt does not reach the mock server, and the server does not stop cleanly. Skip that variant off Linux; the socket by a path still runs everywhere. Part of TNTP-11114
The config.storage role checks once per status_check_interval, 5 s in the template, that it reaches a quorum, and refuses requests until the check passes. An instance that checked before the leader was elected stays disconnected for a while, and test_create_config_storage[3] on EE 3.3 read the replica in that window, and so did its rerun: "bad storage status: disconnected". Wait until every instance reports the storage connected. Part of TNTP-11114
The arm64 and macOS jobs ran all integration tests in one job, 19 and 29 minutes in Full CI 41351778480. Run them by test groups, as it-ce does, in three jobs each. Without the slow tests the cluster group takes under a minute, so it goes with replicaset. By the reports of that run the longest group takes 6.7 minutes of tests on arm64 and 8.7 on macOS, where each job also builds Tarantool for 6.6 minutes. Part of TNTP-11114
tt instance start returns before the instances listen, and the tests gave them 0.5 s. With --timeout 1 the discovery makes one attempt, and the test expects it to reach instances whose config is not applied yet. On macOS no console was open by then, and tt failed with "failed to connect to any instance" in both runs. Locally the consoles opened in 0.6-1.3 s. Wait for every console instead of sleeping. Part of TNTP-11114
test_test_installs_luatest_it_was_never_told_about is the only case that installs rocks from the network. In Full CI 41410666360 installing checks 3.4.1-1 failed in it-ee (latest, instances) twice in a row, 109 and 42 s, while the same case passed in the other jobs of the run in 21-27 s. The error said only that luarocks exited with code 1: tt logs the output of luarocks at the debug level. Run tt test with -V, so a failure shows what luarocks printed, and rerun the case twice with 30 s between the runs. Part of TNTP-11114
The native engine of go-luarocks, under tt package build, fetch and install, read only the rockspec out of every rock it fetched and built from the source the rockspec names. So a .all.rock was rebuilt from a clone of its repository, a .src.rock ignored the sources it carries, and a directory tt package download wrote still sent every rock to the network: building a project that depends on crud ==1.2.0-1 cloned checks, errors, vshard and crud from GitHub even with TT_REGISTRIES naming such a directory. Every install also fetched the manifest of each rock server again, over a new connection, and HTTP_PROXY, HTTPS_PROXY and NO_PROXY were ignored. go-luarocks e4750d3 deploys a .all.rock or a host .rock as built and builds a .src.rock from its sources, fetches a manifest once per command, keeps connections alive and honours the proxy variables, and a failed LuaRocks command reports the error LuaRocks printed. With Tarantool CE 3.8.1 the crud project builds in 16 s instead of 57 s from the registries, and in 2 s from a local mirror with no network, where it failed before.
The integration tests that build projects or install rocks fetched them from rocks.tarantool.org and luarocks.org on every run: the crud project of test_running_base_functionality_crud_app took 166 s in it-ee (latest, instances), the vshard projects of the replicaset and create tests fetched vshard each time, and a slow or dropped download failed the job. CI now downloads the rocks test/rocks-mirror.txt lists once into a directory rock server with test/ci.py rocks-mirror, packing those the registry holds only as a rockspec into source rocks so that installing them clones nothing, and keeps it in the Actions cache under the hash of the list, per runner OS and arch. The tests that fetch rocks run tt with utils.rocks_mirror_env, which names the mirror in TT_REGISTRIES when TT_TEST_ROCKS_MIRROR is set; without it they reach the registries as before. tt rocks install vshard in the replicaset tests now gets vshard 1.0.0-1 from the mirror instead of scm-1, the head of its repository. The test of tt test installing luatest and test_rocks.py, which check the registries themselves, keep fetching from them. The build of the sharded backup project moves to the mirror with its session fixture.
bigbes
approved these changes
Nov 2, 2026
tt package install refetched the dependencies of a --without-deps archive from the default rock servers: runPackageInstall handed install.Run no server list, so TT_REGISTRIES, which every other command that queries rock servers reads, was dropped, and install took no --registry at all. An install against a mirror or a local directory still went to rocks.tarantool.org. Install takes the list of the flag and of the environment now, as the other commands do. The archive's own [platform].registries is not a layer there: its relative entries were written against the directory of another project. test_package_pipeline pointed its install tests at the fixture repository through TT_REGISTRIES and took itself for offline, while the refetch went to the public servers, where stat is found too; the collision test paid for three of them, 84 s in CI. A new test gives install a registry that serves no stat, by the flag and by the environment, and expects the install to fail, which it did not before. The collision test installs a package with no dependency, since the collision is what it tests, and the refetch test checks that stat is the stub of the fixture, which the public stat is not.
test_install_tt installed the latest tt, which is the newest tag. v2.15.1 is tagged and not released yet, so the prebuilt archive was not found and install built tt from source, 59 s in CI, while the prebuilt path the test is about went unexercised without a word. The test installs v2.15.0, released with its prebuilt archives, and checks that the prebuilt archive is what got installed and that it is that version. It is skipped where tt is not released prebuilt. Resolving the latest version is covered offline by test_install_tt_latest_release. test_install_tarantool_commit built 00a9e59 statically: OpenSSL, ICU, readline and the rest from source before Tarantool itself, 396 s in CI. With --dynamic it links the system libraries; in an ubuntu:22.04 container with the development packages CI installs it builds in 231 s. The static build stays covered by test_install_tarantool_in_docker, and resolving a commit by test_install_tarantool_remote_revision.
The coredump tests of choosing the executable to pack ran on a Tarantool built from source by tmpdir_with_tarantool, 125 s of setup in CI, and checked the archive by version alone. They run on a copy of the system tarantool with a marker appended now, so the two are different files and the archive is compared byte for byte; they are not slow any more. The core file is also found when kernel.core_uses_pid makes the kernel append the pid to a pattern without %p: generate_coredump looked for a file named core and found none. test_restore_synchro_boundary waits out the synchro timeout of membership_app, 10 s, once in each test and twice in the second, which took 30 s in CI. Its copy of the app runs on 3 s; the app writes synchronously only once a follower is there to acknowledge it, and test_restore_membership keeps the 10 s. The check that the point creation queued behind the limbo counts from the hung write now: the polls that see the write land ate into the short timeout and failed it. sharded_app was built, and vshard downloaded and installed, by each module that ran the cluster. It is built once for the session, from the rock mirror of CI when there is one, and each module starts a copy of the built tree.
CI reruns a failed test once, and a test that passes the second time is reported by that attempt alone: the run of b7dd505 listed test_running_base_functionality_crud_app in it-ee (min, instances) as flaky with no trace of what failed. pytest-rerunfailures keeps the failed attempt, but -rR prints only its name. Print every failed attempt that was rerun, with its traceback and the output it captured, in a section at the end of the pytest log.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Discover Go modules and enable formatting, spelling and drift checks.
Part of TNTP-11114