Repository navigation
core, distbuild: build distributions of tt and sign their packages - #1427
Merged
Merged
Conversation
sdk/integrity has an RSA provider that signs and checks the files of a project and the data published to a configuration storage, but nothing in tt could select it: every command registered its flags, set up the checks and built the storage signers and verifiers through the package-level functions, which are the disabled provider. A distribution that wants the checks, tt-ee among them, had to replace the integrity module at link time. core.WithIntegrity hands Main the provider, and cmd.Boot makes it the provider of the process before any command is built: the integrity flags are registered while the commands are constructed, so it cannot be chosen later. The provider lives in internal/integrityprovider, outside the SDK, so a module has no way to switch it. Every command of tt goes through it now - the root and tt instance start flags, --with-integrity-check of tt cluster and tt replicaset, Configure, the storage factories of cli/cluster and the name of the hashes file an instance is started with, which the disabled provider leaves empty and the RSA provider names hashes.json. The watchdog revalidates through the repository Configure set up, so it follows as well. Without the option tt keeps the disabled provider: no integrity flags, and its help is the same.
cmd.InjectedCmds let a fork of tt put commands of its own into the tree and replace tt's commands of the same name; tt-ee used it for its commands and its version. A distribution now builds tt from modules with core.Main, presents itself with core.WithFlavour and chooses its integrity provider with core.WithIntegrity, so the commands it adds are mounted like any module's and nothing replaces a command of tt behind the core's back. InjectedCmds, InjectCommands and the step of Main that ran it are gone, with the tests of the shape tt-ee had. GetCmdCtxPtr stays: the core offers its services from the context of the process.
The magefile chose the build tags and the static linker flags of tt from TT_CLI_BUILD_SSL in its own switch. A distribution of tt built outside the magefile has to be compiled with exactly the same flags, or it quietly differs from the tt CI builds; two copies of the table would drift apart. internal/buildflags now holds it: Parse reads the name TT_CLI_BUILD_SSL takes (default, openssl or openssl_static; empty is default), and the SSL it returns gives the tags, the linker flags for a target OS and whether cgo is needed. The magefile builds tt with it, with the same tags and flags as before.
go-storage connects to a Tarantool config storage over TLS only when built with go_storage_ssl; without the tag a connection whose config enables SSL fails with ErrSSLDisabled. tt-ee 2.x was built with openssl_static,go_storage_ssl, while the tags of tt 3 never set it, so neither the cluster commands of the core (sdk/cluster, cli/cluster/storage.go) nor the tt modules using go-storage could reach a config storage over TLS whatever TLS implementation tt was built with. The openssl and openssl_static builds now add go_storage_ssl. The default build, TLS in pure Go, goes without: go-storage has no TLS of its own on top of gostls yet, which is a change of go-storage.
A distribution of tt is the tt core and the compiled-in tt modules one tt binary is built from. It shares the envelope of the manifest of an application - the format version and its rules for unknown fields, the manifest hash, the lock's head - but not its body: no components, no products, no rocks, and a core to link instead of a tt to run. So it is a document of its own, Distribution, read by ParseDistribution, with its lock, DistributionLock. The kind of a manifest is told by manifest_kind, read before the rest as manifest_version is: absent or "app" is an application, so the existing manifests need no change, and "tt-distribution" is a distribution. Each parser refuses the other kind with an error that names the command the manifest is for, rather than an unknown field. Every table and field of a distribution manifest has a default, so manifest_version and manifest_kind make a complete one. [package].name names the binary (tt). [core] takes a Go version query (tt), a local checkout (path) and the module path; without either the build takes the core of the tt that runs it. [distribution] sets the title, the edition, the integrity provider (none or rsa) and whether tt's own commands and the external modules are in. [build] sets the TLS implementation, cgo, and extra tags and linker flags. A dependency is a tt module, written as a local path, a module path or module@version, or as a table with module, version, path, package and symbol; kind and source are accepted for the RFC's spelling and follow from the rest. The lock pins the core with its revision and the other modules of the core's repository, and every tt module; it records the toolchain and the tags of the last build without gating the next one on them.
A distribution is built the way golangci-lint custom builds a linter, except that the core is a Go dependency like the modules rather than a clone: the build generates a main module in a work directory - a go.mod requiring the core and the tt modules and a main package handing their constructors, by name, to core.Main - tidies it and compiles it. The core resolves from [core].tt with go list -m, from a local checkout with [core].path, or, without either, from the version of the core of the tt that builds. The other modules of the core's repository - sdk, sdk/*, modules/* - are required by the core as v0.0.0 and found through replace directives of its go.mod, which a module depending on the core does not see, and they are not in the core's module zip. The build reads those directives and pins each module to the core: to its directory in a local core, to its pseudo-version at the core's commit otherwise, with a replace that covers whatever version anyone requires. A tt module of the core's repository follows the core the same way; a local module is replaced with its directory, any other is resolved to an exact version (latest by default). The main package is generated canonically - imports and the table sorted, aliases unique - with the options the manifest sets: the flavour (title, edition, and the version git describe gives in the manifest's directory, with VERSION_LABEL), the RSA integrity provider, no external modules. go build runs with -trimpath, the tags and linker flags of internal/buildflags for the TLS implementation, the manifest's own after them, and cgo only for OpenSSL unless the manifest says otherwise. The binary's tt version is run before it is installed, so a module whose command cannot be mounted fails the build, not its user. tt.manifest.lock pins the core, its repository's modules and the modules, and tt.manifest.sum keeps the go.sum of the build. While the manifest's hash matches, the next build takes the locked versions; Update resolves again; Locked requires a matching lock, builds only the locked versions, fails when the go.sum differs and writes nothing but the binary. A module, a version or a manifest that does not resolve is exit code 1; a missing go, the network and a failing go build are 2. backend.QuoteFlags is exported for the -ldflags of the build.
tt binary install tt installs a tt someone built; tt binary build tt builds one: the distribution tt.manifest.toml in the project directory (-C, or --manifest) describes, into the package name next to the manifest or into -o. The path to the binary is the result on stdout; the progress goes to stderr, and -V shows the go command at work. --update resolves the core and the modules again instead of taking them from the lock; --locked builds strictly from the lock and the sum, as CI should, and the two exclude each other. --work keeps the generated main module in a directory to look into. TT_CLI_BUILD_SSL overrides [build].ssl, as it selects the TLS of tt's own build, and VERSION_LABEL follows the version of the distribution. The help shows the whole manifest with its defaults; the README gets a section on distributions.
A distribution has to be buildable where there is no network: a certified build runs offline, from sources handed over beforehand. tt binary build tt --vendor DIR resolves and generates as a build does, the lock included, but instead of go build it runs go mod vendor in DIR and leaves a bundle there that the Go toolchain alone turns into the binary: no network, no module cache, no tt, no checkout of the manifest's repository. The bundle holds the generated go.mod, go.sum and main.go and vendor/ with every package the binary is built from - the core, the modules of its repository, the tt modules and the third-party ones, with the files they embed (the all:templates of tt create, the Lua scripts). A local module, a local core and its repository's modules among them, is copied into local/ as the go command would zip it - no VCS directories, nested modules or vendor directories - and replaced by its path relative to the bundle, so that neither go.mod nor vendor/modules.txt names a directory of the machine that made the bundle and it builds wherever it is moved. go build -mod=vendor reads none of local/; it is there for the replace directives to name something real and for whoever audits the sources. build.env records, as shell variables, the tags, the linker flags (and the macOS ones) and cgo of each TLS implementation, the implementation the bundle was made with, the toolchain, the manifest hash and the version; build.sh builds with them, taking the implementation and the output. The bundle is built once offline from vendor/ and its tt version run before the command reports it made, and a directory that holds anything but an earlier bundle is refused.
Two builds need a core other than the one the manifest names, and editing the manifest is wrong for both: a distribution built against changes of the core not released yet (tt-ee against ../tt), and the CI of tt itself, which builds its distributions from the working tree. tt binary build tt --core VALUE, or TT_DIST_CORE, overrides [core] for one build. A value that is an existing directory or a path starting with ./, ../ or / is a local checkout, taken from the project directory when relative; anything else is a version query, as [core].tt takes. The module stays the manifest's, and a checkout holding another module is refused as one in [core].path is. Such a build is a development one. Its core and the modules of the core's repository are resolved again rather than taken from the lock, and neither the lock nor the sum is written, so a manifest's pins do not drift to a working tree; the other modules still come from the lock while the manifest matches it. --locked refuses the override, from the flag or the environment alike.
A configuration published with --with-integrity-check carries its SHA-256 digest and its RSA-PSS signature next to it. tt 2, and the instances of Tarantool Enterprise Edition that check them, keep them at <prefix>/hashes/sha256/<key> and <prefix>/sig/<key> next to <prefix>/config/<key>. The go-storage codec tt 3 publishes and reads through lays its keys out per codec instead: <prefix>/hashes/sha256/config/<key> and <prefix>/sig/rsapss/config/<key>. tt 3 therefore read no signed configuration tt 2 had published - the check failed with 'hash "sha256" not verified (missing)' - and published its own where nothing else looks. The storage now lays the keys out the old way: the object location stays out of the hash and signature keys (namer.LegacyHashSigLayout) and the signature key names no algorithm (namer.CompactSingleSig). Configurations without integrity keep the one key they had. The codec builder of go-storage v2.0.0 has WithSingleSigCompact for the latter, but it cannot be used when collecting: Build requires exactly one signature location from the generator, the validator and the namer that parses keys, and a codec that only verifies has no signer, so it fails with "WithSingleSigCompact requires exactly one unique signer/verifier: signers=0 verifiers=1 union=1" (integrity/codec.go, Build). The storage gives the namers the option itself through WithNamer, to every namer with exactly one signature location; the generator of a verifying codec emits no signatures, so leaving it out there changes nothing. Once go-storage counts the signers and the verifiers that are there, WithSingleSigCompact can replace it. The tests sign data as tt 2 did - the hex SHA-256 digest and the hex RSASSA-PSS signature of it with the salt as long as the hash - put it under the old keys and read it back by prefix and by key; they check the exact keys a publication writes, that publishing the whole configuration drops the hashes and signatures of other keys, and that a changed value, a missing signature, another key and the layout tt 3 betas wrote are refused.
The RSA provider parsed a private key as PKCS #1 and a public key as PKIX, and refused anything else. openssl genrsa of OpenSSL 3 writes PKCS #8 unless told -traditional, so a key made the obvious way was refused with a parse error that named neither format; tt-ee 2 had the same limit, and the tests of tt-ee had to pass -traditional to make their keys. openssl rsa -RSAPublicKey_out writes PKCS #1, which was refused for the public key the same way. A private key is now parsed as PKCS #1 and then as PKCS #8, a public key as PKIX and then as PKCS #1; when neither parses, the error gives both reasons. A PKCS #8 or PKIX key of another algorithm - Ed25519, say - is refused as not an RSA key, as a PKIX one already was. The label of the PEM block is still not checked: the DER decides. The tests sign with a PKCS #8 key and check with a PKCS #1 public key through the provider, and refuse an Ed25519 private key.
The help goldens pin tt with the disabled provider, the one tt itself runs with, so nothing showed what a distribution choosing the RSA provider - tt-ee - prints: TestMainWithIntegrity only checks that the flags are there. A new golden pins the help of "Tarantool CLI EE" with the RSA provider: the root with --integrity-check and --integrity-check-period, tt instance start with how often the watchdog repeats the checks, and tt cluster publish, tt cluster replicaset promote and tt replicaset vshard bootstrap with the key that signs what they publish. Regenerate it with go test ./cli/cmd -run TestHelpIntegrityGolden -update.
tt version of a distribution with a version of its own printed that version alone: nothing told which tt core the binary was built on, and nothing what modules it was built from, though the module system makes both a matter of the build. A binary of tt-ee said "Tarantool CLI EE version 3.0.0" and kept the core it runs to itself. In full, tt version of such a distribution now adds a line with the version of the core and its commit, taken from the build info as the core's own version is when the build stamps no tag. tt's own build, whose version is the core's, prints what it printed before, and so do --short and --commit, which scripts read. tt version --modules lists the modules of the table the main package hands to core.Main: their names, and the Go modules their constructors come from with the versions Go recorded for them, as the modular build specification describes. The core passes the table to Boot with the package of each constructor, which the runtime names; the version command matches the package against the build info by the longest module path that holds it. A module replaced by a directory - a local module, or every module of a core built from a checkout - shows (devel), not the directory, which is the build machine's; one replaced by another module shows the replacement's version and path.
The locks tt writes - tt.manifest.lock of a distribution, the lock of an application - name their author in generated_by as "tt" and the short version of the core. A build no tag precedes as far as it knows, which is every tt built from a checkout of v3 with go build, has no short version, so every such lock said "tt <unknown>" and nothing told which build of tt had written it. version.Stamp gives the files tt writes a version that is never less precise than the build: the short version when there is one, else the exact module version Go recorded for the core - a pseudo-version that names the commit, with +dirty for a modified tree - else the commit alone. generated_by of both kinds of lock takes it. The version [platform].tt constraints are checked against is not touched: it stays the short version, and a development build still satisfies nothing.
A local tt module, or a local core, has no version: the lock recorded its path and nothing else, so tt binary build tt --locked built whatever the directory held that day and still called it the locked build. The modular build specification asks for a content_hash of such a module for exactly this. The lock now records content_hash for every local module and for a local core: "sha256:" and the SHA-256 of a listing of its files, one line per file with the hex SHA-256 of its content and its path, in the order of the paths - the shape of the h1: hash Go keeps in go.sum, so the hash depends on the paths and contents alone, not on modes, times or where the directory is. The files are the ones the bundle of --vendor copies, and the go command would zip: every regular file of the directory but those in VCS and vendor directories and in nested modules; the directory of the build is left out as well. A local core is hashed together with the modules of its repository its go.mod replaces with directories, since the binary is built from them too. The rule now lives in one function for the bundle and the hash. A build from the lock fails, before it generates anything, when the files changed: "./plugins/hello changed since tt.manifest.lock was written; build without --locked to update it". An ordinary build records the new hash. A build on another core, which writes no lock, hashes nothing.
The magefile set CGO_ENABLED=1 for every build, while the builder of distributions and the tags of the default TLS implementation say a default build needs no cgo: TLS is pure Go there, and netgo and osusergo already keep the name and user lookups in Go. tt built with mage linked the C library dynamically for nothing, and a distribution of the same core, built by tt binary build tt, did not. The magefile now takes CGO_ENABLED from internal/buildflags, as the builder does: on for openssl and openssl_static, which link OpenSSL through cgo, off for the default. mage build gives a static binary again, and the two ways to build tt agree. The GitHub workflows build the default and set nothing that needs cgo; the goreleaser configs still set CGO_ENABLED=1 themselves and are left for their own change.
The signer of the RSA provider signed a tt 2 environment: an env_hashes.json of bin/, modules/ and tt.yaml, which it required, and a hashes.json per application of instances.enabled with its config files and its executable files. A package of tt 3 has none of that, and the files tt reads through the checks - app.manifest.toml, the lock, the instances descriptions of .templates/ - were never listed, so even a signed tree failed the first read. Sign now signs the tree of a package as a whole: hashes.json lists what Tarantool loads, the Lua sources, the shared libraries, the cluster configs and the files SignOptions.Loaded names, such as the scripts of the instance records, and is what tt hands Tarantool; env_hashes.json lists every other file, among them the programs tt runs itself. Every file is listed exactly once, but for the directories SignOptions.Skip names and the hashes files themselves. SignFiles lists files under the paths they are to have once laid out, for the runtime a package bundles, which is installed apart from it. InitializeIntegrityCheck takes several roots, an application and the runtimes it runs on, and checks each once; a root without hashes files lists nothing, as before. VerifyTree checks that a whole tree is signed: its hashes files verify, the files they list match, and they list every file of it - the check of what is about to be installed, which an added file must fail as well as a changed one. The salt of RSASSA-PSS is random: signing a tree again gives the same hashes files and other signatures.
tt 2 signed what tt pack bundled with --with-integrity-check; tt package pack had no way to, so a package of tt 3 could not be run with the integrity checks at all. With the flag, which exists when the integrity provider of the distribution signs, the staged tree is signed before it is archived: each runtime component in _runtime/ gets an env_hashes.json listing its executable, and the headers of tarantool, by the paths tt package install lays them out at - a tt binary installation, the executable at the top - and the package gets its hashes.json and env_hashes.json, _runtime/ and tt's install state in .rocks/manifests/ left out, as install leaves them out of the version. A signed package runs only its own files. The instance records of its templates are checked when it is signed: a script or a binary named by a path must be inside the package, a program named without one must be in its .rocks/bin/, and the scripts are listed for Tarantool, which runs them, whatever their names. A package packed --without-deps is not signed: its install fetches dependencies nobody signed. A bundled component that holds its executable both in bin/ and at its top is refused, as the install would take one of them and the signature cannot say which. The salt of the signatures is random, so a signed archive is not reproduced bit for bit, unlike the unsigned one.
tt 2 checked a signed environment where it ran; nothing checked what was laid out in the first place, and a package of tt 3 had no signature to check anyway. With --integrity-check, tt package install installs only what the archive signs: the version is extracted into its staging directory and checked as a whole there - its hashes files verify, every file they list matches, and they list every file of it, so that a file added to the archive is refused as well as a changed one - before it is renamed into place. Each runtime component is checked the same way in the staging directory it is extracted into, in the layout of a tt binary installation it is signed for, together with its hashes file, which is now laid next to the executable whether or not the install checks it. A component already installed, by tt binary or an earlier install, is the one the application is to run on. Without hashes files of its own it gets those of the archive and is checked with them; with them it is checked with what it has. An installation the signature does not fit is an error at install time rather than a refusal at the first start, and the hashes files laid for the check are taken back. A signed install goes into the user or the system scope only: the install root of the project scope is shared with every guest package, whose hashes files would lay over each other. An archive packed --without-deps is refused with --integrity-check, as what its install fetches is signed by nobody.
The integrity checks were rooted at the project the working directory is in, or at the working directory itself, for every command. An instance command acts on applications that are elsewhere: an installed version, <root>/versions/<app>/<app>_<ver>, is signed as the package it was packed from and is checked nowhere else, so its files - the manifest, the description, the config - were refused as unknown wherever tt ran, and its hashes.json given to tarantool was not the one tt had checked. The watchdog, started detached, checked whatever directory it happened to inherit. app.Options gets IntegrityFor, which the instance commands set while the checks are on: Load roots the checks of an application at its directory once it is found, reads its pair through them and hands them on as App.Integrity. An instance carries them in its context, and its watchdog finds them again the same way where it reloads the instance. The specs of the instance and the periodic check of the watchdog use them rather than the checks of the command.
tt 2 listed bin/tt and bin/tarantool of a signed environment in its env_hashes.json and checked them like any of its files. In tt 3 the runtimes are not in the application: tt binary installs them into versions/ of the scope, a package bundles them for tt package install to lay there, and a project selects them in its .tarantool. So the tt that runs the watchdog of an instance was looked up in the checks of the project, where nothing lists it, and the start of every instance with the checks on failed. A runtime a signed package bundles is installed with its hashes files next to its executable. The checks of an application are now rooted at the directories of the runtimes it runs on as well - the tarantool it is resolved to and the tt that runs it - and the checks of the command at those of the tarantool, tt and tcm it found; a directory without hashes files adds nothing to them. The watchdog of an instance is started only by a tt its checks know, and the error says that the tt is not signed.
tt 2 read the tarantool of a signed environment through the checks
before it ran it; tt 3 lost that when the runtimes left bin/ of the
environment, and it ran whatever the instance records named: a
tarantool from PATH, a program of the host found in PATH, a script or
a binary anywhere on the disk.
With the checks on, an instance now runs only what the signatures of
its application and the runtimes it runs on list, each file checked
before it runs, both by the command before it starts the watchdog and
by the watchdog before every start:
- its tarantool, which must be a signed runtime: a tarantool from
PATH is refused before it is looked at;
- the script of a record that runs one, which tarantool then checks
with the hashes file of the application as well;
- the binary of an external process. A program named without a
slash is looked up in .rocks/bin/ of the application only, not in
PATH: the programs a package builds are signed with it, and a
program of the host is not.
The errors name the file and say that it is not signed. With the
checks off, nothing changes.
The pieces of the integrity checks of a package are tested where they are; nothing went through the commands a user runs from one end to the other with a key made by openssl, the way the documentation tells to make one. The test packs a package with --with-integrity-check, its runtime taken from the runtime cache - the tarantool of PATH, and the test binary itself as the bundled tt - refuses to install it into the project, installs it into the user scope with the public key, and checks that the version and each runtime got their hashes files. A tt from outside the signed runtimes does not start the watchdog, and the installed tt refuses a config of the version changed after the install, whichever directory it runs in. Whether Tarantool itself takes --integrity-check is for Tarantool Enterprise to say; the test does not start Tarantool.
The pair of the host, <root>/etc/<app>/, is the configuration of a host, not a part of the package, so the checks of an installed application list none of its files and refuse an application that runs from it. The test pins that, as the documentation of signed packages says it.
The README says how a distribution with the RSA integrity provider signs a package and runs it checked: the keys and their formats, what tt package pack --with-integrity-check lists in the hashes files of the package and of each runtime it bundles, what tt package install checks before it puts a version and a runtime in place, and what tt instance start runs with the checks on. It lists what is not supported yet - a signed install into the project scope, the signing of a project as it is developed, the pair of the host - and that a signed archive is not reproduced bit for bit. The changelog has the new flags and the changes of the checks and of the API of sdk/integrity.
tt --integrity-check KEY cluster show of a prefix whose data was
tampered with, or signed with another key, said that no configuration
data was found there. The integrity store of go-storage leaves out of a
range every value that fails its checks unless it is asked to keep
them, so the collector saw an empty prefix and reported it as one. The
data is there and is refused; the user has to know that it is the
signature, not the address, that is wrong.
The collector now reads with IgnoreVerificationError and reports the
first value that failed with ErrIntegrityCheckFailed, naming the
storage, the key and the reason; a read of one key does the same. A
watch skips a change it cannot read rather than indexing the empty
result of the failed read.
Found running Tarantool EE 3.7.0 against a configuration tt published
into etcd: EE refused the tampered value ("Integrity check failed,
invalid hash or signature by key"), while tt claimed the prefix empty.
An instance of an installed application left an empty var/lib/<instance>
in its version directory when it ran on Tarantool Enterprise, and in the
project directory when the project did: TestInstalledInstanceStartsFromItsVersion
and TestClusterInstanceStart failed on EE 3.7.0 and passed on CE.
Enterprise has a quiver engine whose quiver.dir, like the other data
directories, defaults to var/lib/{{ instance_name }} under the working
directory and is created at the start whether the engine is used or not.
tt gives the defaults of the WAL, the snapshot and the vinyl directories
but not this one.
An instance on Enterprise now also gets TT_QUIVER_DIR_DEFAULT, the data
directory of the instance where its other files go. Community refuses
quiver.dir even as a default ("available only in Tarantool Enterprise
Edition") and does not start with it, so it is given only to an
executable whose version banner says Enterprise - the banner, not the
product the executable was installed as, since what matters is what it
accepts.
The config of an instance may put its files elsewhere than the layout of
tt, and has to on Tarantool Enterprise with the integrity checks on: it
starts with an empty environment then, so the TT_*_DEFAULT variables tt
passes the layout through never reach it, and the defaults of Tarantool
put the data, the pid file and the console socket into the version
directory and no iproto socket anywhere. tt read the data directories
and the console socket from the config, but not all of what such a
config says.
The paths it reads get every variable Tarantool substitutes into the
config of an instance - {{ instance_name }}, {{ replicaset_name }} and
{{ group_name }} - and in every key it reads: app.file and the keys it
checks at start too. Only {{ instance_name }} was substituted, after the
paths were resolved, and a path with either of the others failed every
instance command with "missing vars".
The binary port is the first unix socket of iproto.listen in the config,
when the config lists one and the env of the record does not set
TT_IPROTO_LISTEN, which Tarantool ranks above the config and tt does not
parse. tt connect --binary, tt play and the cleanup at stop look where
the instance listens rather than where the layout would have it listen.
An installed instance is refused a relative path in its config only when
it leads into its version directory, which is never to be written to. A
relative path that leads out of it, such as
../../../var/lib/<app>/{{ instance_name }} to the layout of the scope
root, writes where an absolute path would, and is how a config names the
layout whatever the root of the scope it is installed into.
…erprise Tarantool Enterprise starts with an empty environment when it checks integrity, and the layout tt gives an instance through TT_*_DEFAULT variables does not reach it: the instance writes its data, its pid file and its console socket into its version directory and listens on no iproto socket. The README now says so and shows the part of the cluster config that puts them into the layout of the scope root - absolute paths under /opt/tarantool for the system scope, or paths relative to the version, which is always three levels under the root, for either scope - and why process.work_dir does not do: Tarantool looks for the modules of the application in its working directory and finds none once it moves. TestIntegrityEnterpriseLayout runs that on Tarantool Enterprise (TT_TEST_TARANTOOL_EE, or the tarantool of PATH when it is Enterprise): a package whose config names the layout of the user root is packed signed, installed and started with the checks on; its snapshot, pid file and sockets are in the layout and none of them in the version, and tt instance status, tt connect over the console socket and over the binary port, tt instance logrotate and tt instance stop work. A changed module of the version is refused at the next start.
The section said the private key of a signed package is PKCS #1 and told to make it with openssl genrsa -traditional. The RSA provider takes PKCS #8 private keys and PKCS #1 public keys as well since the keys stopped being PKCS #1 and PKIX only, so the section names both forms and drops -traditional: the key openssl 3 makes by default is taken as it is.
VerifyTree checks that a tree is signed as a whole: every file its hashes files list matches, and every file of the tree is listed. It walked a symbolic link that does not resolve as an error, the way a signature does. A signed version is to link the layout of its instances at var/, and the directories the links lead to are created only when an instance first starts, so the version failed the check of its own tree until then. A link that leads nowhere holds no file, so the check passes over it as it passes over a link to a directory. Nothing escapes that way: a listed file replaced by such a link is still checked by its digest when the hashes files are loaded, and fails there. Sign keeps refusing a link it cannot read, so that what is signed is never short of an entry.
Tarantool Enterprise starts with an empty environment when it checks integrity. The TT_*_DEFAULT variables tt passes the layout of an instance through do not reach it then, and the paths of the instance fall back to the defaults of Tarantool, under its working directory, the installed version: var/lib/<inst>, var/run/<inst>/tarantool.pid, var/run/<inst>/tarantool.control, var/log/<inst>. VarLinks names links that turn those defaults into the layout of the scope root: var/lib, var/run and var/log of the version lead to <root>/var/lib/<app>, <root>/var/run/<app> and <root>/var/log/<app>. The names of the files Tarantool writes there are the names the layout of tt gives them, so nothing else has to agree. LinkVar makes the links a version lacks, relative like the link to the active version so that the root may move, and CheckVarLinks checks them: what stands at their place and is not the link, a directory the instances wrote into or a link elsewhere, is ErrVarLink. Paths are compared as they resolve, so a root reached through a link of its own still matches.
A signed version runs with the integrity checks on, on Tarantool Enterprise, which drops the environment tt passes the layout through and puts the files of an instance where its defaults say, under the version. A signed install now gives the version the links of ttscope.LinkVar once it is in place, so the data, the pid file and the console socket land in the layout of the root, where tt looks for them, and a config that names no path runs checked as it runs unchecked, as in tt 2. The links are made after the archive is checked, and are not part of what was signed. Their targets are not created: tt creates the directories of an instance when it starts it, as the user the instance runs as, which in the system scope is not the user that installs. Every version links the same directories, so the data outlives an upgrade; a version replaced by --force and an uninstalled application lose their links, not the data, since removing a tree does not follow the links in it. An install that is not signed is as before.
An installed signed version links the layout of its instances at var/, so a signed package that has its own var/ at the top would collide with the links; the pack refuses it, as it refuses the names of the hashes files. A package that is not signed may still have one. Tarantool Enterprise starts with an empty environment when it checks integrity, so the env of the record of a Tarantool instance never reaches the instance of a signed package. The pack refuses such a record and says to put the values into the config. An external process keeps its env: tt runs it, not Tarantool.
With the integrity checks on, Tarantool Enterprise starts with an empty environment and finds the layout of an instance installed into the user or the system scope only through the links of its version. Before every start - the one of the command, which reports to the user, the first one of the watchdog and each of its restarts - tt makes the links the version lacks, which is a version installed before tt made them, when it may write into it, and says how to get them when it may not. checkStart refuses a version that has something else in their place: a link elsewhere, or a directory an instance wrote into. Two records cannot work with the environment gone, and are refused under the checks rather than started to fail: the env of the record of a Tarantool instance, which does not reach it, and a record that runs its script, whose launcher takes the script and the paths of the instance from the environment and only logs "Failed to get instance path" while tt instance start reports success. An external process keeps its env: tt runs it. Without the checks, in a project and for an external process nothing changes.
…rprise TestIntegrityEnterpriseDefaultLayout packs a signed package whose config names no path, installs it into the user scope and runs its instance with the integrity checks on Tarantool Enterprise, which starts with an empty environment. var/ of the version holds nothing but the links to the layout; the snapshot, the pid file and the console socket are in the layout of the root; status, connect, logrotate and stop work. A newer version installed next to it links the same layout, and the instance started from it has the tuple the old one stored. A changed module is refused, and an uninstall removes the versions and keeps the data. TestIntegrityEnterpriseDefaultLayoutEtcd runs the same instance with its config published, signed, into etcd. TestIntegrityEnterpriseLayout, whose config names the paths itself, now finds the links in the version rather than no var/ at all, and checks that they are all there is.
The README said a package meant to run checked on Tarantool Enterprise has to name the layout in its config. A signed install now links the layout from the version, so the section tells that a config with no path keeps its files in the layout, how the links come and go with the versions, that a config may still name the paths, and what the checks still take away with the environment, as they did in tt 2: no iproto address by default, the log in tt.log, no env for a Tarantool instance and no script records. The section on signed packages names the two new refusals of the pack, and the changelog follows.
…tion tt cluster publish and tt cluster show --validate refused a valid Tarantool 3 config whose replicaset sets leader (manual failover) or bootstrap_leader, saying that the additional property does not match the schema. Tarantool takes the config and starts the cluster. Besides the cluster config as a whole, the validation checks the config of every instance, folded from the global section, the group, the replicaset and the instance, against the same schema of the cluster config. That schema has leader and bootstrap_leader on the replicaset only, while the folded config of an instance carries them at its top, so every instance of such a replicaset failed. The folded config is now checked by cluster.ValidateInstance, against the schema with the keys of the replicaset schema (everything but its instances) added to the top level. The level of a key is still checked in the cluster config as a whole, where leader and bootstrap_leader on the top, in a group, in an instance and in a section such as replication are refused, as Tarantool 3.8 CE and 3.7 EE refuse them. An unknown key and a value of a wrong type are refused as before.
IsBaseURI and IsCredentialsURI knew the path forms and the unix:// forms with anything after them, but a tcp address ended at the port: host:port?transport=ssl&ssl_ca_file=ca.pem, with or without tcp:// and user:password@, was no URI at all. tt connect then tried it as APP:INSTANCE, split it at the first colon, and reported the rest as "cannot be an instance name", the password included. Accept a query after the port of a tcp address, as Tarantool does for its listen and connection URIs, and accept the unix/:path form that ParseBaseURI already understood but IsBaseURI refused, with or without a query. SplitURIQuery cuts the query off such an address for the caller to apply; the path forms and unix:// keep a "?" as a part of the path, as they did. Its error does not quote the query, which may hold ssl_password. Add RedactCredentials for the messages that must name what a user typed: it hides everything before the last "@" without parsing, so it also works on a password with a "/" or an "@" and on a piece of a URI cut at a colon. CreateURIOpts used to return the *url.Error of the parser as it was, quoting the whole URL with its password; it now redacts the URL of that error.
RedactURL needs a string that parses as a URL, and a Tarantool target often does not: user:password@host:3301 is opaque to net/url, and a caller that cuts a target at its first colon, as the resolver of APP:INSTANCE does, holds a piece such as password@host:3301 that no pattern of a URI recognises. A message that quoted such a piece gave the password away although the target as a whole was masked. RedactTarget hides everything before the last "@" without parsing, so it also covers a password with a "/" in it, and then applies the registry of secrets. It leans to hiding too much, the host after an "@" in a query among it, which is the right side for a message about a string the user got wrong.
tt connect 'user:secretpw@127.0.0.1:3311?transport=ssl&ssl_ca_file=/x' printed '"secretpw@127.0.0.1:3311?..." cannot be an instance name'. The string was no URI to IsBaseURI and IsCredentialsURI, so it went to the resolver of APP:INSTANCE, which cut it at the first colon and quoted the instance side; the error masked the target as a whole but not that piece. With the query accepted by the URI check, tt connect takes the TLS settings of the query (transport, ssl_key_file, ssl_cert_file, ssl_ca_file, ssl_ciphers, ssl_password, ssl_password_file) as it takes them from the flags, refuses a setting given twice and any other parameter, naming the key and never the value, and shows neither the credentials nor the query in the title of the console. A target that reaches the resolver as a URI with credentials is refused as a connection URI, and one that is neither an instance nor a URI is reported by a redacted error, rather than by the error of the lookup that names the pieces it cut. The same holds for the other places that quote what the user typed in place of a URI: the application name of Load, the cluster-level target resolver, tt play, the "invalid URL" errors of tt cluster and tt replicaset, the cluster config source of tt cluster and the endpoints of a Tarantool config storage.
Go applies the replace directives of the main module only, so a tt module that depends on a library that is not published, or not at the version it needs, cannot carry the replacement in its own go.mod, and a distribution of such a module had to be fixed by hand in the go.mod the build generates. [replace] maps a Go module path, optionally with @Version, to a local directory (./, ../ or /) or to module@version. The parser and the validation check the shape of both sides and refuse the core: it has [core].path and --core. A module of the core's repository and a path dependency are known to the build only, which refuses them. The lock gets a [replace] table to pin the entries: the exact version of a module, the content_hash of a directory.
The entries of [replace] become replace directives of the generated go.mod, written before go mod tidy. A directory must hold the module it replaces, as the go command demands; a module version query is resolved to an exact version, or taken from the lock. The core, the modules of its repository and the [dependencies] modules are refused with a hint at the place that picks them. The lock records a directory by the content_hash of its files, hashed as a path module is, and a module by its version; a build from the lock fails when the directory changed. --vendor copies the directories into local/replace/ of the bundle with a relative replace, so the bundle builds offline. An entry no package of the build comes from is reported with a warning, read from the tidied go.mod: go list -m all would add checksums to go.sum and make a locked build differ.
The help of tt binary build tt, the README and the changelog tell the format of [replace], what a key and a value are, why the core and the tt modules are not replaced there, and what the lock, --locked and --vendor do with the entries.
TestBuildReplace replaces uuid by a version query, github.com/google/ uuid@v1.5, and only the module proxy resolves a query: with GOPROXY=off the test failed on "module lookup disabled" while the other end to end tests of the builder pass, since they build from the module cache alone. The test now skips itself when GOPROXY is off, so that an offline run of the integration tests stays green, and runs in full wherever the proxy is reachable.
go-storage v2.0.0 refused WithSingleSigCompact for a codec that only verifies, so the storage gave the namers CompactSingleSig itself through WithNamer when it took the keys of tt 2 back. go-storage now counts the union of the signers and the verifiers instead (3d28af2 on its v2 branch), and the storage asks the codec builder for the compact signature key whenever there is one signature, which is when the namers got it before. The namer still keeps the object location out of the hash and the signature keys (namer.LegacyHashSigLayout): the builder has no option for that. go-storage is taken at the pseudo-version of that commit until v2.1.0 is tagged.
go-storage now builds its TLS to a Tarantool config storage on go-tlsdialer: in pure Go by default and on OpenSSL with the openssl tag the OpenSSL builds of tt set anyway; go_storage_ssl is a deprecated synonym of openssl. The OpenSSL builds no longer set it. The tag was added for the cluster commands of the core as well, but they connect to a config storage through sdk/dial (sdk/cluster/tcs.go) and never used the TLS of go-storage: only modules that connect through go-storage/connect did. The changelog entry says so now.
tt version of a distribution shows the core it is built on, and takes
its version from the module version Go recorded. A core from a
directory, [core].path or --core, is a replaced module, for which Go
records (devel), so a distribution built on a checkout of tt - tt-ee
with TT_SRC, say - showed
Tarantool CLI core version <unknown>. commit: <unknown>
The build now runs git describe in the directory of such a core and
stamps the result into cli/version with the -X flags the magefile sets
for tt itself; a core with none of those variables is built as before.
The flags are taken before the core is copied into a bundle, so a
--vendor bundle, which carries no .git, builds with them too. A core
of a version is left as it was: Go records its version.
The package and the variables are spelled out once, as
buildflags.Version, which the magefile now uses for its own stamp.
tt.manifest.lock recorded [build] - the toolchain, the TLS implementation and the tags of the last build - without --locked ever checking them, and every build but a --locked one wrote the lock anew. So a build with TT_CLI_BUILD_SSL=openssl_static after a default one changed ssl and tags in the lock, a build with another patch release of Go changed toolchain, and a build by another tt changed generated_by: a committed lock went dirty with every build while nothing it pins had changed. The lock now pins what the binary is built from - the core, the modules, the replacements and the files of the local ones - and leaves how it is compiled to each build: one lock builds every TLS implementation. The go directive of the generated module still sets the lowest Go a build takes, and the exact toolchain stays where an audit looks for it, in the build info of the binary and in build.env of a bundle. A build that would write the lock and the sum there are, but for generated_by, leaves both as they are; a lock with fields tt no longer writes, [build] of an earlier lock among them, is written anew.
content_hash of a local module, a local core and a directory of [replace] covered every regular file of the directory, so what a build or an editor leaves there - a ./tt built in a checkout of tt, 55 MB that its .gitignore ignores - counted as a change, and --locked failed on a checkout where nothing tracked had changed. The bundle of --vendor copied the same files, the binary among them. The files of a module are now listed once, by moduleFiles, for both the hash and the copy in a bundle: what the hash covers is what the bundle carries. The list leaves out what the .gitignore files ignore: those of the module, each for its subtree, and those of the directories above it up to the root of its git repository or, outside a repository, up to the directory of the manifest when the module is in it. The rules are read with the gitignore matcher of go-git rather than by git ls-files, so that a tarball of a checkout, which has no .git, hashes as the checkout does. core.excludesFile and .git/info/exclude are not read: each machine has its own, and the same checkout would hash otherwise on CI. A file git tracks that .gitignore ignores does not count. What the go command may build the module from counts even when ignored: the Go and cgo sources, go.mod, go.sum and the files the //go:embed directives of its packages embed under any build constraint, resolved as the go command resolves them. Leaving one out would let it change unnoticed, and a bundle without an embedded file would not build offline, so the walk goes into ignored directories for them. What tests embed is not among them: no test is built into the binary or vendored. The .git file of a worktree or a submodule, which names a directory of the machine, is no longer hashed or copied either.
The help of tt binary build tt, the README and the changelog tell that content_hash and the bundle of --vendor leave out what .gitignore ignores, which .gitignore files are read and which excludes are not, and which files count even when ignored.
The changelog told what tt binary build tt, signed packages and the redaction of credentials do for a user, but not what they changed in the Go API of tt for the programs built on it: core.WithIntegrity, through which a distribution chooses the integrity provider, the redaction helpers of sdk/connect and sdk/log, ErrIntegrityCheckFailed of sdk/cluster, and the removal of cmd.InjectedCmds, through which tt-ee 2 replaced commands of tt. The removal breaks such a program, so it goes under Changed.
tt connect checked that the transport of the query of a URI was ssl or
plain and then dropped it: TLS was taken only when an ssl_* parameter
or a --ssl* flag was set. So
tt connect 'user:pass@host:port?transport=ssl'
to a TLS port with no CA file went plain and failed to read the
greeting, which a TLS port does not send before the handshake, unless
some --ssl* flag, --sslciphers for one, was added. transport=plain with
a TLS setting connected over TLS, and transport=ssl to a plain port
connected without TLS, both without a word.
The transport now reaches the connector through ConnectCtx and SslOpts,
and the connector chooses TLS by it: ssl takes TLS with no setting at
all, plain refuses every setting, and with no transport TLS is taken
when a setting is set, as before. A connection with TLS chosen goes the
way the TLS settings take, for the console and for -f alike: a server
silent before the handshake is dialed as a binary one over TLS, the
dial.Opts naming the transport the connector chose, and a server that
greets is refused as unencrypted. tt connect refuses transport=plain
with a TLS setting before it dials, naming the parameter of the query
or the flag that sets it, never its value. tt replicaset and tt backup,
which resolve a URI as tt connect does, do the same.
cli/status may import sdk/dial now, for the transport it spells out in
the SslOpts of its connection.
The default build of tt dials TLS through the gostls backend of go-tlsdialer, and it did not act as the OpenSSL builds, nor as tt 2, which was built on OpenSSL: without a CA file it checked the chain of the server against the system roots, so a server with a self-signed certificate was refused; it refused a private key in the traditional encrypted PEM of OpenSSL; and a server that refused the client certificate was reported as "expected ChangeCipherSpec, got content type 21", or now and then as a broken pipe. go-tlsdialer v2.1.0, on go-gostls v0.1.1, acts as its OpenSSL backend: without a CA file the chain goes unchecked and the name of the server is still checked, with the fallback to the common name OpenSSL has; the keys OpenSSL loads are taken; and a refusing server is reported by its alert, "remote error: tls: handshake failure". The test of transport=ssl without any other TLS setting no longer points SSL_CERT_FILE at the CA of the test: no system root is looked at now.
sssciel
approved these changes
Oct 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes tt buildable as a distribution: the core with tt modules compiled in, described by
tt.manifest.tomland built bytt binary build tt. tt-ee 3 is the first such distribution; what it needs from the core lands here, so that tt-ee only adds modules and changes nothing of tt.Distributions
tt.manifest.tomlnames the core ([core].tt: version, branch, commit, or[core].path), the modules ([dependencies]), the title, the edition and the integrity provider.tt binary build ttresolves them, generates the main module, builds and smoke-runs the binary, writestt.manifest.lockandtt.manifest.sum. The lock pins what the binary is built from, not how it is compiled; local modules are locked bycontent_hash, which leaves out what.gitignoreignores and is not rewritten when nothing it pins changes.--vendor DIRmakes an offline bundle: the main module with everything vendored andbuild.sh, built by the Go toolchain alone.--corebuilds on another core without touching the lock.[replace]replaces a Go module the build depends on.tt versionshows the core and, with--modules, the modules; a core from a directory is stamped fromgit describe.Integrity
core.WithIntegritylets a distribution choose the integrity provider;cmd.InjectedCmdsis removed.pack, verification atinstall, and only signed files are run. A signed install linksvar/{lib,run,log}to the version's layout, which Tarantool Enterprise needs because it clearsTT_*_DEFAULTfrom the environment.Build and fixes
internal/buildflagsspells out the tags and linker flags once; the default build is pure Go without cgo, andgo_storage_sslis gone.v2pseudoversion until v2.1.0 is tagged, go-tlsdialer v2.1.0 on go-gostls v0.1.1: TLS in pure Go behaves as the OpenSSL builds and tt 2 do.tt cluster publishand validation acceptleader/bootstrap_leader. Passwords no longer leak into errors and logs;?transport=sslin a URI now turns TLS on.API
Added
core.WithIntegrity,sdk/cluster.ErrIntegrityCheckFailed,sdk/connect.RedactCredentials,sdk/log.RedactTarget. Changedsdk/integrity(alpha):Signer.Sign(dir, SignOptions),Signer.SignFiles,Provider.VerifyTree,InitializeIntegrityCheck(pub, roots...). Removedcli/cmd.InjectedCmds.