Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,10 @@ Read the established Logseq operating procedure before touching the user's actua
## Secret gate before publication

Before publishing the current PR branch, commit the complete candidate and run
`python3 -I scripts/secret_gate.py --base refs/remotes/origin/main` (native Windows: `python -I`). Require exit 0;
`python3 -I scripts/secret_gate.py --base refs/remotes/origin/main`.
On native Windows PowerShell, run
`& "$env:MTG_SECRET_GATE_PYTHON" -I scripts/secret_gate.py --base refs/remotes/origin/main`.
Require the gate's exit code 0 (`$LASTEXITCODE` in PowerShell);
missing tooling/authentication, scan errors and skipped scans block publication.
Then run the existing repository gates. Repeat after changing outgoing commits.
Push exactly the scanned HEAD. This PR scan covers the current branch against its
Expand Down
68 changes: 65 additions & 3 deletions SECRET-SCANNING.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,8 @@ python3 -I scripts/secret_gate.py --all-history
python3 -I scripts/test_secret_gate.py -q
```

Use `python` instead of `python3` on native Windows. Python 3.11+, Git, and
On native Windows, invoke the approved absolute interpreter with PowerShell:
`& "$env:MTG_SECRET_GATE_PYTHON" -I scripts/secret_gate.py --base refs/remotes/origin/main`. Python 3.11+, Git, and
ggshield on PATH are required. Require ggshield **>=1.56.1,<2**; this gate was tested with **1.56.1**.
Install ggshield through the approved development-tool environment. Use existing
native ggshield authentication or an approved process-injected
Expand Down Expand Up @@ -62,10 +63,10 @@ Only exit **0 / CLEAN** means a successful scanner run. Other outcomes block:
| 1 | Findings; remediate before publication |
| 2 | Invalid gate arguments |
| 3 | Authentication failed/unavailable; owner must restore approved native auth |
| 4 | Scan/transport/server/timeout error, or HEAD changed during the scan |
| 4 | Scan/transport/server/timeout error, or HEAD/base changed between initial resolution and final verification |
| 10 | Skipped: empty range or deletion-only input; no clean scan claimed |
| 20 | Invalid/unavailable base, Git operation, history or hook input |
| 21 | Required executable unavailable |
| 21 | Missing/untrusted executable, unsupported Python, or unverifiable/unsupported ggshield version |

No skip or fail-open switch is provided. Strict temporary scanner configuration
ignores local/global exclusions and exit-zero settings; dotenv loading and scanner
Expand Down Expand Up @@ -122,3 +123,64 @@ Explicit `--base` accepts only a fully qualified ref (for example,
`refs/remotes/origin/main`) or a full commit SHA. Short names such as
`origin/main` block with exit 20 because a local branch can shadow them.
Unsupported metadata path encodings also block with BASE_ERROR (exit 20).


Git/scanner credential boundary: Git children launched directly by this gate do not receive ggshield credential
variables. ggshield receives the approved API key/instance selectors, and Git
processes launched by ggshield can inherit them.
Git trace and executable proxy overrides are removed. Existing native Git
configuration and credential helpers remain trusted prerequisites. Explicit
global/system config selectors must resolve to approved absolute paths outside
this checkout and its related worktrees; unsafe selectors block without falling
back to another credential context. This gate is not an operating-system sandbox.

Metadata reads reject non-regular records, NUL path bytes and missing registered
worktree pointers. The opener is nonblocking where supported and validates its
descriptor. Grafts symlinks are confined to Git metadata. An empty explicit base
blocks rather than selecting a different destination.


Askpass selectors follow the same external absolute-path rule as explicit Git
configuration selectors; unsafe values block without changing authentication
contexts. Native approved external askpass programs remain available. Git metadata
symlinks, malformed registrations and unreadable registries block. Metadata opens
use O_NOFOLLOW where supported; platforms without it retain pre-open confinement
and descriptor checks. Concurrent replacement of parent directories is outside
the gate isolation guarantee.

The Python interpreter is a trusted bootstrap prerequisite. Use an approved
installation outside all repository checkouts. On Windows, use its absolute path
for direct invocations (for example, `& "$env:MTG_SECRET_GATE_PYTHON" -I scripts/secret_gate.py --base refs/remotes/origin/main`).
Set that variable to the existing approved installation; it is not a credential.


Tool candidates are checked against their own Git association as well as the
registered roots. A stale or rewritten registration cannot admit a scanner while its
Git-file, directory or common-directory association still identifies it as
a linked checkout. Obsolete registrations are left untouched; this gate does not
prune other tasks worktree metadata.


Windows Make/Just gate targets require MTG_SECRET_GATE_PYTHON to name an existing
approved absolute Python installation outside all Git checkouts. Configure the
selector in the trusted caller environment, never from checkout-provided values.
Unix direct entrypoints require the same approved interpreter prerequisite.


Tool association follows commondir for both Git-file and directory metadata.
Inaccessible PATH candidates are excluded using explicit lstat calls, including
on Python versions whose type predicates suppress permission errors.

Approved installation trust boundary: tool ownership checks do not establish
executable provenance. Supply approved external installations through the trusted
caller PATH, never checkout-provided environment values. If both a checkout
marker and its registration are removed or rewritten, the gate cannot reconstruct
that former ownership from an ordinary directory. Tampering with approved tool
installations or their caller environment is outside this gate guarantee.
Obsolete shared worktree registrations are left untouched.

An explicitly empty GIT_ASKPASS is preserved as native Git's helper-disable
value; it is never removed to select a different credential context. An empty
SSH_ASKPASS still blocks with exit 20 under this gate's explicit SSH-selector
contract. Other explicit selectors require approved external absolute
paths. Do not alter authentication selectors merely to obtain a clean scan.
136 changes: 121 additions & 15 deletions scripts/secret_gate.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,13 @@
import os # noqa: E402
import re # noqa: E402
import shutil # noqa: E402
import stat # noqa: E402
import subprocess # noqa: E402
import tempfile # noqa: E402
from pathlib import Path # noqa: E402

MINIMUM_PYTHON = (3, 11)
GITDIR_PREFIX = "gitdir:"


class GateError(Exception):
Expand All @@ -31,7 +33,13 @@ def metadata_path(path, anchor):
"""Confine metadata access, including symlinks, to its allowed Git directory."""
try:
anchor = anchor.resolve()
relative = path.resolve().relative_to(anchor)
path = path.absolute()
if any(p.is_symlink() for p in (path, *path.parents) if p.is_relative_to(anchor)):
raise ValueError("metadata symlink")
resolved = path.resolve()
relative = resolved.relative_to(anchor)
if any(p.is_symlink() for p in (resolved, *resolved.parents) if p.is_relative_to(anchor)):
raise ValueError("unresolved metadata symlink")
except (ValueError, OSError, RuntimeError) as error:
raise GateError(20, "BASE_ERROR: Git metadata escapes its allowed directory.") from error
return anchor / relative
Expand All @@ -40,23 +48,50 @@ def metadata_path(path, anchor):
def metadata_text(path, anchor):
"""Read only a bounded, confined Git metadata record; never display it."""
path = metadata_path(path, anchor)
with path.open("rb") as stream:
data = stream.read(8193)
try:
if not path.is_file():
raise GateError(20, "BASE_ERROR: missing or non-regular Git metadata.")
with open(path, "rb", opener=metadata_opener) as stream:
if not stat.S_ISREG(os.fstat(stream.fileno()).st_mode):
raise GateError(20, "BASE_ERROR: non-regular Git metadata.")
data = stream.read(8193)
except (OSError, ValueError) as error:
raise GateError(20, "BASE_ERROR: unreadable Git metadata.") from error
if len(data) > 8192:
raise GateError(20, "BASE_ERROR: oversized Git metadata record.")
if b"\0" in data:
raise GateError(20, "BASE_ERROR: invalid Git metadata path.")
try:
return data.decode("utf-8").strip()
except UnicodeError as error:
raise GateError(20, "BASE_ERROR: unsupported Git metadata path encoding.") from error


def metadata_opener(path, flags):
"""Avoid blocking on a FIFO swapped in before descriptor validation."""
return os.open(path, flags | getattr(os, "O_NONBLOCK", 0) | getattr(os, "O_NOFOLLOW", 0))


def linked_roots(common):
"""Read registered worktree pointers inside this Git metadata directory."""
roots = set()
for pointer in (common / "worktrees").glob("*/gitdir"):
registry = metadata_path(common / "worktrees", common)
if not registry.exists():
return roots
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
if not registry.is_dir():
raise GateError(20, "BASE_ERROR: non-directory worktree registry.")
try:
entries = list(registry.iterdir())
except OSError as error:
raise GateError(20, "BASE_ERROR: unreadable worktree registry.") from error
for entry in entries:
entry = metadata_path(entry, common)
if not entry.is_dir():
raise GateError(20, "BASE_ERROR: non-directory worktree registration.")
pointer = entry / "gitdir"
try:
target = (pointer.parent / metadata_text(pointer, common)).resolve()
except (OSError, RuntimeError) as error:
except (OSError, ValueError, RuntimeError) as error:
raise GateError(20, "BASE_ERROR: invalid linked-worktree metadata.") from error
if target.name != ".git":
raise GateError(20, "BASE_ERROR: unsupported linked-worktree metadata.")
Expand All @@ -74,32 +109,70 @@ def checkout_roots():
gitdir = marker
if marker.is_file():
text = metadata_text(marker, root)
if not text.startswith("gitdir:") or "\n" in text:
if not text.startswith(GITDIR_PREFIX) or "\n" in text:
raise GateError(20, "BASE_ERROR: unsupported Git worktree metadata.")
gitdir = (root / text[len("gitdir:") :].strip()).resolve()
gitdir = (root / text[len(GITDIR_PREFIX) :].strip()).resolve()
gitdir = gitdir.resolve()
owner = next((p for p in (gitdir, *gitdir.parents) if p.name == ".git"), None)
if owner is None:
raise GateError(20, "BASE_ERROR: unsupported Git metadata layout.")
common_file = metadata_path(gitdir / "commondir", owner)
common = (
metadata_path(gitdir / metadata_text(common_file, owner), owner)
if common_file.is_file()
if common_file.exists()
else gitdir
)
if not (common / "objects").is_dir():
raise GateError(20, "BASE_ERROR: unsupported Git metadata layout.")
return {root, owner.parent, common, gitdir} | linked_roots(common) | linked_roots(owner)


def metadata_owned(directory, roots):
"""Recognize Git associations and exclude unsearchable tool candidates."""
marker = directory / ".git"
try:
try:
marker.lstat()
except (FileNotFoundError, NotADirectoryError):
return False
if marker.is_dir():
gitdir = marker.resolve()
elif marker.is_file():
text = metadata_text(marker, directory)
if not text.startswith(GITDIR_PREFIX) or "\n" in text:
raise GateError(20, "BASE_ERROR: unsupported tool-checkout metadata.")
gitdir = (directory / text[len(GITDIR_PREFIX) :].strip()).resolve()
else:
return True
if any(gitdir.is_relative_to(root) for root in roots):
return True
common_file = gitdir / "commondir"
try:
common_file.lstat()
except FileNotFoundError:
return False
common = (gitdir / metadata_text(common_file, gitdir)).resolve()
return any(common.is_relative_to(root) for root in roots)
except OSError:
# lstat retains permission errors on Python versions whose predicates
# suppress them. Exclude the candidate before constructing child PATH.
return True


def checkout_owned(path, roots):
"""Reject code owned by this repository or its linked worktrees."""
return any(path.is_relative_to(root) for root in roots)
return any(path.is_relative_to(root) for root in roots) or any(
metadata_owned(directory, roots) for directory in (path, *path.parents)
)


def command_environment():
"""Bind children to the current checkout and approved import directories."""
env = {name: value for name, value in os.environ.items() if not name.startswith("PYTHON")}
env = {
name: value
for name, value in os.environ.items()
if not name.startswith(("PYTHON", "GITGUARDIAN_", "GG_", "GIT_TRACE"))
}
for name in (
"GIT_DIR",
"GIT_WORK_TREE",
Expand All @@ -115,13 +188,33 @@ def command_environment():
"GIT_EXEC_PATH",
"GIT_SSH",
"GIT_SSH_COMMAND",
"GIT_PROXY_COMMAND",
"GIT_CURL_VERBOSE",
"PYTHONPATH",
"PYTHONHOME",
"PYTHONUSERBASE",
"PYTHONSTARTUP",
"PYTHONINSPECT",
):
env.pop(name, None)
for name in (
"GIT_CONFIG_GLOBAL",
"GIT_CONFIG_SYSTEM",
"GIT_ASKPASS",
"SSH_ASKPASS",
):
value = env.get(name)
if name == "GIT_ASKPASS" and value == "":
continue
if value is not None and os.path.normcase(value) != os.path.normcase(os.devnull):
path = Path(value)
if not path.is_absolute() or checkout_owned(path.resolve(), checkout_roots()):
raise GateError(
20,
"BASE_ERROR: Git config/askpass override requires an approved absolute path "
"outside the checkout.",
)
env[name] = str(path.resolve())
env["PYTHONSAFEPATH"] = "1"
return env

Expand Down Expand Up @@ -210,15 +303,24 @@ def base_commit(ref):
return commit(ref)


def active_grafts(common):
"""Read confined graft metadata with the documented failure classification."""
grafts = metadata_path(common / "info/grafts", common)
try:
return grafts.exists() and grafts.stat().st_size
except OSError as error:
raise GateError(20, "BASE_ERROR: unreadable graft metadata.") from error


def ranges(args, stream, head):
"""Select complete PR or pre-push commit ranges."""
grafts = Path(git("rev-parse", "--git-common-dir")) / "info/grafts"
if (grafts.exists() and grafts.stat().st_size) or os.environ.get("GIT_GRAFT_FILE"):
common = Path(git("rev-parse", "--git-common-dir"))
if active_grafts(common) or os.environ.get("GIT_GRAFT_FILE"):
raise GateError(20, "BASE_ERROR: grafted history is unsupported; restore complete history.")
if git("rev-parse", "--is-shallow-repository") != "false":
raise GateError(20, "BASE_ERROR: shallow history; fetch --unshallow before scanning.")
if not args.pre_push:
base = base_commit(args.base) if args.base else None
base = base_commit(args.base) if args.base is not None else None
if not base and not args.all_history:
base = remote_base(args.remote)
return [f"{base}..{head}" if base else head]
Expand Down Expand Up @@ -338,7 +440,7 @@ def verify_snapshot(args, head, selected):
)
if not args.pre_push and not args.all_history and ".." in selected[0]:
scanned_base = selected[0].split("..", 1)[0]
current_base = base_commit(args.base) if args.base else remote_base(args.remote)
current_base = base_commit(args.base) if args.base is not None else remote_base(args.remote)
if current_base != scanned_base:
raise GateError(4, "SCAN_ERROR: base changed; rerun for the new destination.")

Expand Down Expand Up @@ -369,7 +471,11 @@ def main(argv=None, stream=None):
args = parser.parse_args(argv)
try:
if sys.version_info < MINIMUM_PYTHON:
raise GateError(21, "UNAVAILABLE: use supported Python 3.11 or newer.")
raise GateError(
21,
"UNAVAILABLE: use supported Python "
f"{MINIMUM_PYTHON[0]}.{MINIMUM_PYTHON[1]} or newer.",
)
if not executable("git"):
raise GateError(21, "UNAVAILABLE: install Git before running this gate.")
head = None if args.pre_push else commit("HEAD")
Expand Down
Loading
Loading