Unit-test CI job killed at 99% (~1 in 10 runs since Oct 12): each test module leaks a full SupersetApp #45362

What's happening

Since 2026-10-12 the Python-Unit → unit-tests (current) job intermittently dies at the very end of the suite. The step reaches [ 99%], then the log ends with The operation was canceled (one run also logged The runner has received a shutdown signal), and the job lingers ~26–28 minutes before being cancelled, which is how GitHub reports a lost runner. All tests that ran had passed.

From the Actions API: 50 such kills against ~500 successful runs between Oct 12 and Oct 15, including master (e.g. runs 38502343489 and 38563726398 on Oct 14). None among the 278 cancelled runs scanned from the xdist switch (2026-09-12, #44178) to Oct 11. Some branches were hit repeatedly. Example PR runs: https://github.com/apache/superset/actions/runs/38619149658, https://github.com/apache/superset/actions/runs/38752136752 (both attempts).

Root cause

Each xdist worker's memory grows by ~10 MB per test module and reaches 5–7 GB per worker by the end of the suite (857 modules). Two workers (-n auto resolves to 2) plus the controller exceed what a standard hosted Linux runner provides (16 GB per GitHub's published spec).

The unit app fixture is module-scoped (tests/unit_tests/conftest.py) and builds a full SupersetApp per module. Those apps are never freed (228 live apps after 110 modules, measured with gc), because two module-level singletons keep references to every app ever created:

1. SupersetAppInitializer.configure_celery() subclasses whatever celery_app.Task currently is and replaces it. Each init wraps the previous wrapper, and each AppContextTask closes over its app:
   

   superset/superset/initialization/__init__.py

   Lines 145 to 171 in e0e728e

   	def configure_celery(self) -> None:
   	celery_app.config_from_object(self.config["CELERY_CONFIG"])
   	celery_app.set_default()
   	superset_app = self.superset_app
   	# Here, we want to ensure that every call into Celery task has an app context
   	# setup properly
   	task_base = celery_app.Task
   	class AppContextTask(task_base): # type: ignore
   	# pylint: disable=too-few-public-methods
   	abstract = True
   	# Grab each call into the task and set up an app context, unless
   	# one is already active on this thread (e.g. Celery eager mode
   	# invoked from within an existing request/test context) - Flask-
   	# SQLAlchemy 3.x scopes db.session by the active app context's
   	# object identity rather than by thread, so pushing a redundant
   	# nested context here would silently hand the task a second,
   	# blind session unable to see the caller's uncommitted work.
   	def __call__(self, *args: Any, **kwargs: Any) -> Any:
   	if has_app_context():
   	return task_base.__call__(self, *args, **kwargs)
   	with superset_app.app_context():
   	return task_base.__call__(self, *args, **kwargs)
   	celery_app.Task = AppContextTask

2. init_views() adds menu items whose lambdas reference the initializer/app; the fixture resets appbuilder.baseviews but not appbuilder.menu, so menu items accumulate per app.

The defect is old (the Celery wrapping is from 2021, the module-scoped fixture from 2022). It started failing now because the suite grew quickly (about 776 → 856 modules and 21.5k → 27.5k tests between Oct 3 and Oct 15), and the leak costs per module. Large transient tests add peaks on top (e.g. test_validation_extraction_is_bounded[100_000-*] allocates 1.5–2.6 GB for over a minute).

Locally, fixing both retention roots makes each module's app collectable: with the fix, full-suite per-worker peaks drop from ~5.1–5.5 GB to ~3.1–3.6 GB (two workers, CI command), with no new test failures.

Fix

A PR is coming (will link it here). The fixture can stay module-scoped once the two roots are fixed:

• make configure_celery subclass the original celery_app.Task instead of the current one, so re-initialization replaces rather than chains (no change for a single init in production);
• reset appbuilder.menu in the fixture;
• a regression test that initializing twice doesn't grow the task class chain and that the first app can be garbage-collected.

Stopgaps if needed before that lands: split the unit job across 2–3 runners, or reduce the largest parametrized case in test_validation_extraction_is_bounded.