Skip to content

fix(pools): take the pool locks of a create from a template before its transaction - #11591

Open
ajtmccarty wants to merge 4 commits into
stablefrom
ai-bug-pipeline-template-pool-division-lock-cycle
Open

ajtmccarty wants to merge 4 commits into
stablefrom
ai-bug-pipeline-template-pool-division-lock-cycle

Conversation

@ajtmccarty

@ajtmccarty ajtmccarty commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Human-written summary

This deadlock bug was uncovered during performance testing for the latest Number Pool and IP Pool-related changes on stable. If multiple requests to create objects using templates came in concurrently and used the same Number Pools, then the requests could deadlock and time out. The fix is to get all the locks (Number Pools included) that are required for a given create before beginning the actual create.

The code changes are mostly refactoring to get all the Number Pool locks (schema or user, scoped or not) for a given create with templates. Ideally, most of this logic would move to its own component and not be in create.py, but this is a problem for a different PR

tried a new thing where Claude leaves (maybe too many) inline comments to help the reviewer (human or not). not sure if this is good or bad

Why

A team creates objects from a template whose component templates take numbers from a number pool. At the same time, another request takes a number from the same pool, in the division of one of the components. Both requests stop responding until the client gives up.

The two requests took the same two locks in opposite order, and neither wait has a time limit:

Step Create from the template Direct create in the component's division
1 Takes the lock of its own division, before its transaction Takes the lock of its division, before its transaction
2 Saves a node with a number from the pool, which locks the number pool vertex in Neo4j until the commit Opens its transaction and picks a number
3 Next component: waits for the lock of the component's division Node create query: waits for the Neo4j lock on the pool vertex

Goal: a create from a template takes every pool lock it needs before it opens its transaction.

Non-goals: moving the template code out of create.py, and the cases listed under "Not covered" below.

Closes: no GitHub issue is filed for this bug yet. Related: IFC-3547 describes a wait cycle that also needs a waiting range change; this pull request has no test for it.

What changed

Behavioral changes:

  • create_node() reads the template's components before it takes the mutation's locks, and adds these locks to the mutation's lock names:
    • the division lock of each number pool that the template or a component template names, with or without an allocation scope
    • the division lock of each NumberPool attribute of a component
    • the lock of each IP address pool and IP prefix pool that the template or a component template names
  • The preview that create_node() builds to compute lock names now has the id of the node it creates. create_node() generates the id before it builds the preview, unless the payload sets id. Before, the two had different generated ids, so relationship_count.<identifier>.<node id> locks named an object that never existed. This applies to every create, not only creates from a template. An id that create_node() generates is not checked against the database; an id the payload sets still is.
  • A validation error of a component that draws from a pool, or holds one that does, is now raised before the object's own checks. Other components are validated after them, as before.

Implementation notes:

  • To read a component's division, create_node() builds an unsaved preview of each component that draws from a pool, and of each component above one. A template whose components draw from no pool builds no preview.
  • Each TemplateComponent holds the id of the component it describes, generated when the component templates are read, so the preview and the created component share it.
  • Node.new_from_fields() takes the generated id as a generated_id parameter, used when the fields hold no id. Node.new() does not take it.
  • Components are built through a new helper, _new_component_node(), which calls new_from_fields() to pass the component's id. Before, they were built with Node.new().
  • handle_template_relationships() is removed; _do_create_node() calls create_components() directly.
  • lock_utils.py: get_pool_lock_names() and ip_pool_lock_name() are extracted from get_lock_names_on_object_mutation(), which returns the same names as before for the same node.

Not covered:

  • A pool fills a field that another pool's scope names. The create takes the lock of an empty division before the transaction and the lock of the real division inside it.
  • An IP pool whose address or prefix kind has a NumberPool attribute. The IP node is created inside the transaction, where it takes that number pool's lock. A direct create with from_pool on a relationship does the same.
  • A caller that passes a database with a transaction already open.

How to review

  1. backend/infrahub/core/node/create.py: create_node(), then get_template_pool_lock_names() and its helpers.
  2. backend/infrahub/core/node/__init__.py: the generated_id parameter of new_from_fields(). It is deliberately not a parameter of Node.new(), which takes field values as keyword arguments: the IP pool GetResource mutations spread their data into it, so a generated_id key there would set an id that is not checked. test_get_resource_data_cannot_set_the_id_of_the_address_without_a_check covers this.
  3. backend/tests/component/core/node/test_template_pool_lock_order.py: the test matrix.

Parts that deserve extra scrutiny:

  • The component templates are now read before the locks and outside the transaction, as the template already is. A component template that another request changes while the create waits for a lock is applied as it was read.
  • The preview repeats work the create does again for each pooled component.

How to test

uv run pytest backend/tests/component/core/node/test_template_pool_lock_order.py
uv run pytest backend/tests/component/core/resource_manager/test_number_pool_scope_allocation.py
uv run pytest backend/tests/component/core/resource_manager/test_ipaddress_pool.py
  • test_template_pool_lock_order.py holds 16 tests. 13 parametrized cases cover the object, a component and a nested component against template-named number pools (with and without a scope), NumberPool attributes, IP address pools and IP prefix pools. Each case holds one pool lock from another task and asserts that the create waits for it with no transaction open. Three more tests cover a payload that sets the id, a division that holds the object created, and the shared id.
  • test_a_template_create_and_a_create_in_the_division_of_its_second_component_both_complete runs the two requests against each other. Without the fix, both are still blocked after 30 seconds.
  • test_get_resource_data_cannot_set_the_id_of_the_address_without_a_check in test_ipaddress_pool.py asserts that a generated_id key in the data of an IP address pool allocation does not set the id of the new address.
  • On the code before this pull request, 11 of the first 14 tests fail. The three that pass pin behavior that was already correct.
  • Local run: 2,102 tests passed in the node, resource manager, template, create and upsert mutation, and unit core suites.

Not tested: the scenario on a full stack with Redis and several API server workers, the wait cycle for IP pools (the tests assert only the lock order), and releases before 1.12.0.

Impact & rollout

  • Backward compatibility: no API or schema change.
  • Performance: a component in a scoped number pool costs one more read of the pool before the locks. In a local measurement taken before the second commit, a template with five components in one scoped pool ran ten more queries, and a template that names no pool ran none.
  • Deployment notes: safe to deploy.

Checklist

  • Tests added/updated
  • Changelog entry added (uv run towncrier create ...)
  • External docs updated (if user-facing or ops-facing change): not applicable
  • Internal .md docs updated (internal knowledge and AI code tools knowledge)
  • I have reviewed AI generated content

🤖 Generated with Claude Code

…s transaction

A create from a template asked for a resource pool's lock inside its
transaction, after it had written a number, which locks the number pool
vertex in Neo4j until the commit. Another allocation that held that lock
and waited for the vertex then waited on the create, and the create
waited on it. Neither wait has a time limit.

create_node() now reads the template's components before it takes the
mutation's locks and adds the pool locks of the object and of its
components to the lock names: the division locks of number pools the
templates name, those of NumberPool attributes, and the locks of IP
address and IP prefix pools.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@ajtmccarty ajtmccarty added type/bug Something isn't working as expected group/backend API server, GraphQL, task worker (Prefect), database, Python code category/pools Resource pools: NumberPool, prefix pools, IPAM allocation labels Nov 14, 2026
@codspeed

codspeed Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

1 warning

⚠️ 1 benchmark is dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

✅ 13 untouched benchmarks


Comparing ai-bug-pipeline-template-pool-division-lock-cycle (d5b9319) with stable (2488fa2)1

Open in CodSpeed

Footnotes

  1. No successful run was found on stable (aaebc8e) during the generation of this report, so 2488fa2 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

ajtmccarty and others added 2 commits November 14, 2026 01:35
create_node() built a preview with one generated id and the node with
another, so a lock name that holds the id of the new node named an
object that never exists. The pool locks of a template's components
therefore left out the divisions that hold a new object, and took them
inside the transaction.

create_node() now generates the id before the preview, unless the
payload sets one, and each TemplateComponent holds the id of the
component it describes. A generated id is not checked against the
database.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Node.new() takes field values as keyword arguments, and the IP address
and IP prefix pools spread the data of their GetResource mutations into
it. As a keyword argument of Node.new(), generated_id could therefore be
set by that data, which gave the new node an id that was not checked
against the database.

generated_id is now a parameter of new_from_fields() only, where the
fields are a dict. Template components are built through
new_from_fields() to pass the id their component holds.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just refactoring to make functions accessible in backend/infrahub/core/node/create.py

subtemplate: CoreObjectTemplate
schema: NonGenericSchemaTypes
components: list[TemplateComponent] = field(default_factory=list)
object_id: str = field(default_factory=lambda: str(UUIDT()))

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New field. The id of the component is decided when the subtemplate tree is read, so the component preview and the created component share it.

)


async def _new_component_node(

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New helper. Both the component preview and create_component() build their node here, through new_from_fields(), so the component's id is passed as a parameter and not as a field.

return any(attribute.kind == "NumberPool" for attribute in schema.attributes)


async def _template_pools(

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New helper. Lists the pool a template names for each field through its <field>_from_resource_pool relationships. The peers are already in memory, so it runs no query.

return pools


async def _template_pool_lock_names(

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New helper. Turns the pools a template names into lock names: resource_pool.<pool id> for an IP pool, .all for a number pool without a scope, and the division key for one with a scope.

lock_names |= await _template_pool_lock_names(
node=preview.node, pools=preview.pools, pool_applier=pool_applier, user_id=user_id
)
lock_names |= get_pool_lock_names(node=preview.node)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This picks up the NumberPool attributes of the component, whose from_pool was stored when the preview was built.


if object_template:
await handle_template_relationships(
await create_components(

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

handle_template_relationships() only read the components and then called this. The read moved to create_node(), so the call is made directly.

schema_branch = db.schema.get_schema_branch(name=branch.name)
components = _build_node_write_components(db=db, branch=branch, schema_branch=schema_branch)
# The preview and the node created under the lock share one uuid, so the lock names hold the uuid of the node.
generated_id = None if data.get("id") else str(UUIDT())

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The preview and the node created under the lock now share one id. An id generated here is not checked against the database; an id from the payload still is.

object_template = preview_obj._object_template
template_components: list[TemplateComponent] = []
if object_template is not None:
template_components = await read_template_components(

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved from inside the transaction to before the locks, because the lock names depend on the components.

pool_applier=components.pool_applier,
user_id=user_id,
)
lock_names = sorted({*lock_names, *component_lock_names})

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorted again after the merge, so InfrahubMultiLock still takes every lock in one global order.

@ajtmccarty
ajtmccarty marked this pull request as ready for review November 14, 2026 02:08
@ajtmccarty
ajtmccarty requested a review from a team as a code owner November 14, 2026 02:08

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 12 files

Shadow auto-approve: would not auto-approve because issues were found.

View guided diff | Re-trigger cubic

Comment thread dev/knowledge/backend/mutations.md Resolved
…e components

Only a component that draws from a pool, or holds one that does, gets a
preview before the locks. Other components are still validated after the
object's own checks.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 1 file (changes from recent commits).

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Shadow auto-approve: would auto-approve. Template creation now acquires all required pool locks before opening a transaction, preventing deadlocks. Regression tests verify the lock order.

View guided diff | Re-trigger cubic

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

category/pools Resource pools: NumberPool, prefix pools, IPAM allocation group/backend API server, GraphQL, task worker (Prefect), database, Python code type/bug Something isn't working as expected

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant