Repository navigation
fix(pools): take the pool locks of a create from a template before its transaction - #11591
ajtmccarty wants to merge 4 commits into
Conversation
…s transaction A create from a template asked for a resource pool's lock inside its transaction, after it had written a number, which locks the number pool vertex in Neo4j until the commit. Another allocation that held that lock and waited for the vertex then waited on the create, and the create waited on it. Neither wait has a time limit. create_node() now reads the template's components before it takes the mutation's locks and adds the pool locks of the object and of its components to the lock names: the division locks of number pools the templates name, those of NumberPool attributes, and the locks of IP address and IP prefix pools. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Merging this PR will not alter performance1 warning
Comparing Footnotes |
create_node() built a preview with one generated id and the node with another, so a lock name that holds the id of the new node named an object that never exists. The pool locks of a template's components therefore left out the divisions that hold a new object, and took them inside the transaction. create_node() now generates the id before the preview, unless the payload sets one, and each TemplateComponent holds the id of the component it describes. A generated id is not checked against the database. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Node.new() takes field values as keyword arguments, and the IP address and IP prefix pools spread the data of their GetResource mutations into it. As a keyword argument of Node.new(), generated_id could therefore be set by that data, which gave the new node an id that was not checked against the database. generated_id is now a parameter of new_from_fields() only, where the fields are a dict. Template components are built through new_from_fields() to pass the id their component holds. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
There was a problem hiding this comment.
just refactoring to make functions accessible in backend/infrahub/core/node/create.py
| subtemplate: CoreObjectTemplate | ||
| schema: NonGenericSchemaTypes | ||
| components: list[TemplateComponent] = field(default_factory=list) | ||
| object_id: str = field(default_factory=lambda: str(UUIDT())) |
There was a problem hiding this comment.
New field. The id of the component is decided when the subtemplate tree is read, so the component preview and the created component share it.
| ) | ||
|
|
||
|
|
||
| async def _new_component_node( |
There was a problem hiding this comment.
New helper. Both the component preview and create_component() build their node here, through new_from_fields(), so the component's id is passed as a parameter and not as a field.
| return any(attribute.kind == "NumberPool" for attribute in schema.attributes) | ||
|
|
||
|
|
||
| async def _template_pools( |
There was a problem hiding this comment.
New helper. Lists the pool a template names for each field through its <field>_from_resource_pool relationships. The peers are already in memory, so it runs no query.
| return pools | ||
|
|
||
|
|
||
| async def _template_pool_lock_names( |
There was a problem hiding this comment.
New helper. Turns the pools a template names into lock names: resource_pool.<pool id> for an IP pool, .all for a number pool without a scope, and the division key for one with a scope.
| lock_names |= await _template_pool_lock_names( | ||
| node=preview.node, pools=preview.pools, pool_applier=pool_applier, user_id=user_id | ||
| ) | ||
| lock_names |= get_pool_lock_names(node=preview.node) |
There was a problem hiding this comment.
This picks up the NumberPool attributes of the component, whose from_pool was stored when the preview was built.
|
|
||
| if object_template: | ||
| await handle_template_relationships( | ||
| await create_components( |
There was a problem hiding this comment.
handle_template_relationships() only read the components and then called this. The read moved to create_node(), so the call is made directly.
| schema_branch = db.schema.get_schema_branch(name=branch.name) | ||
| components = _build_node_write_components(db=db, branch=branch, schema_branch=schema_branch) | ||
| # The preview and the node created under the lock share one uuid, so the lock names hold the uuid of the node. | ||
| generated_id = None if data.get("id") else str(UUIDT()) |
There was a problem hiding this comment.
The preview and the node created under the lock now share one id. An id generated here is not checked against the database; an id from the payload still is.
| object_template = preview_obj._object_template | ||
| template_components: list[TemplateComponent] = [] | ||
| if object_template is not None: | ||
| template_components = await read_template_components( |
There was a problem hiding this comment.
Moved from inside the transaction to before the locks, because the lock names depend on the components.
| pool_applier=components.pool_applier, | ||
| user_id=user_id, | ||
| ) | ||
| lock_names = sorted({*lock_names, *component_lock_names}) |
There was a problem hiding this comment.
Sorted again after the merge, so InfrahubMultiLock still takes every lock in one global order.
There was a problem hiding this comment.
All reported issues were addressed across 12 files
Shadow auto-approve: would not auto-approve because issues were found.
View guided diff | Re-trigger cubic
…e components Only a component that draws from a pool, or holds one that does, gets a preview before the locks. Other components are still validated after the object's own checks. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
There was a problem hiding this comment.
0 issues found across 1 file (changes from recent commits).
Confidence score: 5/5
- Automated review surfaced no issues in the provided summaries.
- No files require special attention.
Shadow auto-approve: would auto-approve. Template creation now acquires all required pool locks before opening a transaction, preventing deadlocks. Regression tests verify the lock order.
View guided diff | Re-trigger cubic
Human-written summary
This deadlock bug was uncovered during performance testing for the latest Number Pool and IP Pool-related changes on
stable. If multiple requests to create objects using templates came in concurrently and used the same Number Pools, then the requests could deadlock and time out. The fix is to get all the locks (Number Pools included) that are required for a given create before beginning the actual create.The code changes are mostly refactoring to get all the Number Pool locks (schema or user, scoped or not) for a given create with templates. Ideally, most of this logic would move to its own component and not be in
create.py, but this is a problem for a different PRtried a new thing where Claude leaves (maybe too many) inline comments to help the reviewer (human or not). not sure if this is good or bad
Why
A team creates objects from a template whose component templates take numbers from a number pool. At the same time, another request takes a number from the same pool, in the division of one of the components. Both requests stop responding until the client gives up.
The two requests took the same two locks in opposite order, and neither wait has a time limit:
Goal: a create from a template takes every pool lock it needs before it opens its transaction.
Non-goals: moving the template code out of
create.py, and the cases listed under "Not covered" below.Closes: no GitHub issue is filed for this bug yet. Related: IFC-3547 describes a wait cycle that also needs a waiting range change; this pull request has no test for it.
What changed
Behavioral changes:
create_node()reads the template's components before it takes the mutation's locks, and adds these locks to the mutation's lock names:NumberPoolattribute of a componentcreate_node()builds to compute lock names now has the id of the node it creates.create_node()generates the id before it builds the preview, unless the payload setsid. Before, the two had different generated ids, sorelationship_count.<identifier>.<node id>locks named an object that never existed. This applies to every create, not only creates from a template. An id thatcreate_node()generates is not checked against the database; an id the payload sets still is.Implementation notes:
create_node()builds an unsaved preview of each component that draws from a pool, and of each component above one. A template whose components draw from no pool builds no preview.TemplateComponentholds the id of the component it describes, generated when the component templates are read, so the preview and the created component share it.Node.new_from_fields()takes the generated id as agenerated_idparameter, used when the fields hold noid.Node.new()does not take it._new_component_node(), which callsnew_from_fields()to pass the component's id. Before, they were built withNode.new().handle_template_relationships()is removed;_do_create_node()callscreate_components()directly.lock_utils.py:get_pool_lock_names()andip_pool_lock_name()are extracted fromget_lock_names_on_object_mutation(), which returns the same names as before for the same node.Not covered:
NumberPoolattribute. The IP node is created inside the transaction, where it takes that number pool's lock. A direct create withfrom_poolon a relationship does the same.How to review
backend/infrahub/core/node/create.py:create_node(), thenget_template_pool_lock_names()and its helpers.backend/infrahub/core/node/__init__.py: thegenerated_idparameter ofnew_from_fields(). It is deliberately not a parameter ofNode.new(), which takes field values as keyword arguments: the IP poolGetResourcemutations spread theirdatainto it, so agenerated_idkey there would set an id that is not checked.test_get_resource_data_cannot_set_the_id_of_the_address_without_a_checkcovers this.backend/tests/component/core/node/test_template_pool_lock_order.py: the test matrix.Parts that deserve extra scrutiny:
How to test
test_template_pool_lock_order.pyholds 16 tests. 13 parametrized cases cover the object, a component and a nested component against template-named number pools (with and without a scope),NumberPoolattributes, IP address pools and IP prefix pools. Each case holds one pool lock from another task and asserts that the create waits for it with no transaction open. Three more tests cover a payload that sets theid, a division that holds the object created, and the shared id.test_a_template_create_and_a_create_in_the_division_of_its_second_component_both_completeruns the two requests against each other. Without the fix, both are still blocked after 30 seconds.test_get_resource_data_cannot_set_the_id_of_the_address_without_a_checkintest_ipaddress_pool.pyasserts that agenerated_idkey in thedataof an IP address pool allocation does not set the id of the new address.Not tested: the scenario on a full stack with Redis and several API server workers, the wait cycle for IP pools (the tests assert only the lock order), and releases before 1.12.0.
Impact & rollout
Checklist
uv run towncrier create ...)🤖 Generated with Claude Code