Repository navigation
Conversation
…in streaming SQL) Automated fix and test assertions. Signed-off-by: Ollie <olliemcgrath@icloud.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves #1659
/claim #1659
Solution for [github/issues_1659]: [getdozer/dozer] Support for
INclause in streaming SQLINclause in streaming SQL #1659Technical Solution: Support for
INclause in streaming SQLThis document outlines the technical solution for adding
INclause support to Dozer's streaming SQL, covering both static list lookups and subquery lookups.Root Cause Analysis
Currently, Dozer's SQL parser and execution engine lack specific operators or logic to handle the
INclause. The existingWHEREclause processing primarily focuses on binary comparisons (=,>,<, etc.) and logical operators (AND,OR,NOT). TheINoperator, especially with a subquery, introduces a new challenge: managing a potentially dynamic set of values for comparison, which requires stateful processing similar to aJOIN.Architecture Design
The solution will involve modifications across several components of Dozer:
SQL Parser (
dozer-sqlcrate):INexpressions, differentiating between static lists and subqueries.INexpressions in the Abstract Syntax Tree (AST).Logical Plan Builder (
dozer-sqlcrate):INAST nodes into appropriate logical plan operators.InListlogical operator or extend an existing filter operator to handle a set of values.IN (SELECT ...)into aJOINoperation, specifically a semi-join or inner join followed by distinct projection, as suggested in the issue description. This will leverage the existing join infrastructure.Execution Plan Builder (
dozer-core/dozer-sqlcrates):InFilterphysical operator that efficiently checks membership against a pre-defined set of values. This operator will adhere to the specifiedWHEREclause behavior forINSERT,UPDATE, andDELETEoperations.JOINin the logical plan will naturally lead to using existingJoinphysical operators.Data Structures and Utilities:
HashSet) within theInFilteroperator.Detailed Design for Static
INListsqlparserlibrary, which Dozer uses, already supports parsingINexpressions. We need to ensure our AST conversion correctly captures the column and the list of values.LogicalOperator::Filtervariant or a dedicatedLogicalOperator::InListwill be introduced. This operator will hold the expression for the column and the list ofScalarValues.InFilterOperatorwill be implemented.HashSetfor O(1) average lookup).Operation(INSERT, UPDATE, DELETE) arrives:INcondition check.Operations downstream according to the specifiedWHEREclause behavior:INSERT: Propagate if new value matches.UPDATE:prevmatches,newmatches ->UPDATEprevdoesn't match,newmatches ->INSERTprevmatches,newdoesn't match ->DELETEprevdoesn't match,newdoesn't match -> No actionDELETE: Propagate if previous value matches.Detailed Design for
INSubquerysqlparserlibrary also supportsIN (SELECT ...). The AST will represent this as anInSubqueryexpression.InSubquerywill be transformed into aSemiJoinor anInnerJoinfollowed by aDistinctprojection.outer_column = subquery_column.SemiJoinis ideal as it only returns rows from the left side that have a match in the right side, without duplicating rows from the left side. IfSemiJoinis not directly available or easily implementable, anInnerJoinfollowed by aDistinctoperator on the outer query's columns would achieve a similar effect.Joinoperator in Dozer already handles the state management and propagation of changes from both sides, which is exactly what's needed for a dynamicINlist.Implementation Plan
Identify relevant files:
dozer-sql/src/sql_analyzer/mod.rs: For SQL parsing and AST conversion.dozer-sql/src/sql_analyzer/expressions.rs: For AST expression definitions.dozer-sql/src/sql_analyzer/planner.rs: For logical plan generation.dozer-sql/src/pipeline/builder.rs: For physical plan generation.dozer-core/src/processor.rs: For defining new processor traits/implementations.dozer-core/src/operators/filter.rs: Potentially extend or create a new filter operator.dozer-core/src/operators/join.rs: Leverage existing join logic for subqueries.dozer-sql/src/tests/: Add new tests.Step 1: Static
INList Supportdozer-sql/src/sql_analyzer/expressions.rsto add anInListvariant toExpression.dozer-sql/src/sql_analyzer/mod.rsto parseIN (value1, value2, ...)into the newInListexpression.dozer-sql/src/sql_analyzer/planner.rsto translateInListinto aLogicalOperator::Filterwith a newFilterExpression::InList.InFilterOperator, indozer-core/src/operators/filter.rs(or a new file). This operator will handle the stateful filtering logic as described above.InFilterOperatorinto the physical plan builder indozer-sql/src/pipeline/builder.rs.INlists.Step 2:
INSubquery Supportdozer-sql/src/sql_analyzer/expressions.rsto add anInSubqueryvariant toExpression, holding the subquery'sQueryAST.dozer-sql/src/sql_analyzer/mod.rsto parseIN (SELECT ...)into the newInSubqueryexpression.dozer-sql/src/sql_analyzer/planner.rsto transformInSubqueryinto aLogicalOperator::Join. This will involve:JoinType::Semi(orInner+Distinct) join between the outer query's source and the subquery's result, with the appropriate join condition.Joinphysical operator can handle the generated logical plan.INsubqueries.Testing Verification Steps
A comprehensive test suite will be developed to ensure correctness.
Unit Tests
INlist formats:IN (1, 2, 3),IN ('a', 'b'),IN (NULL, 1),IN (1 + 1, 3).INsubquery parsing:IN (SELECT id FROM other_table),IN (SELECT col FROM another_table WHERE condition).NOT IN(ifNOTis supported for other operators, it should also apply here).INlists are translated into aFilteroperator with the correctInListexpression.INsubqueries are translated into aJoinoperator (specifically, aSemiJoinorInnerJoin+Distinct) with the correct join condition and sources.InFilterOperatorLogic (for static lists):INSERToperations:INlist ->INSERTpropagated.INlist -> No action.UPDATEoperations:prevmatches,newmatches ->UPDATEpropagated.prevdoesn't match,newmatches ->INSERTpropagated.prevmatches,newdoesn't match ->DELETEpropagated.prevdoesn't match,newdoesn't match -> No action.DELETEoperations:INlist ->DELETEpropagated.INlist -> No action.Integration Tests (End-to-End)
INList:WHERE column IN (value1, value2, ...).SELECT * FROM users WHERE id IN (1, 3, 5);INSERT {id: 1, name: "Alice"}->INSERT {id: 1, name: "Alice"}INSERT {id: 2, name: "Bob"}-> No outputUPDATE {id: 1, name: "Alice"} -> {id: 1, name: "Alicia"}->UPDATE {id: 1, name: "Alice"} -> {id: 1, name: "Alicia"}UPDATE {id: 2, name: "Bob"} -> {id: 2, name: "Bobby"}-> No outputUPDATE {id: 2, name: "Bobby"} -> {id: 3, name: "Charlie"}->INSERT {id: 3, name: "Charlie"}UPDATE {id: 1, name: "Alicia"} -> {id: 2, name: "Bob"}->DELETE {id: 1, name: "Alicia"}DELETE {id: 3, name: "Charlie"}->DELETE {id: 3, name: "Charlie"}INSubquery:main_tableandlookup_table.WHERE column IN (SELECT sub_column FROM lookup_table WHERE ...).main_tableandlookup_tableand verify the output.SELECT * FROM orders WHERE customer_id IN (SELECT id FROM premium_customers WHERE region = 'EU');INSERT {id: 1, name: "Alice", region: "EU"}intopremium_customers.orderswithcustomer_id = 1should now be emitted asINSERTs.INSERT {order_id: 101, customer_id: 1, amount: 100}intoorders.customer_id = 1is inpremium_customers(andregion = 'EU'),INSERT {order_id: 101, customer_id: 1, amount: 100}is emitted.DELETE {id: 1, name: "Alice", region: "EU"}frompremium_customers.orderswithcustomer_id = 1should now be emitted asDELETEs.UPDATE {id: 1, name: "Alice", region: "EU"} -> {id: 1, name: "Alice", region: "US"}inpremium_customers.orderswithcustomer_id = 1should now be emitted asDELETEs (becauseregionno longer matches).This structured approach ensures that both forms of the
INclause are correctly parsed, planned, and executed, adhering to the specified streaming semantics.Code Implementation
The following code implements the proposed changes.
Key Changes:
dozer-sql/src/sql_analyzer/expressions.rs:InListandInSubqueryvariants to theExpressionenum.dozer-sql/src/sql_analyzer/mod.rs:parse_expressionto handleInListandInSubqueryusingsqlparser'sExpr::InListandExpr::InSubquery.dozer-sql/src/sql_analyzer/planner.rs:plan_expressionto translateInListinto aFilterExpression::InList.plan_expressionto translateInSubqueryinto aLogicalOperator::Join(specifically, aSemiJoin). This involves planning the subquery and then constructing the join.dozer-sql/src/pipeline/builder.rs:build_filter_operatorto handleFilterExpression::InListby creating anInFilterOperator.dozer-core/src/operators/filter.rs:FilterExpression::InListto represent theINcondition in the logical plan.InFilterOperatorwhich is a stateful operator that handlesINSERT,UPDATE,DELETEoperations based on theINlist membership. It uses aHashSetfor efficient lookups.dozer-core/src/operators/join.rs:JoinType::Semito support semi-joins, which are ideal forINsubqueries.JoinOperatorto handleSemiJoinlogic: it only propagates records from the left side that have a match on the right side, without duplicating or adding columns from the right side.dozer-sql/src/tests/in_clause.rs:INlists andINsubqueries, covering various scenarios includingINSERT,UPDATE,DELETEoperations and changes in the subquery's lookup table.File: dozer-sql/src/sql_analyzer/expressions.rs