ask about projects #41

Question 1: iFixAi acts as an open-source diagnostic for AI misalignment, running 32 tests across five specific risk categories (fabrication, manipulation, deception, unpredictability, and opacity). From an evaluation engineering perspective, how does the codebase programmatically define the test logic for qualitative risk dimensions like "evaluation-awareness sandbagging" (B10) or "covert side tasks" (B14) to ensure they produce objective, empirical evidence rather than subjective observations?

Question 2: The framework supports three distinct evaluation methods: structural, judge, and atomic claims. When executing a "judge" type inspection, iFixAi prevents self-scoring by default in Standard mode, requiring a separate secondary provider key (e.g., Anthropic judging OpenAI). How does the system dynamically structuralize the prompt payload and parse the judge's response to guarantee that variations in the judge's formatting do not cause a parser failure or pollute the final scoring schema?

Question 3: For high-assurance validation, iFixAi introduces Full mode, which runs two or more independent LLM judges across different providers and applies a majority voting mechanism. How does the execution engine handle a split decision or a tie vote among judges, and how is the disagreement recorded inside the content-addressed manifest.json file to maintain a clear audit trail?

Question 4: The diagnostic suite guarantees deterministic reproducibility by exporting a content-addressed manifest that captures all inputs, seeds, rubric hashes, and test corpus versions. Given that live LLM API endpoints are inherently non-deterministic due to server-side sampling drift and model upgrades, how does iFixAi's replay engine bypass live inference to achieve bit-identical score card generation during a regression test rerun?

Question 5: A core feature of the platform is its "Wiring Governance" interface, which allows users to supply an external GovernanceFixture YAML that automatically wraps the target provider via a GovernanceMixin. How does this mixin dynamically intercept the System Under Test's (SUT) outbound API calls and tool execution at runtime to verify compliance against custom permission scopes and authorization policy bundles?

Question 6: iFixAi is explicitly designed to integrate natively into CI/CD pipelines as a regression gate, utilizing specific exit codes (e.g., exit code 2 for a score drop below the --min-score threshold). Since running a full 32-test suite against heavy models can incur high API costs and network latency, how does the run_strategic() async Python API surface optimize test selection and parallel concurrency bounds to provide a rapid, cost-effective signal before code deployment?