Skip to content

add conformance test suite#291

Open
anuraaga wants to merge 3 commits into
cel-rust:masterfrom
anuraaga:conformance-tests-runner
Open

add conformance test suite#291
anuraaga wants to merge 3 commits into
cel-rust:masterfrom
anuraaga:conformance-tests-runner

Conversation

@anuraaga

@anuraaga anuraaga commented May 1, 2026

Copy link
Copy Markdown

This continues from #251 to hopefully get into a shape that can be merged, and then iterated to add more features / fixes to pass more of the conformance test.

I have redesigned things quite a bit to address feedback and simplify (IMO) things

  • Separate a generate.rs script for doing all preprocessing of cel-spec. build.rs only checks for version mismatch and fails the build if there is one
    • To address point of not regenerating on every build
  • Avoid the complexity of git submodules by fetching the zip archive during generation.
  • Use protox for proto compilation and prost-reflect for textproto parsing to avoid protoc dependency
  • Use standard cargo test as much as possible by codegenning test files themselves from the textproto. This allows test reports to show just like any cargo unit tests, no need for us to implement reporting logic. And we don't depend on cel-spec at all during test execution, it's only needed during the generation step
  • A system using a text file of ignored tests to reference during generation to mark failing tests as #[ignored]. In the future, let's make this file empty!

AI disclaimer: I used AI for autocomplete and also to generate much logic that is mostly just string crushing. I read every single line and cleaned up / rewrote significant parts - for better or worse, I stand by each line of code.

Co-authored-by: flaque <evan@sfcompute.com>
Signed-off-by: Anuraag Agrawal <anuraaga@gmail.com>
@anuraaga anuraaga force-pushed the conformance-tests-runner branch from 5b1bcdf to 3b8df7d Compare May 1, 2026 09:43
Signed-off-by: Anuraag Agrawal <anuraaga@gmail.com>
@anuraaga anuraaga force-pushed the conformance-tests-runner branch from df1728d to ad569f5 Compare May 1, 2026 10:03
@alexsnaps

alexsnaps commented May 1, 2026

Copy link
Copy Markdown
Member

Hey @anuraaga,
Thanks for taking on that work 🙏
From a quick look, what would you think if we'd "#[should_panic(expected = "This is known limitation")]" instead of #[ignore]'ing the known tests that fail? That way, when one actually passes, it would be called out explicitly. Wdyt?

Signed-off-by: Anuraag Agrawal <anuraaga@gmail.com>
@anuraaga

anuraaga commented May 1, 2026

Copy link
Copy Markdown
Author

Great idea @alexsnaps updated to that. I left out an error expectation since the message doesn't matter for the test, just whether it fails or not.

@alexsnaps

Copy link
Copy Markdown
Member

Sorry, somewhat slow following up on this, but haven't lost track of this PR 🙏

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants