Skip to content

Add support for white-box explainers to alibi-explain runtime#1279

Merged
adriangonz merged 20 commits into
SeldonIO:masterfrom
ascillitoe:feature/white_box_explainers
Jul 6, 2023
Merged

Add support for white-box explainers to alibi-explain runtime#1279
adriangonz merged 20 commits into
SeldonIO:masterfrom
ascillitoe:feature/white_box_explainers

Conversation

@ascillitoe

@ascillitoe ascillitoe commented Jul 3, 2023

Copy link
Copy Markdown

Summary

This PR adds support for the sklearn API white-box explainers (TreeShap, TreePartialDependence, PartialDependenceVariance) to the AlibiExplainRuntime.

Implementation details

  • New runtime: Support is implemented via the new AlibiExplainSKLearnAPIRuntime subclass. For the remaining white-box explainers two further subclassed runtimes will be required (one being the IntegratedGradientsWrapper). See the notion doc for more details.

  • Tests: Some explainer's have various "nitty-gritty" details wrt to serving, for example the target kwarg for IntegratedGradients, calling fit with no data for TreeShap, and black-box/white-box mode of PartialDependenceVariance being determined by type(predictor). Therefore, the PR is proposing to perform an E2E test (test_end_2_end) on every explainer (with relevant parametrisation such as testing uri vs init_param's based loading). This would provide more confidence that all explainers can actually be served. However, it will increase the number of tests being run somewhat, so needs discussion...

TODO

  • Linting etc
  • Docs (leave until all white box explainers supported...)
  • Add tests to check loading of all supported tree models e.g. lightgbm models etc - done in afae1e6

Comment thread runtimes/alibi-explain/mlserver_alibi_explain/explainers/white_box_runtime.py Outdated
Comment thread runtimes/alibi-explain/pyproject.toml
Comment thread runtimes/alibi-explain/tests/helpers/sk_model.py Outdated
Comment thread runtimes/alibi-explain/tests/test_white_box_sklearn_cases.py

@adriangonz adriangonz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a great PR @ascillitoe !!

I love the test thoroughness - the more tests the better IMO (as long as they are not massively slow!). Added a few comments here-and-there, but they are mainly questions and suggestions - nothing biggie.

Comment thread runtimes/alibi-explain/mlserver_alibi_explain/explainers/sklearn_api_runtime.py Outdated
Comment thread runtimes/alibi-explain/mlserver_alibi_explain/explainers/sklearn_api_runtime.py Outdated
Comment thread runtimes/alibi-explain/tests/conftest.py Outdated
Comment thread runtimes/alibi-explain/tests/helpers/sk_model.py Outdated
Comment thread runtimes/alibi-explain/tests/helpers/sk_model.py Outdated
Comment thread runtimes/alibi-explain/tests/test_white_box_sklearn.py Outdated
Comment thread runtimes/alibi-explain/tests/test_white_box_sklearn_cases.py

@adriangonz adriangonz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice one @ascillitoe! Thanks for making those changes.

Once the linter passes, this looks good to go from my side! Conscious it's still marked as a draft though, so happy to re-review once you think it's ready.

@ascillitoe

Copy link
Copy Markdown
Author

Thanks @adriangonz, I'll make the linter changes now and turn off draft once done!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants