Skip to content

Add support for online drift detectors#1108

Merged
adriangonz merged 10 commits into
SeldonIO:masterfrom
ascillitoe:feature/online_drift_detectors
Apr 20, 2023
Merged

Add support for online drift detectors#1108
adriangonz merged 10 commits into
SeldonIO:masterfrom
ascillitoe:feature/online_drift_detectors

Conversation

@ascillitoe

@ascillitoe ascillitoe commented Apr 19, 2023

Copy link
Copy Markdown

This PR adds support for online drift detectors to Alibi Detect runtime.

Overview

This involves two primary changes:

  1. Online drift detectors are run on one instance at a time, with the time-step self.t updated each time. If a request contains a batch of instances, the payload must be unpacked and passed to the detector one instance at a time.
  2. Online drift detectors have state, which must be periodically saved for checkpointing. State is saved every state_save_freq time-steps, where state_save_freq is a new configurable parameter added to AlibiDetectSettings.

Limitations

Parallel inference is NOT currently supported for online detectors (i.e. parallel_workers=0 is required). This would require careful thought wrt to how state is managed/shared across parallel workers, in addition to considering the implications of distributing non-iid samples between workers.

TODO's

Comment thread runtimes/alibi-detect/mlserver_alibi_detect/runtime.py
Comment thread runtimes/alibi-detect/mlserver_alibi_detect/runtime.py
# If batch is configured, wrap X in a list so that it is not unpacked
X = np.array(input_data)
if self._ad_settings.batch_size:
X = [X]

@ascillitoe ascillitoe Apr 19, 2023

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wrap X in a list (of length 1) so that we don't unpack into single instances when running in batch mode (i.e. offline drift detectors or outlier detectors). pred is then returned as a list of length 1.

This slightly circuitous logic is intended to avoid having two duplicate try-except blocks for the two (offline vs online) use cases...

Comment thread runtimes/alibi-detect/mlserver_alibi_detect/runtime.py Outdated
Comment thread runtimes/alibi-detect/mlserver_alibi_detect/runtime.py Outdated
Comment thread runtimes/alibi-detect/tests/conftest.py Outdated

@adriangonz adriangonz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great @ascillitoe! 🚀

I've added a minor comment around the state save logic and check, but everything else should be good to go once the linter and tests pass!

Comment thread runtimes/alibi-detect/mlserver_alibi_detect/runtime.py Outdated
Comment thread runtimes/alibi-detect/mlserver_alibi_detect/runtime.py Outdated
Comment thread runtimes/alibi-detect/tests/conftest.py Outdated
@SachinVarghese SachinVarghese self-requested a review April 19, 2023 17:25
@ascillitoe ascillitoe marked this pull request as ready for review April 20, 2023 12:05
@adriangonz adriangonz merged commit 1e7bc4b into SeldonIO:master Apr 20, 2023
adriangonz pushed a commit that referenced this pull request Apr 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants