KerasClassifier : No extension registered which can handle model

[sahithyaravi]

Description

No extension registered error when trying to use keraswrapper
The lastest flow with Keraswrapper/KerasClassifier seems to be based on openml v0.8.0

Steps/Code to Reproduce

Example:

from keras.wrappers.scikit_learn import KerasClassifier
task = openml.tasks.get_task(3573)  # mnist
model = KerasClassifier(build_fn=your_model_build_function, epochs=8, batch_size=32, verbose=1)
openml.runs.run_model_on_task(model, task)

Expected Results

No error is thrown.

Actual Results

ValueError                                Traceback (most recent call last)
<ipython-input-6-c3c90f60e704> in <module>()
      2 mnist_task=openml.tasks.get_task(3573)
      3 # Run the Keras model on the task (requires an API key).
----> 4 mnist_lenet_run =openml.runs.run_model_on_task(mnist_task, lenet_model, dataset_format ="array")
      5 # Publish the experiment on OpenML (optional, requires an API key).
      6 #myrun = mnist_lenet_run.publish()

1 frames
/usr/local/lib/python3.7/dist-packages/openml/extensions/functions.py in get_extension_by_model(model, raise_if_no_extension)
     90     if len(candidates) == 0:
     91         if raise_if_no_extension:
---> 92             raise ValueError("No extension registered which can handle model: {}".format(model))
     93         else:
     94             return None

ValueError: No extension registered which can handle model: <tensorflow.python.keras.wrappers.scikit_learn.KerasClassifier object at 0x7f5bfdf6e2d0>

Versions

Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
Python 3.7.10 (default, Feb 20 2021, 21:17:23)
[GCC 7.5.0]
NumPy 1.19.5
SciPy 1.4.1
Scikit-Learn 0.22.2.post1
OpenML 0.12.1

[mfeurer]

Hey, I just tried running the snippet, but the network definition isn't there (NameError: name 'your_model_build_function' is not defined). Could you please add that to the snippet?

[sahithyaravi]

Hey, I just tried running the snippet, but the network definition isn't there (NameError: name 'your_model_build_function' is not defined). Could you please add that to the snippet?

Hi Matthias I meant any arbitrary model function. i can share the colab notebook . u need to configure api key. https://colab.research.google.com/drive/1PgHf4LpwzgZkAkzZGbGO_A2L2lKh_wan?usp=sharing

[mfeurer]

Thanks @sahithyaravi1493.

The issue turns out to be that our definition of a scikit-learn-compatible model is isinstance(model, sklearn.base.BaseEstimator) in the function can_handle_model.

We should probably follow the suggestions in the box "BaseEstimators and mixins:" in the scikit-learn docs, which state:

We tend to use “duck typing”, so building an estimator which follows the API suffices for compatibility, without needing to inherit from or even import any scikit-learn classes.

[sahithyaravi]

Thanks @sahithyaravi1493.

The issue turns out to be that our definition of a scikit-learn-compatible model is isinstance(model, sklearn.base.BaseEstimator) in the function can_handle_model.

We should probably follow the suggestions in the box "BaseEstimators and mixins:" in the scikit-learn docs, which state:

We tend to use “duck typing”, so building an estimator which follows the API suffices for compatibility, without needing to inherit from or even import any scikit-learn classes.

Ah I see, they also have an is_classifier method, which if I check against the KerasClassifier returns False:
sklearn.base.is_classifier(lenet_model) = False.

So we must also do ducktyping I think? Or if scikit learn has a separate method which performs this, we could use it

[sahithyaravi]

• Based on https://github.com/adriangb/scikeras, scikit learn is no longer maintaining the keras wrapper tf.keras.wrappers.scikit_learn. scikeras seems to be the suggested way, as found here https://www.adriangb.com/scikeras/stable/migration.html.
• Another page which confirms the wrapper is not supported in sklearn : Keras scikit-learn wrapper not compatible with keras functional model tensorflow/tensorflow#34689.
• scikeras works with the OpenML sklearn extension without any changes from our end, as it passes isinstance(model, sklearn.base.BaseEstimator).
• We need to document that isinstance(model, sklearn.base.BaseEstimator) is the required condition for using sklearn extension.

[mfeurer]

This can be closed, isn't it?

A suggestion in case this happens more often: every extensions returns a string explanation on why it didn't handle the current model - then the user knows what went wrong.

[sahithyaravi]

This can be closed, isn't it?

A suggestion in case this happens more often: every extensions returns a string explanation on why it didn't handle the current model - then the user knows what went wrong.

I am not sure how often this happens. We could close this issue and have a new enhancement issue for adding the explanation.