Matthews correlation coefficient for multi-class

[ZeynepP]

HI to all,

I am trying to interpret different results I am getting in tensorflow for MCC. I have 3 classes and I am using categorical_crossentropy as a loss function. I added mcc in metrics from tensoflow addons as follows:

tfa.metrics.MatthewsCorrelationCoefficient(name="mcc", num_classes=3)
...
test_metrics = model.evaluate(test_gen, verbose=1)
test_mcc = test_metrics[model.metrics_names.index("mcc")]

# THIS PART IS ADDED TO TEST DIFF BETWEEN EVAL AND PREDICT
y_classes = test_gen.targets.argmax(axis=-1)
pred_prop = model.predict(test_gen, verbose=1)
test_mcc_sklearn = matthews_corrcoef(y_classes, pred_prop.argmax(axis=-1))

Training output (ex for one epoch) : ... - loss: 0.7549 - acc: 0.7276 - auc: 0.8313 - mcc: 0.3735 - val_loss: 0.7677 - val_acc: 0.7302 - val_auc: 0.8211 - val_mcc: 0.3745...
Evaluate output : ... loss: 0.7879 - acc: 0.7124 - auc: 0.8107 - mcc: 0.3768 ...
test_mcc = [0.24070711 0.29297936 0.5968444 ]
test_mcc_sklearn = 0.04130715666673915

It is the first time I am using mcc and I am really confused with this different results !?? What am I missing?
Thanks in advance.

[ZeynepP]

Very basic example :

https://colab.research.google.com/drive/1dErU9wesGrXC7ILJEfpocF9zK5laRfX7?usp=sharing

Bests

[jonpsy]

Hi! May I take this up?

[ZeynepP]

I wish someone does...

[jonpsy]

I think our current implementation is wrong, it uses one vs all for getting Matthew's coefficient for each class. Instead, it should've used the generalized Matthew's Formula(for Multi & binary) as is provided here: https://en.wikipedia.org/wiki/Matthews_correlation_coefficient#Multiclass_case. The same is implemented in sklearn.

Have I gathered the context right @autoih @marload?

[ZeynepP]

I debug the tfa.mcc and sklearn mcc. As @jonpsy mentioned the formula implemented in the actual version is not correct. I updated it following to wiki and sklearn code :

def update_state(self, y_true, y_pred, sample_weight=None):
        y_true = tf.cast(y_true, dtype=self.dtype)
        y_pred = tf.cast(y_pred, dtype=self.dtype)

        C = tf.math.confusion_matrix(
            labels=tf.argmax(y_true, 1),
            predictions=tf.argmax(y_pred, 1),
            num_classes=self.num_classes,
            weights=sample_weight,
            dtype=self.dtype,
        )

        t_sum = tf.reduce_sum(C, axis=1)
        p_sum = tf.reduce_sum(C, axis=0)

        n_correct = tf.linalg.trace(C)
        n_samples = tf.reduce_sum(p_sum)

        cov_ytyp = n_correct * n_samples - tf.tensordot(t_sum, p_sum, axes=1)
        cov_ypyp = n_samples ** 2 - tf.tensordot(p_sum, p_sum, axes=1)
        cov_ytyt = n_samples ** 2 - tf.tensordot(t_sum, t_sum, axes=1)
        self.mcc = cov_ytyp / tf.math.sqrt(cov_ytyt * cov_ypyp)
        if tf.math.is_nan(self.mcc ) :
            self.mcc = tf.constant(0, dtype=self.dtype)

def result(self):
       return self.mcc

I am getting the same result as sklean with this implementation.

FYI.
Bests

[jonpsy]

I was meaning to implement this myself but since nobody was replying I reconsidered if I was permitted to do it(and whether you were interested still) . Regardless, I'm happy to see your question answered :)

[WindQAQ]

Hi all, I am sorry I have just noticed this thread. It's more than welcome to submit a PR for it! Thanks!

[jonpsy]

@WindQAQ I was wondering, how about we keep them both? That is, Matthew correlation in one vs all method & the actual method using a boolean switch?

[aminzabardast]

Hi, Everyone. Since we have this fix in the merge queue, I was wondering if the stable version's (0.12.1) MCC for Multiclass is implemented correctly or not. Shall I refrain from using it?

[jonpsy]

Hi, Everyone. Since we have this fix in the merge queue, I was wondering if the stable version's (0.12.1) MCC for Multiclass is implemented correctly or not. Shall I refrain from using it?

It's not wrong per sé, it gets the coefficients for each of the class, so in a sense it is "multi-class".

In a general sense though, the implementation is "wrong". Personally, I'd refrain from using it.