I noticed that the docstrings for gctf.centralized_gradients_for_optimizer have an error in the example usage section. The example creates an Adam optimizer instance and saves it to opt however the centralized_gradients_for_optimizer is applied on optimizer which ideally does not exist and running the example would result in an error.

On windows 10 with miniconda enviroment tensorflow 2.5 gives error on centralized_gradients.py file.

the solution is change
import keras.backend as K
with
import tensorflow.keras.backend as K

I am specifically thinking of adding three more categories of pypi classifiers:

• Development status
• Intended Audience
• Topic

Apart from this I also think it would be great to add the Programming Language :: Python :: 3 :: Only to make sure the audience to know that this package is intended for Python 3 only.

It would be great to write an "About the example" section which could demonstrate in short what the example notebooks aim to achieve and show.

It would be great to include an example showing the use of this package with a custom optimizer built on top of tf.keras.Optimizer class.

Describe the bug
The results in your colab ipython notebook are misleading: https://colab.research.google.com/github/Rishit-dagli/Gradient-Centralization-TensorFlow/blob/main/examples/gctf_mnist.ipynb

In this example, the model is first trained with a normal Adam optimizer:

model.compile(optimizer = tf.keras.optimizers.Adam(),
              loss = 'sparse_categorical_crossentropy',
              metrics = ['accuracy'])

history_no_gctf = model.fit(training_images, training_labels, epochs=5, callbacks = [time_callback_no_gctf])

And afterwards the same model is recompiled with the gctf.optimizers.adam(). However, recompiling a keras model does not reset the weights. This means that in the first fit call the model is trained and then in the second fit call with the new optimizer the same model is used and of course then the results are better.

This can be fixed, by recreating the model for the second run, by just adding these few lines:

import gctf #import gctf

time_callback_gctf = TimeHistory()

# Model architecture
model = tf.keras.models.Sequential([
                                    tf.keras.layers.Flatten(), 
                                    tf.keras.layers.Dense(512, activation=tf.nn.relu),
                                    tf.keras.layers.Dense(256, activation=tf.nn.relu),
                                    tf.keras.layers.Dense(64, activation=tf.nn.relu),
                                    tf.keras.layers.Dense(512, activation=tf.nn.relu),
                                    tf.keras.layers.Dense(256, activation=tf.nn.relu),
                                    tf.keras.layers.Dense(64, activation=tf.nn.relu), 
                                    tf.keras.layers.Dense(10, activation=tf.nn.softmax)])

model.compile(optimizer = gctf.optimizers.adam(),
              loss = 'sparse_categorical_crossentropy',
              metrics=['accuracy'])

history_gctf = model.fit(training_images, training_labels, epochs=5, callbacks=[time_callback_gctf])

However, then the results are not better than without gctf:

Type                   Execution time    Accuracy      Loss
-------------------  ----------------  ----------  --------
Model without gctf:           24.7659    0.88825   0.305801
Model with gctf               24.7881    0.889567  0.30812

Could you please clarify what happens here. I tried this gctf.optimizers.adam() optimizer in my own research and it didn't change the results at all and now after seeing it doesn't work in the example which was constructed here. Makes me question the results of this paper.

To Reproduce
Execute the colab file given in the repository: https://colab.research.google.com/github/Rishit-dagli/Gradient-Centralization-TensorFlow/blob/main/examples/gctf_mnist.ipynb

Expected behavior
The right comparison would be if both models start from a random initialization, not that the second model can start with the already pre-trained weights.

Looking forward to a fast a swift explanation.

Best,
Max

I wonder which versions of TensorFlow are supported by this package

The package as of now to be installed requires tensorflow ~= 2.4.0 and keras ~= 2.4.0. It turns out that this is sometimes problematic for folks who have custom installations of TensorFlow and a winder requirement could be set up.