Comments (7)
hi, thanks for your interests. Could you please provide your experimental results? Just FYI, you need to tune the hyperparameters following the experimental setup described in our paper.
from fedchem.
Thanks for your reply, I just run the sample code given in the readme file, like
python main.py -dataset esol -fedmid avg -part_alpha 0.1
but the result is as follows:
Surprised that this result is better than the one you gave in the article (even better than the FLIT(+) results reported in the article which marked as Best federated-learning results)
However, when we tried FLIT/FLIT+,
python main.py -dataset esol -fedmid oursvatFLITPLUS -tmpFed 0.5 -lambdavat 0.01 -part_alpha 0.1
we got worse results than FedAvg
from fedchem.
as I mentioned, you need to tune the hyperparameters for FLIT(+) follow our paper. We do not find a set of hyperparameters that fits all datasets. However, it is wired to see that fedavg has such good performance. I will check our experiments, and will get you back soon.
from fedchem.
Hi, I have checked our previous experimental results and also re-run the experiments.
First, I did obtain the reported results for FedAvg on ESOL dataset as shown below.
I also admit that I cannot reproduce the results with our current code for this dataset. However, I should note that ESOL is extremely small and the training/testing performance is pretty unstable. I may suggest you play with our code on larger datasets e.g. Lipo. Anyway, thanks very much for pointing out the problem, and let me know if you have any other questions.
from fedchem.
Thanks for your prompt response, with reference to your suggestion, I have also executed the relevant FedAvg code on other datasets, but also got surprising results on some datasets, as follows.
- Freesolv: As the degree of data heterogeneity increases, the test metrics of the Freesolv dataset also increase, however, the lower the metrics of this dataset, the better.
- ClinTox: As with the ESOL problem, the Avg results on this dataset significantly outperform the state-of-the-art results for all methods reported in the paper.
from fedchem.
As for problem 1, we make the claim in our paper that our current heterogenous simulation method is not perfect and may not result in heterogeneous datasets. We give discussion in the main results section and also in the conclusions section. More research should be done in this direction.
As for problem 2, I beleive there may be some small differences between our current code and the one when we run the experiments. I am really sorry for this. Our results on these two datasets are consistent for all methods and I thus believe the results should still be able to work as a reference for comparision. I also paste the experimental records on fedavg for clintox here.
from fedchem.
Thanks for your reply. In view of the current problems, I suggest that you carefully check the current open source code for errors and update the correct code. If the current code has no errors in FedAvg, it is obvious that you did not find the optimal baseline of FedAvg. I also recommend that you re-run the relevant experiments of FLIT(+) (especially on the ESOL and ClinTox datasets) to ensure that the conclusions in the paper are correct.
from fedchem.
Related Issues (1)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fedchem.