Comments (11)
Thanks! I have new reporting almost rebuilt, which is now also more focused giving us access to details that are vital to the next level of abstraction i.e. optimizing the optimizing. Once we get the current dev to master, I'll add them to dev. What do you think about merging with master now?
from talos.
I think we could but if you’re refactoring Reporting we might as well wait until you’re done with that. Is there any reason to merge sooner?
from talos.
Ok great, yes I think it's good we merge now first. For reporting I'm actually writing it from scratch on the basis of what I found was useful. What do you think would be useful features to add in it?
from talos.
@mikkokotila Definitely a few things from my experience with it that I would recommend.
- Allow for the option to sort by different quantities (validation accuracy, validation loss, training loss, etc.).
- Definitely let the user easily trim the columns. In other words, maybe we don't care about anything other than the validation accuracy. Therefore, we can let the user easily not display the training loss, validation loss and training accuracy. There is a very easy way to do this with pandas that I've used before. Let me know if you have any trouble finding/using it and I can dig it up.
- Easy callbacks of various histories. Perhaps there's a way to get Scan to save all those. This would be incredibly useful I think.
- Can also trim the rows, so maybe only display the top 5 permutations. This is also easy to do in pandas I think.
- Customizable display options (e.g. if user is running on terminal, let them set something to true/false so that instead of trying to display using pandas it just saves the .csv).
Can't think of anything else off the top of my head but yeah that's a good place to start! Let me know if there's anything I can do. Otherwise when you're done I'll try to help comment/cleanup and whatnot. 👍
from talos.
@x94carbone thanks a lot. In terms of the callbacks, could you share some examples for the use-cases. Generally storing the history is of course easy, and it seems that the data size should be totally ok as long as we handle it in an array the right way during, and dump to the df only in the end of the process.
For the displaying stuff I made the whole thing simpler in the way where you have a class object that takes in one parameter, which is the log, and then after that you have various properties (like peak round). Then it would be very easy to extend it. The code is super clean without any of the complexities of the current one.
from talos.
Nice, that sounds good. By callbacks do you just mean how I manipulate the dataframe to trim columns and that sort of thing?
from talos.
Well, in case that was what you were referring to, here's the answer. For the current state of Talos:
First, if using a notebook, this will make the output easier for you to deal with.
import pandas as pd
from IPython.display import clear_output, display
Load the data you just ran using Talos, but suppress the output since it's not always going to be clearly formatted depending on some unknown factors.
r = ta.Reporting('my_exp_1.csv')
clear_output()
Example of removing the columns in remove_columns
and sorting the list by sort_by
.
remove_columns = ['round_epochs', 'acc', 'loss']
sort_by = 'val_fbeta_score'
rr = pd.DataFrame(r.data.sort_values(sort_by, ascending=True)).drop(remove_columns, axis=1)
display(rr)
Is this helpful?
from talos.
Hi there,
For the format of the output, I don't know why the column names are not in the right order.
Do you guys know how to fix this ?
Thanks
from talos.
@channhan007 the whole reporting piece is going to be replaced with a new much cleaner approach that also supports the various use-cases including far better integration with plots. In the meantime for the simple table view, you might just use:
from pandas import read_csv
read_csv('experiment.csv')
from talos.
@mikkokotila Thank you for your response. I look at the csv output file and the column labels are not in the right order. I have to rearrange those column names.
from talos.
The new reporting is now included in dev. The old is removed, so closing here. Later we can have separate issues for new features for Reporting(). It's in /utils/reporting.py
from talos.
Related Issues (20)
- talos execution fails of very first example: HOT 1
- Resume run possible? HOT 2
- Key duplication results in exception: param dict vs. csv file stored (e.g., 'loss') HOT 3
- Heatmap always have missing parts HOT 2
- Failed to build scipy while installing talos HOT 1
- Using `Evaluate` with multiple inputs HOT 6
- ImportError: cannot import name 'get_config' HOT 4
- add arguments to the function syntax HOT 1
- Using train, val data HOT 1
- Cannot replicate typical example from documentation HOT 1
- Can't find description of layer shapes HOT 3
- Talos installation stuck at downloading statsmodel HOT 1
- [RFC00102] Increase Visibility and Access to Power Draw Data HOT 1
- Getting errors when handling custom object HOT 4
- Add comprehensive support for n-dimensional `x` or `y`
- AttributeError: module 'talos.utils' has no attribute 'ExperimentLogCallback' HOT 1
- Live is not in talos HOT 1
- Advanced hyperparameter configuration HOT 1
- Error with Autom8 import HOT 1
- pip install talos is failing due to sklearn in requirements.txt instead of scikit-learn HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from talos.