aimhubio / aim Goto Github PK
View Code? Open in Web Editor NEWAim π« β An easy-to-use & supercharged open-source experiment tracker.
Home Page: https://aimstack.io
License: Apache License 2.0
Aim π« β An easy-to-use & supercharged open-source experiment tracker.
Home Page: https://aimstack.io
License: Apache License 2.0
While setup.py
specifies
REQUIRES_PYTHON = '>=3.5.2'
I think aim doesn't work with Python 3.5.2 because of this part.
Lookup the way git does environment variables based on folder.
Ideally AIM_DIR should be set on aim init
Trying to get aim up and running. From the user account it reports docker not installed (not true, but I'm assuming it requires root). Fresh conda install as root, conda create -n keras python=3.8 pip
then pip install aim
aim up
File "/root/anaconda3/envs/keras/bin/aim", line 5, in <module>
from aim.commands.cli import cli
File "/root/anaconda3/envs/keras/lib/python3.8/site-packages/aim/commands/cli.py", line 3, in <module>
from aim.commands.cmd_provision import provision_command
File "/root/anaconda3/envs/keras/lib/python3.8/site-packages/aim/commands/cmd_provision.py", line 4, in <module>
from aim.commands.helpers import controller_args, aim_home_option, init_aim_home_option, pass_aim_context, \
File "/root/anaconda3/envs/keras/lib/python3.8/site-packages/aim/commands/helpers.py", line 4, in <module>
from aim.config.aim_context import AimContext, AccountContext
File "/root/anaconda3/envs/keras/lib/python3.8/site-packages/aim/config/aim_context.py", line 3, in <module>
import aim.controllers
File "/root/anaconda3/envs/keras/lib/python3.8/site-packages/aim/controllers/__init__.py", line 1, in <module>
from aim.controllers.ctl_network_environment import NetEnvController
File "/root/anaconda3/envs/keras/lib/python3.8/site-packages/aim/controllers/ctl_network_environment.py", line 10, in <module>
from aim.stack_grps.grp_application import ApplicationStackGroup
ModuleNotFoundError: No module named 'aim.stack_grps'```
Any ideas?
I install docker in Ubuntu, then I start it and my docker service is active and add the my user in docker group also.
Then I run aim up
in my anaconda environment. But it says Oops I donβt have docker installed.
Could u please specify how to run theaim up
?
.aim/deploy_temp
Hello!
I am looking at your logger internal, and here I see that the args are optional.
class _PytorchLightningLogger(LightningLoggerBase):
def __init__(
self,
repo: Optional[str] = None,
experiment: Optional[str] = None,
train_metric_prefix: Optional[str] = "train_",
val_metric_prefix: Optional[str] = "val_",
test_metric_prefix: Optional[str] = "test_",
flush_frequency: int = DEFAULT_FLUSH_FREQUENCY,
):
Nevertheless, if I do not fill anything inside and try to run a model, I get the following error when running model.fit
File "/home/gem/.pyenv/versions/3.8.6/lib/python3.8/posixpath.py", line 90, in join
genericpath._check_arg_types('join', a, *p)
File "/home/gem/.pyenv/versions/3.8.6/lib/python3.8/genericpath.py", line 152, in _check_arg_types
raise TypeError(f'{funcname}() argument must be str, bytes, or '
TypeError: join() argument must be str, bytes, or os.PathLike object, not 'NoneType'
It might be useful to have some native colab integration. I am wondering what are your plans in that direction?
Would it be possible to add a host option to aim up
in addition to the port one? I'd like to run the server on 0.0.0.0
instead of localhost, like this:
aim up -p 5000 -h 0.0.0.0
Maybe it would be possible to take advantage of create_host_config
or something along those lines...
It would be super helpful if highlighting one metric plot (e.g. mouse hovering on the plot) causes all the other metrics from that specific run in every chart to be highlighted, too!
I believe it is a common scenario to compare hyperparameter sets with more than one evaluation metrics at the same time.
Thank you.
Currently, the only way to delete the run(s) is by using the CLI. The thing is that as far as I understand you have to delete the runs 1 by 1, i.e. :
aim experiment rm -n experiment_name
It would be very nice to have functionality like
aim experiment rm -rf *
or
aim experiment rm -n run1* (For specific batches of runs)
Taking regexp like expressions to find the runs and delete/archive them.
It would also be very helpful to have a "delete by date" type of functionality. I think it should also be easy to use AimQL to find experiments and delete them in the same line. I.E.
aim experiment rm ("metric-loss" , score_1, l_reg_1 if dataset.dataset_run==FB15K)
Of course, I would love to see a simple deletion/archive button in frontend as well.
The UX of the aim-deploy when the deployable is being created should be transparent and provide a good insight on what is happening behind the scenes.
There is an ability at Aim to read Tensorboard logdir and search through the experiments recorded by TensorBoard.
Add section in docs to explain how to run Aim on Tensorboard logdir.
Sometimes, especially for testing purposes, I want to run a multitude of experiments/runs without having to manually delete the runs/experiments from the previous test script. Is there a way to manager runs and experiments from the SDK for this purpose?
I'm trying to track multiple experiments within a Jupyter notebook by reinitializing a session like so:
import aim
session = aim.Session()
session.track(1, name='val')
session.track(2, name='val')
session.close()
session = aim.Session()
session.track(3, name='val')
session.track(4, name='val')
session.close()
The runs show correctly in the dashboard but for the second run, the metric starts showing at step 2 instead of 0. This prevents me from comparing two runs in the Explore tab as they are on different time scales. Is this the expected behaviour? If so, how can I track multiple runs within a single process?
Hello, I am facing this issues trying to get aim up and running:
File "/conda/envs/torch/lib/python3.8/site-packages/aim/engine/container/container.py", line 50, in image_exist
images = cls._get_docker_client().images.list()
AttributeError: 'function' object has no attribute 'list'
I dug a little further, solving the issues below and found that docker_client.containers
is an empty list. I am using a third-party code that uses aim, and I'm wondering if these issues are related to a different version used to create the logging files or if it is a docker issue.
Thanks in advance.
File "/conda/envs/torch/lib/python3.8/site-packages/aim/engine/container/container.py", line 52, in image_exist
for t in i.attrs['RepoTags']:
AttributeError: 'dict' object has no attribute 'attrs'File "/conda/envs/torch/lib/python3.8/site-packages/aim/engine/container/container.py", line 100, in get_container
containers = docker_client.containers.list(all=~running_only,
AttributeError: 'function' object has no attribute 'list'File "/conda/envs/torch/lib/python3.8/site-packages/aim/engine/container/container.py", line 112, in kill
aim_containers = docker_client.containers.list(all=True, filters={
AttributeError: 'function' object has no attribute 'list'File "/conda/envs/torch/lib/python3.8/site-packages/aim/engine/container/container.py", line 83, in up
container = docker_client.containers.run(image_name,
AttributeError: 'function' object has no attribute 'run'File "/conda/envs/torch/lib/python3.8/site-packages/aim/engine/container/container.py", line 83, in up
container = docker_client.containers(image_name,
TypeError: containers() got an unexpected keyword argument 'name'
Hi, thanks for your fantastic work!! But i hava a question: i use pytorch and argparse, when i use like these:
`
args = parser.parse_args()
timestamp = time.strftime("%m-%d_%H-%M", time.localtime())
aim.set_params(args, name='hparams') # Passing name argument is optional
`
the name of my runs will automaticly be 'default' ? how can i specify this ?
and i can not see my args in runs?
thank you!!
Can we also just have
aim down
instead of
aim de down
Problem
New users who are not familiar with Aim and Aim query language don't know how to search/retrieve tracked experiments and how to make search queries in explore screen.
Solution
Show examples of Aim QL and provide links to QL documentation and detailed specs.
Details
Inside "IF" input component of explore screen add hint icon. When clicking on that, open popover with following information:
Code examples to show:
context.subset in ("train", "val") and hparams.learning_rate >= 0.01
experiment == "XLM-syntok-dynamic-full-titles-avg" and hparams.align == "XLM" and hparams.dataset in ("qqp", "mrpc")
Footer content:
"Aim Query Language is pythonic and fairly easy to get used to. If having issues, please refer to the docs for more examples or the detailed spec."
docs href: https://github.com/aimhubio/aim#searching-experiments
spec href: https://github.com/aimhubio/aim/wiki/Aim-Query-Language
It might be a nice addition to the aim UI to support selecting one of the existing metrics and using it for the x-axis instead of the epochs.
Thank you.
Build Packaging to properly create the aim executable.
Should be installable by Pip? Make decision on this too.
Hi,
can I get some help regarding some situation probably regarding Aim UI?
I have many (48-64) runs in each experiment and also there exist almost 90 metrics.
At the early stages of the training, I could use Aim UI as usual.
But at some point (not sure when exactly), this issue appeared.
While Dashboard lists all the runs, they only show the run parameters set by set_params()
, but not the metrics.
In Dashboard, I can click on each run to view its details.
However, Aim UI does not display anything but the message Experiment "[experiment name]" does not exist
although the drop-down menu above still shows [experinemt name]
, for some of the runs.
I also saw some cases that in the run information page, only a subset of the metrics was shown.
Plus, in Explore, I can query only some subset of the metrics, as well.
Fortunately, it looks like the saving part is fine; I can see all of them (runs and metrics) and that the disk consumptions are consistent.
Is there any workaround that might help me resolve the issue, please?
It would be great to be able to list all of your runs in CLI as well as frontend. I.E:
aim experiment list
and filter runs in CLI as well.
Hi and thanks for the great work here :D So I am using the Session api, and I was wondering if there is any way to programmatically restart the experiment. I was thinking perhaps something like
sess = aim.Session(experiment = 'some_name')
sess.clear()
Hi!
it would be very helpful if the metric can be add in one tag like the tensorboard?
eg:
writer.add_scalar(f'train_detail_loss/WholeLoss' %
, WholeLoss, epoch)
writer.add_scalar(f'train_detail_loss/OriginalLoss' %
OriginalLoss, epoch)
besides, if the number of params is too big, the web page is all blank after ' aim up ' command?
thank you!!
Hello,
It is possible to change how graphs are plotted switching from linear to log scale?
Best,
Tommaso
Is there a way to record multiple runs, belonging to the same experiment, in the same Python script without explicitly using the Session object? So far, I was able to do it with the Session object, but it seems to me that it would be cleaner if it was possible by adding an extra argument in the aim.set_params function.
Also having an issue with the keras tracking callback:
File "GeneFindingTestingTrainedTransformer.py", line 276, in <module>
callbacks=[csvlog, AimTracker.metrics([aobj])])
File "/home/josephguhlin/miniconda3/envs/keras/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py",
line 108, in _method_wrapper
return method(self, *args, **kwargs)
File "/home/josephguhlin/miniconda3/envs/keras/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py",
line 1074, in fit
steps=data_handler.inferred_steps)
File "/home/josephguhlin/miniconda3/envs/keras/lib/python3.7/site-packages/tensorflow/python/keras/callbacks.py", line 2
35, in __init__
cb._implements_train_batch_hooks() for cb in self.callbacks)
File "/home/josephguhlin/miniconda3/envs/keras/lib/python3.7/site-packages/tensorflow/python/keras/callbacks.py", line 2
35, in <genexpr>
cb._implements_train_batch_hooks() for cb in self.callbacks)
AttributeError: 'TrackMetricsCallback' object has no attribute '_implements_train_batch_hooks'```
Hi, can aim fold several graphs together? As in medical image segmentaion field, we ofter need to try k-fold validation. so if aim can organize the metrics of same fold together, it would be great!! Actually, right now, i use the following code under tensorboard to realize this:
## fold can be in [fold1,fold2,...fold_n]
writer.add_scalar(f'%s/train_detail/Loss' %fold, train_loss, epoch)
writer.add_scalar(f'%s/train_detail/Acc' %fold, train_acc, epoch)
writer.add_scalar(f'%s/val_detail/Loss' % fold, val_loss, epoch)
writer.add_scalar(f'%s/val_detail/Acc' %fold, val_acc, epoch)
so the page will be like π:
Moreover, can aim track several metrics at one time ? like < add_scalars > in tensorboard?
Huge thanks !!
Aim needs to work seamlessly for environments such as Pytorch distributed
or Slurm
or other well-adopted distributed setups.
failed in windows,there was no output after i type" aim up"
i have docker pull aimhubio/aim-board
Thanks for your awesome product!
I found that archived runs reappear after a while.
I'm using the latest version (2.1.5).
Is there any debug log that I can provide to you? How to find these logs?
Aim init initializes folder as an aim repo.
Aim init should set initial folders up and enable further development and easy usage of the commands.
Thank you for your awesome work! I want to manually add some notes for some experiments on the webpage. How should I do this?
Hi,
I am sorry for another maybe basic question which I cannot find doc or explanation from the README.
When I tried aim up --tf_logs <my_location> --host localhost --port 10008
It directly returned me
Failed to run Aim UI.
Please check if address localhost:10008 is accessible.
Is it related to firewall? Or some setting I must do before launching it?
I originally thought it was functioned similarly as tensorboard and I am trying the same way.
My log files are located in a remote server, I tried to run aim at the server for some port, and I will forward it to my local machine's local port so that I can open it in a browser.
Thank you very much
Problem
Not possible to select multiple metrics that meet different conditions
Solution
Pass metric name to match
function
I look at the docs and find that track func only accept epoch as input.
In fact, in modern deep learning, we always use iteration as the points in time axis.
Although I can use iteration as epoch, but I also want to record some metrics which only corresponding to epoch.
So, can you have any solutions?
I want aim to store the hyperparameter configuration that was used for the experiment. I have tried the following, but it does not seems to work
model = Training_Models.ED_Training() lighting_data = datamodule.DataModule(data, conf) trainer = pl.Trainer( gpus=-1, logger=AimLogger(experiment="Test"), max_epochs=10, ) model.hparams.update(conf.hyperparameters)
How can I achieve that?
Hi, I installed aim using pip and type aim up
, it returns me:
Oops! You don't have docker installed. Aim UI runs on docker. Please install docker
I have followed https://docs.docker.com/engine/install/ubuntu/
to install docker engine on my machine. But the above errors are still there. What should I do for that?
I see artifact names are validated to only include alphanumeric characters by the follow regex: re.search(r'^[^\d\W]\w*\Z', str(name))
This does not allow for artifact names such as total_loss
or fast_rcnn/cls_accuracy
. Is there any particular reason why artifact names cannot contain special characters?
The plain templates for docker app are not included in the egg when built locally.
This needs to be fixed.
In order to save aim logs in more secure and reliable place and also share it with other team members, it would be better to have this feature added.
When I run aim up
in conda environment, Aim can't find docker which is already installed. Could you please let me know how to resolve this?
Problem
Parallel coordinates plot doesn't handle long texts
Solution
Format long texts in parallel coordinates plot axes ticks:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. πππ
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google β€οΈ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.