Again, with multi_gpu_model (haven't tried with normal model) I get the following chart if using "steps" as X axis:

I looks like it's going back to step=epoch after each epoch, i.e. after the first epoch it logs the loss with step=0, then logs all the subsequent steps correctly, then again goes back to the left side of X axis. It makes the chart unreadable. "Wall time" on X axis works fine:

• on second run in Jupyter Notebook get error:
• Could not connect to server after multiple tries.
• issue only resolves with kernel shutdown

enable users to export data into csv file

From a user:

The API key doesn’t appear when starting with a new account, when logged in. However, it appears when logged out in the QuickStart Guide. Then, I add the project to my account and it works fine thereafter.

In the current version of Comet during the experiment it is possible to log metrics split into steps (of iterations through a dataset) and epochs.

It could be useful to have a third layer of hierarchy - folds. It would be useful to have overlapping charts plotted coming from different folds of cross-validation.

It can be overcome as separate experiments however in fact these are different folds of one experiment.

I'm not sure if this is the right place to post this question.
I've just started to use comet-ml and it's a great product. I was always wanted to have such functionality.

In Keras I'm training a network using a custom loss function and custom metrics.
My model.compile() looks like this:
model.compile(loss=contrastive_loss, metrics=[my_custom_metric1, my_custom_metric2])

When the network is training, custom loss and metrics are not reported on the comet-ml experiment site. I can see these values in stdout section (as printed by Keras on the terminal). But metrics section and graph is empty.

How can I report and graph these values during the training. Documentation mentions using experiment.log_metric() function. But where/how do I call this function. Should I call this function from within my custom loss and metric functions? So the custom loss and metrics is reported for each minibatch?

Or is there a way to automatically report and graph custom loss and metrics passed as arguments to model.compile?

The chart function is very useful but sometimes limits the number of datapoints displayed. Sometimes it will not display the entire trend and will simply display the last 20 or so points. With long running experiments this is not ideal - is there a way to 'zoom out' or display the entire series?

I have a currently running project at:

https://www.comet.ml/jarrelscy/crop/753d855f88654ec3b4ccecfe0277fbbe

This has been running for a while now but only the last 16 or so datapoints are displayed.

Summary

Hi,
I'm getting a traceback raised during training of a Keras' model. This only happenes with an instantiated comet.Experiment class at the top of the file.

If you need any more information, please let me know!
Many thanks,
Dom

Environment

comet-ml (0.1.40)
Python 3.5.2 (default, Sep 14 2017, 22:51:06) 

Distributor ID:    Ubuntu
Description:   Ubuntu 16.04.3 LTS
Release:      16.04
Codename: xenial

Traceback

Traceback (most recent call last):
  File "run_polarity2.py", line 481, in <module>
    model.fit(X_train, y_train, epochs = 2, validation_data = (X_test, y_test))
  File "/home/dom/env/lib/python3.5/site-packages/keras/models.py", line 893, in fit
    initial_epoch=initial_epoch)
  File "/home/dom/env/lib/python3.5/site-packages/comet_ml/keras_logger.py", line 15, in wrapper
    return real_fit(*args, **kwargs)
  File "/home/dom/env/lib/python3.5/site-packages/keras/engine/training.py", line 1631, in fit
    validation_steps=validation_steps)
  File "/home/dom/env/lib/python3.5/site-packages/keras/engine/training.py", line 1161, in _fit_loop
    callbacks.on_epoch_begin(epoch)
  File "/home/dom/env/lib/python3.5/site-packages/keras/callbacks.py", line 59, in on_epoch_begin
    callback.on_epoch_begin(epoch, logs)
  File "/home/dom/env/lib/python3.5/site-packages/keras/callbacks.py", line 276, in on_epoch_begin
    verbose=self.verbose)
  File "/home/dom/env/lib/python3.5/site-packages/keras/utils/generic_utils.py", line 270, in __init__
    self._dynamic_display = (sys.stdout.isatty() or
AttributeError: 'StdLogger' object has no attribute 'isatty'
uploading stats to Comet before program termination (may take several seconds)
go to https://www.comet.ml/view/jLVzb*****************to view your run.

Model

model = Sequential()
model.add(Embedding(output_dim = 300, input_dim = 10000, mask_zero = True))        
model.add(Bidirectional(LSTM(32, dropout = 0.2, activation = 'tanh')))
model.add(Dense(1, activation = 'sigmoid'))

model.compile(
    optimizer = 'adam',
    loss = 'binary_crossentropy',
    metrics = ['accuracy']
)

model.fit(X_train, y_train, epochs = 2, validation_data = (X_test, y_test))

When using comet_ml, the Keras progress bar refreshes with every minibatch, as if the end='\r' parameter was ignored in print():

1/30 [>.............................] - ETA: 45s - loss: -0.2396
 2/30 [=>............................] - ETA: 43s - loss: -0.2406
 3/30 [==>...........................] - ETA: 42s - loss: -0.2410

FWIW, I'm using multi_gpu_model in Keras.

when using TensorBoard and Keras we double report metrics and plot them on chart:

Not sure if I should report this problem here, but on Comet.ml FAQ webpage there's an inconsistent use of "?".

For example, there are questions like "Would Comet slow down my model’s training?" and "Is this only for deep learning?" where there's no space between the word and the question mark.

However, there are other questions like "Does work with library X ?" and "Do you provide an on-premise installation of comet.ml ?" where it does.

Image example:

This is just a silly problem, but it calls my attention.

If this was intentional, you can ignore this issue. Thanks!

experiment seems to disconnect from Comet after a while.

It says:

“Could not connect to server after multiple tries.” 

The experiment appears as finished (checkmark) on Comet.ml experiment list, but it's still training on our server

when viewing the chart while running the experiment,
sometimes, only results from within a certain range is shown (say epoch 20 onwards).
When the experiment is done the full chart can be loaded and viewed

The following code snippet randomly generates a Validation Error.

CODE:

from comet_ml import Experiment, Optimizer

experiment = Experiment(api_key=API_KEY,
                        auto_param_logging=False, auto_metric_logging=False)
experiment.disabled_monkey_patching = False
optimizer = Optimizer(api_key=API_KEY)

params = """
lstm_layers_1 integer [1, 8] [8]
lstm_layers_2 integer [1, 4] [4]
lstm_units integer [32, 512] [512]
lr real [0.0001, 0.02] [0.002]
embedding integer [1, 256] [32]
activation integer [0, 2] [1]
optimizer integer [1, 4] [2]
batch_size integer [1, 128] [128]
filters integer [16, 128] [64]
kernel integer [3, 10] [10]
units integer [32, 256] [128]
"""
#optimizer.set_params_file(pcs_file)
optimizer.set_params(params)

OUTPUT:

Traceback (most recent call last):
  File "Desktop/MaterialProposal/Elementor_AutoML.py", line 31, in <module>
    optimizer.set_params(params)
  File "/home/christianlagares/anaconda3/lib/python3.5/site-packages/comet_ml/__init__.py", line 959, in set_params
    self._send_params(pcs_content)
  File "/home/christianlagares/anaconda3/lib/python3.5/site-packages/comet_ml/__init__.py", line 990, in _send_params
    six.raise_from(ValidationError(error), None) 
  File "<string>", line 3, in raise_from
comet_ml.exceptions.ValidationError: Invalid PCS configuration file, please fix it and retry.
COMET INFO: Uploading stats to Comet before program termination (may take several seconds)

Worth noting that the issue applies to variations (different defaults or ranges) of the same PCS input.

System Details:

• Ubuntu 16.04
• Tensorflow 1.4
• CUDA 8.0
• cuDNN 6.0
• Python 3.5
• Comet 1.0.8

Hello, I am using this example from the docs https://www.comet.ml/docs/python-sdk/keras/#end-to-end-example
But it shows nothing after the completion on the above said tabs. Hyper-parameters, Metrics, and Installed Packages are shown correctly.

I am using:
Python 3.5.3
Anaconda 4.4.0 (64-bit)
IPython 5.3.0

As of this morning, it looks like the output tab of experiments has been changed to a black background. That's all well and good, however, when monitoring a live experiment, the screen flashes very quickly, making it unreadable. Attached is a quick video showing the issue.

First suggested by Dany R.

Ability to recover experiments if you delete them (undo/archive). Could entail have a trash/archive section for experiments.

I log my experiments from a pod running on a cluster managed by kubernetes. I get the following error just before the dashboard says experiment ended. However, my training script is still running. I followed the guide to log metrics and I don't think there is a problem in that. Could you help me figure this out?

COMET ERROR: Streaming error Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/comet_ml/comet.py", line 117, in _loop self.ws_connection.send(messages) File "/usr/local/lib/python3.5/dist-packages/comet_ml/connection.py", line 336, in send self._send(data) File "/usr/local/lib/python3.5/dist-packages/comet_ml/connection.py", line 364, in _send self.wait_for_connection() File "/usr/local/lib/python3.5/dist-packages/comet_ml/connection.py", line 384, in wait_for_connection "Could not connect to server after multiple tries. ") ValueError: Could not connect to server after multiple tries.

I don't know if this is just for me, but whenever I have the Code, Graph Definition, or Installed Packages tabs open, scrolling lags quite a bit - not only scrolling the code itself, but also scrolling the entire web page if the code box is visible. This stops being a problem as soon as the code box is out of view. No bug here, obviously, but it would be a nice enhancement if this part of the interface were more fluid. Right now I just click on View Raw when looking at code/graph defs; since this workaround already exists, this isn't high priority for me, but I thought I'd raise the issue (and at least see if others have this as well).

First reported via Intercom for the FAQ page (https://www.comet.ml/faq)

1. "Does it work with library (including it)
2. I found a bug I'd like to report". (capitalizing the i)

See the screenshot:

Hi, thank you for developing a nice product.

In comet.ml, it uses 0.0001 like expression in dashboard.
And if the number is too small, it is rounded in some precision.
However, we sometimes need the expression with any precision like 1e-4.
Could you consider adding this expression as an option?

Thanks.

When using keras.utils.multi_gpu_model, Comet raises the following exception:

  File ".../lib/python3.6/site-packages/comet_ml/keras_logger.py", line 32, in wrapper
    return real_fit(*args, **kwargs)
  File ".../lib/python3.6/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
    return func(*args, **kwargs)
  File ".../lib/python3.6/site-packages/keras/engine/training.py", line 2141, in fit_generator
    callbacks.on_train_begin()
  File ".../lib/python3.6/site-packages/keras/callbacks.py", line 130, in on_train_begin
    callback.on_train_begin(logs)
  File ".../lib/python3.6/site-packages/comet_ml/frameworks.py", line 95, in on_train_begin
    self.experiment.set_model_graph(self.model.to_json())
  File ".../lib/python3.6/site-packages/keras/engine/topology.py", line 2699, in to_json
    model_config = self._updated_config()
  File ".../lib/python3.6/site-packages/keras/engine/topology.py", line 2666, in _updated_config
    config = self.get_config()
  File ".../lib/python3.6/site-packages/keras/engine/topology.py", line 2421, in get_config
    return copy.deepcopy(config)
  File ".../lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File ".../lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File ".../lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File ".../lib/python3.6/copy.py", line 215, in _deepcopy_list
    append(deepcopy(a, memo))
  File ".../lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File ".../lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File ".../lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File ".../lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File ".../lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File ".../lib/python3.6/copy.py", line 220, in _deepcopy_tuple
    y = [deepcopy(a, memo) for a in x]
  File ".../lib/python3.6/copy.py", line 220, in <listcomp>
    y = [deepcopy(a, memo) for a in x]
  File ".../lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File ".../lib/python3.6/copy.py", line 220, in _deepcopy_tuple
    y = [deepcopy(a, memo) for a in x]
  File ".../lib/python3.6/copy.py", line 220, in <listcomp>
    y = [deepcopy(a, memo) for a in x]
  File ".../lib/python3.6/copy.py", line 169, in deepcopy
    rv = reductor(4)
TypeError: can't pickle module objects

As stated in Keras docs:

To save the multi-gpu model, use .save(fname) or .save_weights(fname) with the template model (the argument you passed to multi_gpu_model), rather than the model returned by multi_gpu_model.

I suppose Comet should either allow for disabling graph saving or detect when multi_gpu_model is called and use its first argument as the graph source.

in experiment table:

• filter
• select all
• all experiment on page are selected
• expected only the filtered experiment to select.

Currently, experiments do not have any 'name' property. So, it's harder to keep track of which experiment is which. A name property would allow naming with the associated hyperparameters.

Hello,

Reading comet.ml Python client code, I noticed that environment variable COMET_API_KEY can be defined to set API key (instead of passing as parameter for Experiment).
Maybe it should be add in https://www.comet.ml/docs/python-sdk/getting-started/

Kind regards

HI,

When I tried to use Comet.ml, I got the following error with Keras. Any suggestions on why?

how to reproduce:

• create empty repo (can have a readme file)
• linked comet project with the empty repo
• create comet experiment
• issue a PR from view experiment page
• PR failed.

temporary fix is to move training files inside a folder with repo name and re-run experiment
long term fix is to support empty repo for pull request

Hello. I am a comet.ml user in Japan(It is awesome products. Thank you.)

When I trying to type multi-byte strings(japanese), these strings were not saved correctly.

(before reload)

(after reload)

I think this cause may be that strings is not saved in utf-8 encodings.

Thank you.

Hello!

After experimenting with Comet, I noticed every time I run a model, the data related to the model is incomplete.

Here's an example:

(EDIT): Output tab is incomplete too:

Here my model:

import numpy as np
from comet_ml import Experiment 
from keras.layers import Dense
from keras.models import Sequential

experiment = Experiment(api_key="REDACTED", project_name="mac")

X = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]).reshape(-1, 1)
y = np.array([2, 3, 4, 5, 6, 7, 8, 9, 10, 11]).reshape(-1, 1)

model = Sequential()
model.add(Dense(1, input_shape=(1,)))

model.compile(loss='mse', optimizer='adam')
model.fit(X, y, validation_split=0.2, epochs=10000)

You may notice that my model runs until epoch number 10000, but in the dashboard, it stops at epoch 1592. I've tried to use a smaller number of epochs like 100 for example, but I still have the same issue (it stops working close to the 80th epoch).

I noticed that after my train ends, it shows an error:

[Errno 9] Bad file descriptor

And sometimes it just throws a fatal python error:

Fatal Python error: could not acquire lock for <_io.BufferedWriter name='<stdout>'> at interpreter shutdown, possibly due to daemon threads

Thread 0x000070000278b000 (most recent call first):
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 642 in write
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 941 in send
  File "/usr/local/lib/python3.6/site-packages/websocket/_socket.py", line 117 in send
  File "/usr/local/lib/python3.6/site-packages/websocket/_core.py", line 430 in _send
  File "/usr/local/lib/python3.6/site-packages/websocket/_core.py", line 265 in send_frame
  File "/usr/local/lib/python3.6/site-packages/websocket/_core.py", line 240 in send
  File "/usr/local/lib/python3.6/site-packages/websocket/_app.py", line 147 in send
  File "/usr/local/lib/python3.6/site-packages/comet_ml/connection.py", line 282 in _send
  File "/usr/local/lib/python3.6/site-packages/comet_ml/connection.py", line 258 in send
  File "/usr/local/lib/python3.6/site-packages/comet_ml/comet.py", line 92 in run
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x0000700001d85000 (most recent call first):
  File "/usr/local/lib/python3.6/site-packages/comet_ml/connection.py", line 331 in on_error
  File "/usr/local/lib/python3.6/site-packages/websocket/_app.py", line 315 in _callback
  File "/usr/local/lib/python3.6/site-packages/websocket/_app.py", line 280 in run_forever
  File "/usr/local/lib/python3.6/site-packages/comet_ml/connection.py", line 248 in run
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 884 in _bootstrap

Current thread 0x00007fff9752c380 (most recent call first):
[1]    6612 abort      python3 mac.py

I tried to use comet_ml in a Linux Docker container and still had incomplete data, but these errors didn't occur.

My setup is:
macOS 10.13.4
Python 3.6.5
Keras 2.1.5
Tensorflow 1.7.0
Comet_ML 1.0.2

Thanks!

For any Keras model, if it has callbacks, Comet registers only the first epoch.

If GradientBoostingRegressor is used in a project, multiple times "<mtrand.RandomState object> is not JSON serializable" error will be printed in the output.

OS - mac & windows
IDE - pyCharm, .py file, not jupyter notebok

code:

from sklearn.datasets import load_boston
from sklearn.model_selection import train_test_split
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.preprocessing import StandardScaler
random_state = 42

boston = load_boston()

X_train, X_test, y_train, y_test = train_test_split(
boston.data,
boston.target,
random_state=random_state)

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

model = GradientBoostingRegressor()
model.fit(X_train_scaled, y_train)
y_pred = model.predict(X_test_scaled)

output:

Experiment is live on

<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable
<mtrand.RandomState object at 0x107dfb500> is not JSON serializable

Uploading stats to Comet before program termination (may take several seconds)

Experiment is live on comet.ml:

Process finished with exit code 0

This is a feature request. It would be useful if nested dictionaries were supported as parameters, e.g.

params = {
   'model_params': {
       'input_size': [32, 32, 3],
       'filters': 4
       }
   'training_params': {
       'batch_size': 16,
       'lr': 3e-4
   }
}

Currently the lowest level of the dictionary is lost, and only the keys are saved in Comet.ml dashboard, e.g. model_params = ['input_size', 'filters'].

In my current workflow, I rerun my training script when I make changes to it. However, my code isn't perfect and things break during run-time (e.g.: I mistype the key of a pandas column). On these runs, comet still records an experiment and I have to go into the console and delete the incomplete run.

Could a flag be added to the Experiment object to silence all logging to the comet.ml API? This would make development of scripts much easier as the flag could be removed once the script is in good shape. I understand there's individual flags for some of the other logs, but prevent the creation of runs entirely would be the goal of this flag.

Perhaps this is too picky, but it would be nice to make better use of the chart space.

Here is a on-going run

Most charts will almost always be wide, as we all run lots of steps and tend to have basically 0-1 axis. In this case, it would make more sense to put the widget below the chart. There is alot of white space there, and it would allow one to see a larger set of data.

Other suggestions

1. Dramatically increase the font size of the selector hover box. 2x size. Probably of the axis font too.
2. A way to quickly see the current values of the metrics without hovering. Hovering doesn't work for very long if you have a model that's running. As the chart reloads data your hover box disappears. Especially if you have fast batches in a model, it can be challenging to actually see the values as it flicks on and off. One solution would be to have a widget in the white space below just reporting the latest values of the metrics.

Lambda layers in Keras calling non JSON serializable code is causing an issue. Is it possible to add an argument to disable set_model_graph?

I am interested in using comet.ml for my projects. I have projects in Java and Python. Is it possible to use Java code with Comet ?
Do you have any documentation ? I can only find with python.

When I try to run https://www.comet.ml/docs/python-sdk/matplotlib/#end-to-end-example I get following error on windows:

LINES [<matplotlib.lines.Line2D object at 0x00000233778BDD30>, <matplotlib.lines.Line2D object at 0x00000233778BDEF0>]
Traceback (most recent call last):
File "C:/Users/Danilo/PycharmProjects/Kaggle/house-price-prediction.py", line 151, in
experiment.log_figure(plt)
File "C:\Users\Danilo\AppData\Local\Programs\Python\Python35\lib\site-packages\comet_ml_init_.py", line 530, in log_figure
url_params)
File "C:\Users\Danilo\AppData\Local\Programs\Python\Python35\lib\site-packages\comet_ml\file_uploader.py", line 210, in upload_file_process
p.start()
File "C:\Users\Danilo\AppData\Local\Programs\Python\Python35\lib\multiprocessing\process.py", line 105, in start
self._popen = self._Popen(self)
File "C:\Users\Danilo\AppData\Local\Programs\Python\Python35\lib\multiprocessing\context.py", line 212, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Users\Danilo\AppData\Local\Programs\Python\Python35\lib\multiprocessing\context.py", line 313, in _Popen
return Popen(process_obj)
File "C:\Users\Danilo\AppData\Local\Programs\Python\Python35\lib\multiprocessing\popen_spawn_win32.py", line 66, in init
reduction.dump(process_obj, to_child)
File "C:\Users\Danilo\AppData\Local\Programs\Python\Python35\lib\multiprocessing\reduction.py", line 59, in dump
ForkingPickler(file, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <class 'module'>: attribute lookup module on builtins failed

On MAC upload finishes, but graph is blank on website

While running my not too complex Keras model experiment, I get the following error:

comet.ml error: failed to report experiment status

After the exceptions:

socket.timeout: The read operation timed out

[...]

urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='www.comet.ml', port=443): Read timed out. (read timeout=10)

[...]

requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='www.comet.ml', port=443): Read timed out. (read timeout=10)

It subsequently uploads data to Comet.ml, and I can see the experiment in the Projects page. I'm not sure of what information, if any, is missing.

Is it a temporary service issue, a mis-configuration of my setup, or a bug in the comet_ml module?

Hello, first of a couple issues i'll post after working with comet for a few days. I'm definitely really happy with the tool.

It would be very useful to export the data back out of the project page. For example, if i wanted to know how my average accuracy was changing across experiments. Or compare the effect of a given hyperparameters on performance, I'd want to select multiple experiments and dump the data to file/json. Then I could go and plot those data in Python/R. It would be extra nice to make this a method in the comet_ml API, since it already has our key authentication. The "compare experiments" idea is getting there, but exporting the data would allow a user to balance a quick visual check of a run versus more in-depth graphing that would be too cumbersome to customize in the browser.

Let me know if this kind of feedback is better in-channel.

I'm currently unable to view the hyperparameter tab in any of my experiments, even those that are not running. It just loads indefinitely, this makes it pretty hard to use comet.

I am getting a connection error when uploading my experiment to comet.ml. Minimal working example below:

from comet_ml import Experiment
experiment = Experiment(api_key="my_api_key", project_name="my_project_name", team_name="my_team_name", log_code=False)

# (data loading)

# Setting up hyperparameter grid
param_grid =  {'n_estimators': [10, 50, 100, 500, 1000],
                'max_features': [5, 15, 50],
                'max_depth': [1, 3, 5],
                'min_samples_split': [2, 10]
                }


# Setting up the searcher
clf = RandomForestClassifier(random_state=52)
searcher = RandomizedSearchCV(clf, param_grid, scoring=['accuracy', 'precision', 'recall', 'f1'], cv=N_FOLDS, refit='f1', verbose=True, n_jobs=-1) # f1 is the decision scorer
searcher.fit(train_data_win, train_labels_win)

I am always getting back the following error:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/comet_ml/__init__.py", line 296, in _authenticate
    self.project_name, self.team_name
  File "/usr/local/lib/python3.6/site-packages/comet_ml/connection.py", line 140, in get_run_id
    raise ValueError(r.content)
ValueError: b'{"msg":"unexpected error","version":"9beed354d5345e881864ad8f3d7e527d73795dca","code":500,"data":null}'
 
 Failed to establish connection to Comet server. Please check your internet connection. Your experiment would not be logged

Does it have something to do with the use of RandomizedSearchCV or GridSearchCV? Thanks!

getting error in sdk:

COMET ERROR: File could not be uploaded
Traceback (most recent call last):
  File "/anaconda/lib/python3.6/site-packages/comet_ml/file_uploader.py", line 220, in upload_file
    additional_params,
  File "/anaconda/lib/python3.6/site-packages/comet_ml/file_uploader.py", line 122, in send_file
    _send_file(post_endpoint, params=params, files=files)
  File "/anaconda/lib/python3.6/site-packages/comet_ml/file_uploader.py", line 107, in _send_file
    raise ValueError("POSTing file failed: %s" % r.content)
ValueError: POSTing file failed: b'<html>\r\n<head><title>413 Request Entity Too Large</title></head>\r\n<body bgcolor="white">\r\n<center><h1>413 Request Entity Too Large</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n'

• reproducing on py2,py3
• Failed to read source code file from disk: <ipython-input-2-decb1e63997f>

A role for team admins, so that the first person to create the organization can delegate permissions to add members, etc. to someone else

Experiment is uploaded twice, but graphics not at all.

The exception I'm getting is:

RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.

OS - windows 10
python 3.5
IDE - pyCharm, .py file, not jupyter NB

code:

import matplotlib as mpl
mpl.use('TkAgg')
import matplotlib.pyplot as plt
import numpy as np

t = np.arange(0.0, 2.0, 0.01)
s = 1 + np.sin(2np.pit)
plt.plot(t, s)

plt.xlabel('time (s)')
plt.ylabel('voltage (mV)')
plt.title('About as simple as it gets, folks')
plt.grid(True)

experiment.log_figure(figure=plt)

output:

Experiment is live on comet.ml https://www.comet.ml/danilo/matplotlib/ea2bf8c69f4b40ada4158c8869961bc1

LINES [<matplotlib.lines.Line2D object at 0x0000016DB2361B00>]

Experiment is live on comet.ml https://www.comet.ml/danilo/matplotlib/fa4b15da2dcc48a7878cf165be1bdead

LINES [<matplotlib.lines.Line2D object at 0x000002087ED8E9E8>]
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\Danilo\AppData\Local\Programs\Python\Python35\lib\multiprocessing\spawn.py", line 106, in spawn_main
exitcode = _main(fd)
File "C:\Users\Danilo\AppData\Local\Programs\Python\Python35\lib\multiprocessing\spawn.py", line 115, in _main
prepare(preparation_data)
File "C:\Users\Danilo\AppData\Local\Programs\Python\Python35\lib\multiprocessing\spawn.py", line 226, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\Danilo\AppData\Local\Programs\Python\Python35\lib\multiprocessing\spawn.py", line 278, in _fixup_main_from_path
run_name="mp_main")
File "C:\Users\Danilo\AppData\Local\Programs\Python\Python35\lib\runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "C:\Users\Danilo\AppData\Local\Programs\Python\Python35\lib\runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "C:\Users\Danilo\AppData\Local\Programs\Python\Python35\lib\runpy.py", line 85, in run_code
exec(code, run_globals)
File "C:\Users\Danilo\PycharmProjects\Kaggle\test.py", line 18, in
experiment.log_figure(figure=plt)
File "C:\Users\Danilo\AppData\Local\Programs\Python\Python35\lib\site-packages\comet_ml_init.py", line 530, in log_figure
url_params)
File "C:\Users\Danilo\AppData\Local\Programs\Python\Python35\lib\site-packages\comet_ml\file_uploader.py", line 210, in upload_file_process
p.start()
File "C:\Users\Danilo\AppData\Local\Programs\Python\Python35\lib\multiprocessing\process.py", line 105, in start
self._popen = self._Popen(self)
File "C:\Users\Danilo\AppData\Local\Programs\Python\Python35\lib\multiprocessing\context.py", line 212, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Users\Danilo\AppData\Local\Programs\Python\Python35\lib\multiprocessing\context.py", line 313, in _Popen
return Popen(process_obj)
File "C:\Users\Danilo\AppData\Local\Programs\Python\Python35\lib\multiprocessing\popen_spawn_win32.py", line 34, in init
prep_data = spawn.get_preparation_data(process_obj._name)
File "C:\Users\Danilo\AppData\Local\Programs\Python\Python35\lib\multiprocessing\spawn.py", line 144, in get_preparation_data
_check_not_importing_main()
File "C:\Users\Danilo\AppData\Local\Programs\Python\Python35\lib\multiprocessing\spawn.py", line 137, in _check_not_importing_main
is not going to be frozen to produce an executable.''')
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.

Uploading stats to Comet before program termination (may take several seconds)

Experiment is live on comet.ml: https://www.comet.ml/danilo/matplotlib/fa4b15da2dcc48a7878cf165be1bdead
close status: 26479

Uploading stats to Comet before program termination (may take several seconds)

Experiment is live on comet.ml: https://www.comet.ml/danilo/matplotlib/ea2bf8c69f4b40ada4158c8869961bc1
close status: 26479

Process finished with exit code 0