cta-observatory / ctapipe Goto Github PK

Right now, the CameraDisplay is a generic class that uses a backend renderer (from a class hierarchy). Is there a nicer way to support multiple backends?

@kosack – I've moved this example from a __main__ section in the ctapipe.reco code to the docs
https://github.com/cta-observatory/ctapipe/blob/master/docs/reco/shower_example.py
https://cta-observatory.github.io/ctapipe/reco/index.html#getting-started

I think it's outdated though and doesn't work?
Should it be replaced with one of the examples in the examples folder or should it be fixed?

Generally it would be good to have some working examples in the docs and not just in the examples folder.

This will drastically reduce time and memory usage.

Give the telescope table a column camera id and then just create tables for each camera, not for each telescope.

See perhaps here on how to add an C-file + swig wrapper using setup.py's Extension module:
https://scipy-lectures.github.io/advanced/interfacing_with_c/interfacing_with_c.html

Make example of an algorithm implemented in c as an extension. This is not quite the same thing as including a C library in the package (like libeventio), which should perhaps be done via a conda package or similar...

There are some data items that should be stored along with the repository, for example files containing camera geometries and array layouts. In the future those would be read directly from the raw data files, but there should be "default" versions included to make it possible to debug code without loading raw data. Where should this go? The format and naming wil. be submitted as a redmine issue for the MODEL group.

perhaps to start with:

  data/ 
     camera_geometry/
         hess_v0.0_camerageom.fits.gz
         chec_v0.0_camerageom.fits.gz
     array_geometry/
         hess_v0.0_arraygeom.fits.gz
     ...

I don't manage to load the module hessio. I have already included pyhessioxxx in my PYTHONPATH directory, yet I keep getting this message:

from ctapipe.io.hessio import hessio_event_source
the hessio python module is required to access MC data: No module named 'hessio'
Traceback (most recent call last):
File "", line 1, in
File "/nfs/cta-ifae/moralejo/CTA/Pipeline/ctapipe/ctapipe/io/hessio.py", line 19, in
raise err
File "/nfs/cta-ifae/moralejo/CTA/Pipeline/ctapipe/ctapipe/io/hessio.py", line 15, in

As discussed today, I plan to set up a test dataset repo and way to use the test data the way the Sherpa devs did:

• https://github.com/sherpa/sherpa-test-data
• https://github.com/sherpa/sherpa

@kosack - Maybe you could look over the details and their reasoning to set it up exactly like this in sherpa/sherpa#31? If no, that's OK ... I'll just try it out tomorrow and we'll see if we like it.

pyhessio installation instruction is incorrect in docs/getting_started/index.rst
Correct instruction is:
conda install --use-local pyhessio
Instead of:
conda install pyhessio

When plotting a GCT camera image there are unexplained white lines on the pixels (attached). This seems to also happen with other square-pixel cameras.

Is this due to the pixel positions? Camera curvature?

@kosack I see you've added a hillas_parameters_2 function in ctapipe.reco.

I guess it's purpose is to use Hillas parameter computation as an example for speed comparisons / teaching and we'll add C and Numba versions?
So I'm +1 to keep it.

But can you make the returned parameters consistent between the different functions?
It might also make sense to return a HillasParameters object.
One advantage would be that it then shows up in the docs and we can document it's parameters.
Another advantage is that we can add properties and convenience methods to lazily compute only things the user needs.

Could you explain what the channels mean for the cameras?
These are not the pixels, right? Is it lowgain / highgain?

Parts of the calib package are not python3 compatible.

For example in ctapipe/calib/camera/mycam.py

 if (len(sys.argv) < 1):
        print TAG, "Usage %s " % sys.argv[0]
        sys.exit(0)

Is missing the paratheses for the call to the print function.

There is a TypeError error when trying to display an event. see error log bellow

(root)jacquem>python read_hessio.py
/home/jacquem/workspace/CTA/cta-pipe/ctapipe/ctapipe-extra/datasets/gamma_test.simtel.gz

Input file '/home/jacquem/workspace/CTA/cta-pipe/ctapipe/ctapipe-extra/datasets/gamma_test.simtel.gz' has been opened.
I/O block extended by 744748 to 1744748 bytes

Starting run 31964
EVENT_ID: 408 TELS: [38 47]
============================================
n or [enter] - go to Next event
d - Display the event
p - Print all event data
i - event Info
q - Quit
Choice: d
Displaying... please wait (this is an inefficient implementation)
draw cam 38...
Traceback (most recent call last):
File "read_hessio.py", line 88, in
display_event(event)
File "read_hessio.py", line 53, in display_event
signals -= signals.mean()
TypeError: Cannot cast ufunc subtract output from dtype('float64') to dtype('int32') with casting rule 'same_kind'

gzip: stdout: Broken pipe

The hessio_event_source reuses the same container again and again, this
has the strange side effect, that you cannot get a list of events from it:

from ctapipe.io.hessio import hessio_event_source

events = list(hessio_event_source('path/to/file', max_events=10))

This gives you 10 references to the same object, the last read event.

I changed the calibration code, so that it goes faster. It took 0.2 sec to run the nb_peak_integration_mc function.

I merged the ctapipe master branch into my repository and only asked me to include the focal length as a parameter to the CameraGeometry.guess function.
After that the function now takes 1 second to run.
Timing the different parts of the code I found that calling the CameraGeometry.gues function takes 0.8 seconds, while it takes 0.2 seconds for the rest of the code.
Is it a feature or a bug?

Using

from ctapipe.io.hessio import hessio_event_source
from ctapipe.instrument.InstrumentDescription import read_hessio

filename = 'ctapipe-extras/datasets/gamma_test.simtel.gz'

telescopes, cameras, optics = read_hessio(filename)
events = list(hessio_event_source(filename, max_events=10))

I get a segfault, doing it the other way around works:

events = list(hessio_event_source(filename, max_events=10))
telescopes, cameras, optics = read_hessio(filename)

right now, we only have a few example "tools" (executable programs), that eventually need to have a common interface. A good place to start is to migrate them to use the configuration class added recently by Jean. The only tools that are there so far are ctapipe-info (used to list what tools are available as well as other version info), and ctapipe-camdemo (an example tool). Both do some command-line handling that can be migrated.

@kosack – Tests are failing at the moment:
https://gist.github.com/cdeil/6480edb512002e449317#file-gistfile1-txt-L579

Can you please add the missing file to the ctapipe-extra repo?

I'm starting to implement some of the data model group's concept for an ObsConfig, which is a class the contains a hierarchy of the description of the array layout, telescope optics, camera layout, etc. That is independent of the "configuration service" (which handles the parameters of the algorithms). I am also looking into an implementation of the provenance data (which is actually quite closely connected to the configuration service, and is maybe the same thing).

Where should those go? (right now it's in ctapipe.io, but that doesn't really make sense, since that should really just include data readers/writers).

Should we create a module like ctapipe.core or ctapipe.config?

I want to avoid 100s of sub-modules , but there should be a somewhat clear hierarchy.

In my opinion having classes for the instrument description might be a nicer interface.

The different loading and writing functions could then be class methods and be used like this:

from ctapipe.instrument import InstrumentDescriptiion

instrument = InstrumentDescription.from_hessio('/path/to/file')
instrument.to_fits('instrument.fits')

Make CameraGeometry and ArrayLayout subclasses of astropy.table.Table?

Right now we have 2 instrumental data classes: CameraGeometry and ArrayLayout that will eventually be merged into a comprehensive instrument model. However, their implementation leaves a lot to be desired.

Currently, CameraGeometry is simple a class with members like pix_x, pix_y that should contain an array of pixel positions. There is separate support to read them from various formats.

There is a common use case (working with cleaned images) that is made difficult by this implementation. After an image is cleaned, one has a boolean mask that can be applied to the image data. To apply that also to the pixel positions requires masking each member of CameraGeometry manually:

geom = CameraGeometry(...)
image = ...
mask = tailcuts_clean(geom, image)

cleaned_image = image[mask]
cleaned_x = CameraGeometry.pix_x[mask]
cleaned_y = CameraGeometry.pix_y[mask]

etc.

It would be nicer to allow slicing on the whole CameraGeometry object, but currently it contains both vector quantities and scalar quantities like pix_type.

cleaned_image = image[mask]
cleaned_geom = geom[mask]

One suggestion: use astropy.table.Table as the base class for all instrument classes.

• Advantages
  • get automatic "slicing" of all data (as above)
  • get automatic read/write to all table formats (so data model is directly connected to the class)
  • get nice print-outs (pretty-printed columns, html in notebook mode)
  • can "browse" the data structures in a web interface
  • could extend to add more reader/writer routines like the current classes
• Disadvantages:
  • can access members by "dot" syntax (would need geom['pix_x'] rather thangeom.pix_x`)
  • scalar quantities would need to be in the metadata (which is probably correct anyhow), e.g. geom.meta.pix_type = 'hexagonal'

If implemented this way, you can do:

geom = CameraGeometry.from_file("test.simtel.gz",5)
geom.write("CT5.fits")

from ctapipe.instrument.InstrumentDescription import load_hessio
from telescopes, cameras, optics = load_hessio('/media/Max1TB/gamma_20deg_0deg_run10251___cta-prod3-merged_desert-2150m-Paranal-subarray-3_cone10.simtel.gz')

camera = cameras['CameraTable_VersionFeb2016_TelID9']

This results in

IndexError                                Traceback (most recent call last)
/home/maxnoe/.local/anaconda3/envs/ctapipe/lib/python3.5/site-packages/IPython/core/formatters.py in __call__(self, obj)
    697                 type_pprinters=self.type_printers,
    698                 deferred_pprinters=self.deferred_printers)
--> 699             printer.pretty(obj)
    700             printer.flush()
    701             return stream.getvalue()

/home/maxnoe/.local/anaconda3/envs/ctapipe/lib/python3.5/site-packages/IPython/lib/pretty.py in pretty(self, obj)
    381                             if callable(meth):
    382                                 return meth(obj, self, cycle)
--> 383             return _default_pprint(obj, self, cycle)
    384         finally:
    385             self.end_group()

/home/maxnoe/.local/anaconda3/envs/ctapipe/lib/python3.5/site-packages/IPython/lib/pretty.py in _default_pprint(obj, p, cycle)
    501     if _safe_getattr(klass, '__repr__', None) not in _baseclass_reprs:
    502         # A user-provided repr. Find newlines and replace them with p.break_()
--> 503         _repr_pprint(obj, p, cycle)
    504         return
    505     p.begin_group(1, '<')

/home/maxnoe/.local/anaconda3/envs/ctapipe/lib/python3.5/site-packages/IPython/lib/pretty.py in _repr_pprint(obj, p, cycle)
    692     """A pprint that just redirects to the normal repr function."""
    693     # Find newlines and replace them with p.break_()
--> 694     output = repr(obj)
    695     for idx,output_line in enumerate(output.splitlines()):
    696         if idx:

/home/maxnoe/.local/anaconda3/envs/ctapipe/lib/python3.5/site-packages/astropy/table/table.py in __repr__(self)
    771 
    772     def __repr__(self):
--> 773         return self._base_repr_(html=False, max_width=None)
    774 
    775     def __unicode__(self):

/home/maxnoe/.local/anaconda3/envs/ctapipe/lib/python3.5/site-packages/astropy/table/table.py in _base_repr_(self, html, descr_vals, max_width, tableid, show_dtype, max_lines, tableclass)
    759             self, tableid=tableid, html=html, max_width=max_width,
    760             show_name=True, show_unit=None, show_dtype=show_dtype,
--> 761             max_lines=max_lines, tableclass=tableclass)
    762 
    763         out = descr + '\n'.join(data_lines)

/home/maxnoe/.local/anaconda3/envs/ctapipe/lib/python3.5/site-packages/astropy/table/pprint.py in _pformat_table(self, table, max_lines, max_width, show_name, show_unit, show_dtype, html, tableid, tableclass, align)
    494             lines, outs = self._pformat_col(col, max_lines, show_name=show_name,
    495                                             show_unit=show_unit, show_dtype=show_dtype,
--> 496                                             align=align_)
    497             if outs['show_length']:
    498                 lines = lines[:-1]

/home/maxnoe/.local/anaconda3/envs/ctapipe/lib/python3.5/site-packages/astropy/table/pprint.py in _pformat_col(self, col, max_lines, show_name, show_unit, show_dtype, show_length, html, align)
    240                                                show_length=show_length,
    241                                                outs=outs)
--> 242         col_strs = list(col_strs_iter)
    243         if len(col_strs) > 0:
    244             col_width = max(len(x) for x in col_strs)

/home/maxnoe/.local/anaconda3/envs/ctapipe/lib/python3.5/site-packages/astropy/table/pprint.py in _pformat_col_iter(self, col, max_lines, show_name, show_unit, outs, show_dtype, show_length)
    405                     else:
    406                         col_str = (format_func(col_format, col[(i,) + multidim0]) +
--> 407                                   ' .. ' +
    408                                   format_func(col_format, col[(i,) + multidim1]))
    409                 else:

astropy/table/_column_mixins.pyx in astropy.table._column_mixins._ColumnGetitemShim.__getitem__ (astropy/table/_column_mixins.c:900)()

astropy/table/_column_mixins.pyx in astropy.table._column_mixins.base_getitem (astropy/table/_column_mixins.c:786)()

astropy/table/_column_mixins.pyx in astropy.table._column_mixins.column_getitem (astropy/table/_column_mixins.c:837)()

IndexError: index 0 is out of bounds for axis 1 with size 0

These are all failing telescopes:

CameraTable_VersionFeb2016_TelID8
CameraTable_VersionFeb2016_TelID9
CameraTable_VersionFeb2016_TelID10
CameraTable_VersionFeb2016_TelID11
CameraTable_VersionFeb2016_TelID12
CameraTable_VersionFeb2016_TelID13
CameraTable_VersionFeb2016_TelID14
CameraTable_VersionFeb2016_TelID15
CameraTable_VersionFeb2016_TelID16
CameraTable_VersionFeb2016_TelID17
CameraTable_VersionFeb2016_TelID18
CameraTable_VersionFeb2016_TelID19
CameraTable_VersionFeb2016_TelID20
CameraTable_VersionFeb2016_TelID21
CameraTable_VersionFeb2016_TelID22
CameraTable_VersionFeb2016_TelID23
CameraTable_VersionFeb2016_TelID24
CameraTable_VersionFeb2016_TelID25
CameraTable_VersionFeb2016_TelID26
CameraTable_VersionFeb2016_TelID27
CameraTable_VersionFeb2016_TelID28
CameraTable_VersionFeb2016_TelID29
CameraTable_VersionFeb2016_TelID30
CameraTable_VersionFeb2016_TelID31
CameraTable_VersionFeb2016_TelID32
CameraTable_VersionFeb2016_TelID33
CameraTable_VersionFeb2016_TelID34
CameraTable_VersionFeb2016_TelID35
CameraTable_VersionFeb2016_TelID36
CameraTable_VersionFeb2016_TelID37
CameraTable_VersionFeb2016_TelID38
CameraTable_VersionFeb2016_TelID39
CameraTable_VersionFeb2016_TelID40
CameraTable_VersionFeb2016_TelID41
CameraTable_VersionFeb2016_TelID42
CameraTable_VersionFeb2016_TelID43
CameraTable_VersionFeb2016_TelID44
CameraTable_VersionFeb2016_TelID45
CameraTable_VersionFeb2016_TelID46
CameraTable_VersionFeb2016_TelID47
CameraTable_VersionFeb2016_TelID48
CameraTable_VersionFeb2016_TelID49
CameraTable_VersionFeb2016_TelID50
CameraTable_VersionFeb2016_TelID51
CameraTable_VersionFeb2016_TelID52
CameraTable_VersionFeb2016_TelID53
CameraTable_VersionFeb2016_TelID54
CameraTable_VersionFeb2016_TelID55
CameraTable_VersionFeb2016_TelID56
CameraTable_VersionFeb2016_TelID57
CameraTable_VersionFeb2016_TelID58
CameraTable_VersionFeb2016_TelID59
CameraTable_VersionFeb2016_TelID60
CameraTable_VersionFeb2016_TelID61
CameraTable_VersionFeb2016_TelID62
CameraTable_VersionFeb2016_TelID63
CameraTable_VersionFeb2016_TelID64
CameraTable_VersionFeb2016_TelID65
CameraTable_VersionFeb2016_TelID66
CameraTable_VersionFeb2016_TelID67
CameraTable_VersionFeb2016_TelID68
CameraTable_VersionFeb2016_TelID69
CameraTable_VersionFeb2016_TelID70
CameraTable_VersionFeb2016_TelID424
CameraTable_VersionFeb2016_TelID425
CameraTable_VersionFeb2016_TelID426
CameraTable_VersionFeb2016_TelID427
CameraTable_VersionFeb2016_TelID428
CameraTable_VersionFeb2016_TelID429
CameraTable_VersionFeb2016_TelID430
CameraTable_VersionFeb2016_TelID431
CameraTable_VersionFeb2016_TelID432
CameraTable_VersionFeb2016_TelID433
CameraTable_VersionFeb2016_TelID434
CameraTable_VersionFeb2016_TelID435
CameraTable_VersionFeb2016_TelID436
CameraTable_VersionFeb2016_TelID437
CameraTable_VersionFeb2016_TelID438
CameraTable_VersionFeb2016_TelID439
CameraTable_VersionFeb2016_TelID440
CameraTable_VersionFeb2016_TelID441
CameraTable_VersionFeb2016_TelID442
CameraTable_VersionFeb2016_TelID443
CameraTable_VersionFeb2016_TelID444
CameraTable_VersionFeb2016_TelID445
CameraTable_VersionFeb2016_TelID446
CameraTable_VersionFeb2016_TelID447
CameraTable_VersionFeb2016_TelID448
CameraTable_VersionFeb2016_TelID449
CameraTable_VersionFeb2016_TelID450
CameraTable_VersionFeb2016_TelID451
CameraTable_VersionFeb2016_TelID452
CameraTable_VersionFeb2016_TelID453
CameraTable_VersionFeb2016_TelID454
CameraTable_VersionFeb2016_TelID455
CameraTable_VersionFeb2016_TelID456
CameraTable_VersionFeb2016_TelID457
CameraTable_VersionFeb2016_TelID458
CameraTable_VersionFeb2016_TelID459
CameraTable_VersionFeb2016_TelID460
CameraTable_VersionFeb2016_TelID461
CameraTable_VersionFeb2016_TelID462
CameraTable_VersionFeb2016_TelID463
CameraTable_VersionFeb2016_TelID464
CameraTable_VersionFeb2016_TelID465
CameraTable_VersionFeb2016_TelID466
CameraTable_VersionFeb2016_TelID467
CameraTable_VersionFeb2016_TelID468
CameraTable_VersionFeb2016_TelID469
CameraTable_VersionFeb2016_TelID470
CameraTable_VersionFeb2016_TelID471
CameraTable_VersionFeb2016_TelID472
CameraTable_VersionFeb2016_TelID473
CameraTable_VersionFeb2016_TelID474
CameraTable_VersionFeb2016_TelID475
CameraTable_VersionFeb2016_TelID476
CameraTable_VersionFeb2016_TelID477
CameraTable_VersionFeb2016_TelID478
CameraTable_VersionFeb2016_TelID479
CameraTable_VersionFeb2016_TelID480
CameraTable_VersionFeb2016_TelID481
CameraTable_VersionFeb2016_TelID482
CameraTable_VersionFeb2016_TelID483
CameraTable_VersionFeb2016_TelID484
CameraTable_VersionFeb2016_TelID485
CameraTable_VersionFeb2016_TelID486
CameraTable_VersionFeb2016_TelID487
CameraTable_VersionFeb2016_TelID488
CameraTable_VersionFeb2016_TelID489
CameraTable_VersionFeb2016_TelID490
CameraTable_VersionFeb2016_TelID491
CameraTable_VersionFeb2016_TelID492
CameraTable_VersionFeb2016_TelID493
CameraTable_VersionFeb2016_TelID494
CameraTable_VersionFeb2016_TelID495
CameraTable_VersionFeb2016_TelID496
CameraTable_VersionFeb2016_TelID497
CameraTable_VersionFeb2016_TelID498
CameraTable_VersionFeb2016_TelID499
CameraTable_VersionFeb2016_TelID500
CameraTable_VersionFeb2016_TelID501
CameraTable_VersionFeb2016_TelID502
CameraTable_VersionFeb2016_TelID503
CameraTable_VersionFeb2016_TelID504
CameraTable_VersionFeb2016_TelID505
CameraTable_VersionFeb2016_TelID506
CameraTable_VersionFeb2016_TelID507
CameraTable_VersionFeb2016_TelID508
CameraTable_VersionFeb2016_TelID509
CameraTable_VersionFeb2016_TelID510
CameraTable_VersionFeb2016_TelID511
CameraTable_VersionFeb2016_TelID512
CameraTable_VersionFeb2016_TelID513
CameraTable_VersionFeb2016_TelID514
CameraTable_VersionFeb2016_TelID515
CameraTable_VersionFeb2016_TelID516
CameraTable_VersionFeb2016_TelID517
CameraTable_VersionFeb2016_TelID518
CameraTable_VersionFeb2016_TelID519
CameraTable_VersionFeb2016_TelID520
CameraTable_VersionFeb2016_TelID521
CameraTable_VersionFeb2016_TelID522
CameraTable_VersionFeb2016_TelID523
CameraTable_VersionFeb2016_TelID524
CameraTable_VersionFeb2016_TelID525
CameraTable_VersionFeb2016_TelID526
CameraTable_VersionFeb2016_TelID527
CameraTable_VersionFeb2016_TelID528
CameraTable_VersionFeb2016_TelID529
CameraTable_VersionFeb2016_TelID530
CameraTable_VersionFeb2016_TelID531
CameraTable_VersionFeb2016_TelID532
CameraTable_VersionFeb2016_TelID533
CameraTable_VersionFeb2016_TelID534
CameraTable_VersionFeb2016_TelID535
CameraTable_VersionFeb2016_TelID536
CameraTable_VersionFeb2016_TelID537
CameraTable_VersionFeb2016_TelID538
CameraTable_VersionFeb2016_TelID539
CameraTable_VersionFeb2016_TelID540
CameraTable_VersionFeb2016_TelID541
CameraTable_VersionFeb2016_TelID542
CameraTable_VersionFeb2016_TelID543
CameraTable_VersionFeb2016_TelID544
CameraTable_VersionFeb2016_TelID545
CameraTable_VersionFeb2016_TelID546
CameraTable_VersionFeb2016_TelID547
CameraTable_VersionFeb2016_TelID548
CameraTable_VersionFeb2016_TelID549
CameraTable_VersionFeb2016_TelID550
CameraTable_VersionFeb2016_TelID551
CameraTable_VersionFeb2016_TelID552
CameraTable_VersionFeb2016_TelID553
CameraTable_VersionFeb2016_TelID554
CameraTable_VersionFeb2016_TelID555
CameraTable_VersionFeb2016_TelID556
CameraTable_VersionFeb2016_TelID557
CameraTable_VersionFeb2016_TelID558
CameraTable_VersionFeb2016_TelID559
CameraTable_VersionFeb2016_TelID560
CameraTable_VersionFeb2016_TelID561
CameraTable_VersionFeb2016_TelID562
CameraTable_VersionFeb2016_TelID563
CameraTable_VersionFeb2016_TelID564
CameraTable_VersionFeb2016_TelID565
CameraTable_VersionFeb2016_TelID566
CameraTable_VersionFeb2016_TelID567
CameraTable_VersionFeb2016_TelID568

CameraDisplay should have a friendly interface to set the normalization to e.g. log(value), sqrt(value), etc. Internally it can be done simply by using self.pixels.set_norm(x) and self.cmap.set_norm(x) where x is a matplotlib.colors.Normalize instance (e.g. matplotlib.colors.LogNorm())

Perhaps making this a attribute like cmap so that it works like this:

disp = CameraDisplay(...)
disp.norm = "log"
disp.norm = "sqrt"

# or they can set it from an instance:
disp.norm = matplotlib.colors.LogNorm(vmin=0.1)

Alternately follow the matplotlib convention of using a function like disp.set_norm(x)

The module names are currently quite confusing and should be restructured. One suggestion:

• move CameraGeometry from io to instrument (it's really a instrumental data model, and io should be purely for data access).
• create an image module that handles all image-level routines that are currently in reco
  • hillas parameterization , image cleaning, feature extraction, etc, should all be in image
  • shower reconstruction methods should be in reco

Should we chunk the data or not? Here we are not talking at the file or i/o level (which is chunked by default in some optimal way), but rather for the algorithms. Therefore it implies a major difference to the internal data structure.
#70 is an example implementation of a chunked data structure that could replace the current RawDataContainer structure in ctapipe

• event-wise:
  • simple to implement algorithms
  • low memory footprint
• chunk-wise:
  • can make use of CPU and GPU features, since have many events in a block of memory
  • the camera trace data is a 3D array (nevnt, npix, ntimeslice)
  • simplifies some algorithms that need to loop multiple times over a set of events

However, it was pointed out that if we use a fancier stream processing system, that system can buffer events into a chunk, and provide efficient event-wise access.

@kosack – We should release early and often.

So I've added a 0.1 milestone for Friday next week (July 3).

This is a test release where I want to see if making the source tarball works and things like accessing the right version of cta-extra data from a stable release. It will not be advertised to anyone.

IMO it's not important to have any features ready by then, although having direction reco working by the end of next week would be nice.

The update method of CameraDisplay only updates when autoupdate == True.

I think this is misleading and also does not allow the user to force an update if autoupdate == False.

Should we change update to always update and change our calls to:

if self.autoupdate:
    self.update()

?

libhessio from hessioxxx displays an error at the tenth event with read_hessio.py and stops read_hession.py

Wrong number of bytes were read (635256 instead of 1560656)

Dear Karl,

I think there is a bug in the ctapipe.io.hessio class
Only one Container instance is create before the loop on the event.
Then inside the loop, it always uses the same instance.
So we cannot keep an instance of this container in a list for example:
See example bellow

For information, copy a Container class does not work.

============================ Test code =====================
from ctapipe.io.hessio import hessio_event_source
from ctapipe.configuration.core import Configuration, ConfigurationException
from ctapipe.core import Container
from ctapipe.utils.datasets import get_path

if name == 'main':
raw_data = '/home/jacquem/workspace/data/gamma_20deg_0deg_run31964___cta-prod2_desert-1640m-Aar.simtel.gz'
source = hessio_event_source(get_path(raw_data), max_events=10)

l = list()
for event in source:
    print (event.dl0.event_id)
    l.append(event)

print( "+---- list ----+") 
for event in l:
    print (event.dl0.event_id)

RESULT__
jacquem>python test_hessio.py
/home/jacquem/workspace/data/gamma_20deg_0deg_run31964___cta-prod2_desert-1640m-Aar.simtel.gz

Input file '/home/jacquem/workspace/data/gamma_20deg_0deg_run31964___cta-prod2_desert-1640m-Aar.simtel.gz' has been opened.

Starting run 31964
408
409
803
4907
9508
10104
10109
11905
12202
12203
17301
+---- list ----+
17301
17301
17301
17301
17301
17301
17301
17301
17301
17301
17301

I think matplotlib should be an optional dependency for ctapipe and python setup.py test should succeed if it's not present.

But currently there's a module-level import of matplotlib in ctapipe/visualization/mpl.py:6 and then pytest errors out trying to collect tests:
https://travis-ci.org/cta-observatory/ctapipe/jobs/89389169#L613

In Astropy / Gammapy this is solved by putting the imports of the optional packages like matplotlib into functions / methods instead of the start of the file, and then by marking tests to declare optional dependencies those tests are skipped if the dependency is not present (see https://gammapy.readthedocs.org/en/latest/api/gammapy.utils.testing.requires_dependency.html).

It should also be possible to configure the pytest test collection to avoid collecting tests from certain modules / packages if a dependency needed there isn't present.

I don't plan to fix this, but merge #33 as-is, so CI will still be broken in master for now.

In matplotlib the kwarg for an axes is ax, or cax for colorbar().

I think ctapipe should use the same kwargs as matplotlib.

Currently all exception handling in the instrument module is using:

try:
    do_stuff()
except:
    pass

This catches every exception that occuress, so also SystemExit and KeyboardInterrupt. This makes it impossible to kill a running program.

Also the exceptions should probably be logged.

build_sphinx is the last failing test on travis.
It return none-zero value because of warnings, not error.
The easy solution is to remove '-w' option, but detect warning are clear advantages.
Does someone know how to remove this 3 warnings ?
https://travis-ci.org/cta-observatory/ctapipe/jobs/113655229

@kosack – Can you please commit this example file to the ctapipe-extra repo?

https://github.com/cta-observatory/ctapipe/blob/master/examples/read_hessio.py#L7

filename = "/Users/kosack/Data/CTA/Prod2/proton_20deg_180deg_run32364___cta-prod2_desert-1640m-Aar.simtel.gz"

If it's too large (say > 10 MB), do you have a smaller example file?

cc @jacquemier

@kosack - How should we use hessioxxx from ctapipe?

I guess the versions are decoupled, so no need to have hessioxxx as a git submodule here.
Could still be convenient though for devs to get it via make init (I added a Makefile) and know the location of hessioxxx so that we can call hessioxxx cmake from ctapipe setup.py?

Add the Python wrappers in hessioxxx or ctapipe?

Should we try to build a conda package for hessioxxx and use it as a library, or should I build it from source on travis-ci in the ctapipe tests for now?

The container classes (e.g. io.RawData) are still very c-like and could be improved to be nicer. So far code using them requires many nested for-loops, which is ugly. One thing would be to make them iterable, e.g.:

rawdata = RawData()..

for tel in rawdata:
    for chan in tel:
        do_something(tel,chan)

or even

for tel,chan in rawdata:
     do_something(tel, chan)

this would allow one to use map() or similar functions to process data as well

It would be nice to have a shower image browser (with buttons to select the event) for files like https://github.com/cta-observatory/ctapipe-extra/blob/master/datasets/televentlist_img_small.fits.gz .

@kosack – Do you have something like that? If no, would it be best to implement it with matplotlib or QT or IPython widgets or as a small web app (e.g. using bottle)?

traitlets.config looks reasonable for a system to configure classes using config files or command-line args, which may be better than using argparse. We'd just need a json to FITS header function to write them to output files for provenance data:

http://traitlets.readthedocs.org/en/stable/config.html

Might be worth making a prototype to see if it meets our needs.

when running python setup.py bdist_conda (or now make dist) the conda package is built and installed in a virtualenv. It all works until the tests are run, and they fail because scipy is not found (because it is not instelled in the virtualenv).

setup.py seems to know the following dependencies:

    astropy:    1.0.3-np19py34_0 
    ctapipe:    0.0.dev110-py34_0
    numpy:      1.9.2-py34_0     
    openssl:    1.0.1k-1         
    pip:        7.0.3-py34_0     
    python:     3.4.3-0          
    readline:   6.2-2            
    setuptools: 17.1.1-py34_0    
    sqlite:     3.8.4.1-1        
    tk:         8.5.18-0         
    xz:         5.0.5-0          
    zlib:       1.2.8-0    

but it's not obvious to me where it gets this list. How can we add scipy to the list of required dependencies? Where are the deps listed?

Somehow the ctapipe-extra git submodule pointer is pointing to a non-existing commit:

$ make init
git submodule init
Submodule 'astropy_helpers' (https://github.com/astropy/astropy-helpers.git) registered for path 'astropy_helpers'
Submodule 'ctapipe-extra' (https://github.com/cta-observatory/ctapipe-extra.git) registered for path 'ctapipe-extra'
git submodule update
Cloning into 'astropy_helpers'...
remote: Counting objects: 2178, done.
remote: Total 2178 (delta 0), reused 0 (delta 0), pack-reused 2178
Receiving objects: 100% (2178/2178), 614.26 KiB | 396.00 KiB/s, done.
Resolving deltas: 100% (1439/1439), done.
Checking connectivity... done.
Submodule path 'astropy_helpers': checked out 'd51f726673510586d4b3bd04fa4799fdab389aab'
Cloning into 'ctapipe-extra'...
remote: Counting objects: 8, done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 8 (delta 0), reused 5 (delta 0), pack-reused 0
Unpacking objects: 100% (8/8), done.
Checking connectivity... done.
fatal: reference is not a tree: b5e07c6c1591d253cd45fadcded8610f60cb682a
Unable to checkout 'b5e07c6c1591d253cd45fadcded8610f60cb682a' in submodule path 'ctapipe-extra'
make: *** [init] Error 1

I have no idea how that happened ... I'll fix this now.

It's no longer possible to build the ctapipe Sphinx docs if hessio isn't present.
(Sphinx tries to import hessio and fails with an ImportError).

This also broke continuous integration:
https://gist.github.com/cdeil/7fe1706ed368f7e52c72#file-gistfile1-txt-L133
https://travis-ci.org/cta-observatory/ctapipe/jobs/90762905#L1017

Related: #22 and the efforts to improve hessio install by @jacquemier and @maxnoe .

What's the solution here?
Is hessio a required or optional dependency for ctapipe?

@kosack The tests on travis-ci are currently broken because of this doctest fail:
https://travis-ci.org/cta-observatory/ctapipe/jobs/68173830#L552

By default python setup.py test will run all doctests ... we can change this, but I think it's the better default.

Here's docs how to skip doctests (or mark them to need remote data):
http://astropy.readthedocs.org/en/latest/development/testguide.html#skipping-doctests

I modified phyhessio installation procedure on ctapipe documents (ctapipe/doc/getting_started/index.rst) but it seems that upload to documantion page is not automatic

This is a matplotlib bug, not much we can do about this at the moment, see

matplotlib/matplotlib#5424

@astrofrog or @embray – Could you help me with an issue with the astropy-helpers setup in this package?

I first bundled astropy_helpers because I didn't want the submodule. Then I always got some warning from python setup.py that the git submodule wasn't there. So I changed it and now I have astropy-helpers as a git submodule.

But there must be something wrong, I'm getting this error from the sphinx build on travis-ci (which I don't see locally):
https://travis-ci.org/cta-observatory/ctapipe/jobs/68434189#L942

Traceback (most recent call last):
  File "setup.py", line 129, in <module>
    **package_info
  File "/home/travis/miniconda/envs/test/lib/python3.4/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/home/travis/miniconda/envs/test/lib/python3.4/distutils/dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "/home/travis/miniconda/envs/test/lib/python3.4/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/home/travis/build/cta-observatory/ctapipe/.eggs/astropy_helpers-1.0.2-py3.4.egg/astropy_helpers/commands/build_sphinx.py", line 120, in run
    ah_path = os.path.abspath(ah_importer.path)
AttributeError: 'NoneType' object has no attribute 'path'

And locally there's this:

$ python setup.py build_sphinx
git submodule command failed unexpectedly:
error: pathspec 'astropy_helpers' did not match any file(s) known to git.
Did you forget to 'git add'?

$ git status
On branch master
Your branch is up-to-date with 'origin/master'.
Untracked files:
  (use "git add <file>..." to include in what will be committed)

    astropy_helpers/

nothing added to commit but untracked files present (use "git add" to track)

What do I have to do to get the astropy-helpers setup corrected in this package?

I'm starting to make issues for myself as reminders for things I can't resolve immediately.

The docs build on readthedocs is currently still broken:
https://readthedocs.org/builds/ctapipe/3001195/

Might be an issue with Python 3 installation of dependencies ...

CameraDisplay now uses plt.sci(self.pixels) to set the current image to update the color bar if it exists.

This fails with a ValueError if more than one plot is on a figure (see e.g. examples/camera_display_multi.py).

For now, I catch the ValueError and ignore it, but that is not a good solution as the color bar does not get updated in that case.

@maxnoe do you know how to fix that?

Reading the peds and gains using pyhessio like this:

peds = pyhessio.get_pedestal(tel_id)[0]
gains = pyhessio.get_calibration(tel_id)[0]
plot(peds, gains, 'r+')

Works for Prod2 files:

But for Prod3 files you get 2 distinct distributions of pedestals and gains:

Looking at it in image space, here is what you get when applying the coefficients, which looks like every other pixel has the wrong calibration (implying mixing of the two gain channels):

And the raw data

Hello,
I think it makes sense to start some clean up right now. I think there are some obsolete files we can delete.
There are a lot of scripts and notebooks in the examples folder. Keeping them all up to date will be hard. Maybe we should concentrate on the main use case for now and just delete/move other stuff.

Here is the output of the vulture tool to find unused functions and classes:

ctapipe/_astropy_init.py:14: Unused attribute 'ASTROPY_SETUP'
ctapipe/_astropy_init.py:31: Unused function 'test'
ctapipe/configuration/core.py:21: Unused attribute 'msg'
ctapipe/configuration/core.py:237: Unused attribute 'optionxform'
ctapipe/configuration/core.py:418: Unused function 'get_list'
ctapipe/configuration/core.py:438: Unused function 'get_section_list'
ctapipe/configuration/core.py:448: Unused function 'dynamic_class_from_module'
ctapipe/configuration/tests/test_core.py:31: Unused variable 'res'
ctapipe/coordinates/frames.py:67: Unused variable 'default_representation'
ctapipe/coordinates/frames.py:84: Unused class 'TiltedTelescopeFrame'
ctapipe/coordinates/frames.py:90: Unused variable 'pointing_direction'
ctapipe/coordinates/frames.py:91: Unused variable 'telescope_location'
ctapipe/coordinates/frames.py:95: Unused class 'GroundFrame'
ctapipe/coordinates/frames.py:100: Unused variable 'observatory_location'
ctapipe/coordinates/frames.py:103: Unused function 'camera_to_telescope'
ctapipe/coordinates/frames.py:116: Unused function 'telescope_to_camera'
ctapipe/core/init.py:12: Unused function 'component'
ctapipe/instrument/CameraDescription.py:57: Unused function 'to_table'
ctapipe/instrument/InstrumentDescription.py:15: Unused class 'Atmosphere'
ctapipe/instrument/InstrumentDescription.py:19: Unused attribute 'rho'
ctapipe/instrument/InstrumentDescription.py:20: Unused attribute 'thickness'
ctapipe/instrument/InstrumentDescription.py:21: Unused attribute 'ext_coeff'
ctapipe/instrument/InstrumentDescription.py:23: Unused function 'load_profile'
ctapipe/instrument/InstrumentDescription.py:40: Unused function 'load_extinction_coeff'
ctapipe/instrument/obsconfig.py:36: Unused function 'get_site_id_for_run'
ctapipe/instrument/obsconfig.py:41: Unused function 'get_site_id_for_time'
ctapipe/instrument/obsconfig.py:61: Unused attribute 'site_id'
ctapipe/instrument/obsconfig.py:66: Unused attribute 'version_id'
ctapipe/instrument/obsconfig.py:67: Unused attribute 'pixel_type'
ctapipe/instrument/obsconfig.py:70: Unused attribute 'pix_z'
ctapipe/instrument/obsconfig.py:72: Unused attribute 'focal_plane_offset'
ctapipe/instrument/obsconfig.py:74: Unused function 'load_from_file'
ctapipe/instrument/obsconfig.py:79: Unused class 'ArrayTriggerConfig'
ctapipe/instrument/obsconfig.py:84: Unused class 'TelescopeTriggerConfig'
ctapipe/instrument/obsconfig.py:101: Unused attribute '_optics'
ctapipe/instrument/obsconfig.py:102: Unused attribute '_camera'
ctapipe/instrument/obsconfig.py:121: Unused attribute 'tel_x'
ctapipe/instrument/obsconfig.py:122: Unused attribute 'tel_y'
ctapipe/instrument/obsconfig.py:123: Unused attribute 'tel_z'
ctapipe/instrument/obsconfig.py:137: Unused property 'num_tels'
ctapipe/instrument/obsconfig.py:154: Unused attribute 'optics_type'
ctapipe/instrument/obsconfig.py:155: Unused attribute 'mirror_area'
ctapipe/instrument/obsconfig.py:157: Unused attribute 'facet_x'
ctapipe/instrument/obsconfig.py:158: Unused attribute 'facet_y'
ctapipe/instrument/obsconfig.py:159: Unused attribute 'facet_z'
ctapipe/instrument/obsconfig.py:160: Unused attribute 'facet_area'
ctapipe/instrument/obsconfig.py:169: Unused attribute 'run_id'
ctapipe/instrument/obsconfig.py:178: Unused attribute '_run_id'
ctapipe/instrument/obsconfig.py:181: Unused attribute '_array'
ctapipe/instrument/obsconfig.py:184: Unused attribute '_subarray'
ctapipe/instrument/obsconfig.py:187: Unused attribute '_trigger'
ctapipe/instrument/obsconfig.py:190: Unused class 'SimObsConfig'
ctapipe/instrument/obsconfig.py:195: Unused attribute '_mc_run_id'
ctapipe/io/array.py:29: Unused function 'tel_class_name'
ctapipe/io/camera.py:86: Unused function 'from_file'
ctapipe/io/data_container.py:12: Unused variable 'eventnumber'
ctapipe/io/data_container.py:66: Unused property 'telescope_id'
ctapipe/io/data_container.py:70: Unused property 'telescope_name'
ctapipe/io/hessio.py:71: Unused attribute 'event_id'
ctapipe/io/hessio.py:82: Unused attribute 'tels_with_trigger'
ctapipe/io/hessio.py:85: Unused attribute 'gps_time'
ctapipe/io/hessio.py:87: Unused attribute 'energy'
ctapipe/io/hessio.py:88: Unused attribute 'alt'
ctapipe/io/hessio.py:89: Unused attribute 'az'
ctapipe/io/hessio.py:90: Unused attribute 'core_x'
ctapipe/io/hessio.py:91: Unused attribute 'core_y'
ctapipe/io/hessio.py:110: Unused attribute 'num_channels'
ctapipe/io/mock.py:16: Unused function 'mock_event_source'
ctapipe/reco/hillas.py:37: Unused variable 'HighOrderMomentParameters'
ctapipe/reco/hillas.py:49: Unused function 'hillas_parameters_1'
ctapipe/reco/hillas.py:108: Unused variable 'azwidth'
ctapipe/reco/hillas.py:195: Unused variable 'hillas_parameters'
ctapipe/tests/setup_package.py:1: Unused function 'get_package_data'
ctapipe/tools/camdemo.py:30: Unused attribute 'cmap'
ctapipe/tools/camdemo.py:58: Unused attribute 'image'
ctapipe/tools/camdemo.py:62: Unused variable 'anim'
ctapipe/tools/dump_triggers.py:34: Unused attribute 'description'
ctapipe/tools/utils.py:47: Unused function 'get_all_main_functions'
ctapipe/utils/datasets.py:47: Unused function 'get_example_simtelarray_file'
ctapipe/utils/fitshistogram.py:95: Unused property 'bins'
ctapipe/utils/fitshistogram.py:99: Unused property 'ranges'
ctapipe/utils/fitshistogram.py:107: Unused function 'outliers'
ctapipe/utils/fitshistogram.py:144: Unused function 'asFITS'
ctapipe/utils/fitshistogram.py:293: Unused function 'draw2D'
ctapipe/utils/fitshistogram.py:321: Unused function 'draw1D'
ctapipe/utils/fitshistogram.py:328: Unused function 'interpolate'
ctapipe/version.py:201: Unused variable 'major'
ctapipe/version.py:202: Unused variable 'minor'
ctapipe/version.py:203: Unused variable 'bugfix'
ctapipe/version.py:206: Unused variable 'timestamp'
ctapipe/version.py:207: Unused variable 'debug'
ctapipe/version.py:212: Unused variable 'compiler'
ctapipe/version.py:217: Unused variable 'cython_version'
ctapipe/visualization/mpl.py:214: Unused property 'cmap'
ctapipe/visualization/mpl.py:232: Unused property 'image'
ctapipe/visualization/mpl.py:313: Unused function 'overlay_moments'
ctapipe/visualization/mpl.py:390: Unused property 'values'
ctapipe/visualization/tests/test_mpl.py:48: Unused attribute 'intensities'

ctapipe.io.__init__:

from .containers import *

ctapipe.io.containers:

from ctapipe.core import Container

ctapipe.core.__init__

from ctapipe.io.files import get_file_type