writing a byte array to a char waveform is broken, leaving trailing junk

sending a string is worse, adding an extra 'b' to the result

sending a length-1 string might be broken too.

Hi,

I am really not sure about this, but wouldn't it be saver to use pv_get() instead of PV() to prevent creating lots and lots of objects when this script is used for a data logging system?

regards
menno

Steps to reproduce:

• Remove any pre-existing installations of pyepics
• Clone the repository
• Build with
  cd pyepics
  python setup.py build

The build fails with:

Traceback (most recent call last):
  File "setup.py", line 7, in <module>
    import lib
  File "/home/mgigg/Software/Git/pyepics/lib/__init__.py", line 30, in <module>
    from . import multiproc
  File "/home/mgigg/Software/Git/pyepics/lib/multiproc.py", line 20, in <module>
    import epics
ImportError: No module named epics

It looks as if this commit introduced a requirement for the module to be available as epics and not lib

We've stumbled across a rather inconvenient bug, that is after we disconnect from the PV the PV can not be reconnected anymore. Neither PV.connect nor PV.force_connect work correctly. The first one never connects, the second one successfully connects to the channel but fails with key error when re-adding the callback. See attached files for stand-alone demonstration of the bug...

"""
Test-case that demonstrates that after disconnect and channel can not be reconnected using connect method

To try it out, simply start example CAS server bundled with EPICS base (excas binary) before starting the test

The output for me is: 

Connecting
UPD 6.63983507548
SUM 6.63983507548
UPD 76.9632902148
Disconnecting
Reconnecting
Traceback (most recent call last):
  File "test.py", line 33, in <module>
    assert(pv_test.wait_for_connection(5.0) == True)
AssertionError

"""

import epics
import time

pv_test = epics.PV("alan",auto_monitor=True)


def callback(value=None,**kwargs):
    print("UPD",value.sum())



pv_test.add_callback(callback)

print("Connecting")
assert(pv_test.wait_for_connection(5.0) == True)

print("SUM",pv_test.get().sum())

time.sleep(3)

print("Disconnecting")
pv_test.disconnect()
print("Reconnecting")

# Always fails :( 
pv_test.connect()
assert(pv_test.wait_for_connection(5.0) == True)
pv_test.add_callback(callback)


time.sleep(5)

"""
Test-case that demonstrates that after disconnect a channel can be
reconnected using undocumented force_connect, however the callback can not be added again (after force_connect no original callbacks are invoked)

To try it out, simply start example CAS server bundled with EPICS base (excas binary) before starting the test

The output for me is: 

Connecting
UPD 6.92243429227
SUM 6.92243429227
UPD -6.11459633615
Disconnecting
Reconnecting
Traceback (most recent call last):
  File "test2.py", line 49, in <module>
UPD -6.11459633615
    pv_test.add_callback(callback)
  File "/usr/lib/python3.5/site-packages/epics/pv.py", line 502, in add_callback
    self.get_ctrlvars()
  File "/usr/lib/python3.5/site-packages/epics/pv.py", line 425, in get_ctrlvars
    kwds = ca.get_ctrlvars(self.chid, timeout=timeout, warn=warn)
  File "/usr/lib/python3.5/site-packages/epics/ca.py", line 419, in wrapper
    return fcn(*args, **kwds)
  File "/usr/lib/python3.5/site-packages/epics/ca.py", line 1418, in get_ctrlvars
    ncache = _cache[current_context()][name(chid)]
KeyError: 'alan'


"""

import epics
import time

pv_test = epics.PV("alan",auto_monitor=True)


def callback(value=None,**kwargs):
    print("UPD",value.sum())



pv_test.add_callback(callback)

print("Connecting")
assert(pv_test.wait_for_connection(5.0) == True)

print("SUM",pv_test.get().sum())

time.sleep(3)

print("Disconnecting")
pv_test.disconnect()
print("Reconnecting")

# Always fails :( 
pv_test.force_connect()
assert(pv_test.wait_for_connection(5.0) == True)
pv_test.add_callback(callback)


time.sleep(5)

Devices() should have a "mutable" arg (default=False) to indicate that, once created with a set of attributes, new attributes cannot be added simply by accessing attributes. That is:
d = Device('prefix', attrs=('foo', 'bar'), delim='.', mutable=False)

should mean that
d.baz

does NOT look for and wait for connection to a PV named "prefix.baz" -- it should raise an AttributeError.

To allow the current "please-hang-on-all-typos and discover PVs" behavior, use
d = Device(mutable=True)

Note that Motor() would need mutable=True, but I think all others would not.

looks like its in ca.put?

Hi @newville, I am very interested on the callbacks for write_access. Do you plan to cut a release of PyEpics soon that will include those changes on master?

Thanks!

had an incomplete report of using epics.caget() in a wx GUI causing threading errors, even with '@EpicsFunction' decorator, and that replacing this with creating a PV() worked....

there were many issues with 64-bit python on Windows, basically preventing it from working at all.

I've recently been implementing some controls to edit "banks" of values which are stored as arrays in waveform records.

I tried a few different approaches, including making a PV subclass that wrapped a single value from the array but otherwise behaved like a PV.

That mostly worked, but for now I've settled on subclassing the controls I need like this:

class PVItemTextCtrl(PVTextCtrl):
    """ A PVTextCtrl subclass that only sets one single item
    in an array-type waveform record.
    """
    def __init__(self, parent,  pv, index, **kws):
        self.index = index
        print pv
        PVTextCtrl.__init__(self, parent=parent, pv=pv, **kws)

    @DelayedEpicsCallback
    def OnEpicsConnect(self, pvname=None, conn=None, pv=None):
        PVTextCtrl.OnEpicsConnect(self,pvname,conn,pv)
        self._SetValue(None) # do an initial value callback, seems pyepics doesn't always send one

    def SetValue(self, value):
        "override all setvalue"
        full_value = self.pv.get(as_string=False, as_numpy=False) # as a list
        full_value[self.index] = value
        self.pv.put(full_value)

    def _SetValue(self, value):
        "set widget value"
        full_value = self.pv.get(as_string=False, as_numpy=False) # as a list
        PVTextCtrl._SetValue(self, full_value[self.index])

Does anyone else have any thoughts on how to proceed? Would a generic "PVArrayItem" class that behaves like a PV possibly be a better solution?

There are several issues with getting values, especially for moderately sized arrays.

a) as_numpy not respected
b) the issue with "unpacking" arrays

I think there needs a careful re-working of this, to make pyepics 3.2 , with options for
auto_monitoring and whether "unpack"ing happens.

I'm trying to use PyEPICS with multiple threads on Mac OS X 10.9.5 with EPICS base 3.14.12.5. All approaches seem to lead to channel access context errors. For example, the following code:

import epics
from epics.ca import CAThread

pv = epics.PV('SR11BCM01:CURRENT_MONITOR')

def callback():
    print(pv.get())

CAThread(target=callback).start()

will generate this traceback:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/local/lib/python2.7/site-packages/epics/ca.py", line 1667, in run
    Thread.run(self)
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "app.py", line 7, in callback
    print(pv.get())
  File "/usr/local/lib/python2.7/site-packages/epics/pv.py", line 275, in get
    if not self.wait_for_connection():
  File "/usr/local/lib/python2.7/site-packages/epics/pv.py", line 219, in wait_for_connection
    ca.attach_context(self.context)
  File "/usr/local/lib/python2.7/site-packages/epics/ca.py", line 347, in wrapper
    return fcn(*args, **kwds)
  File "/usr/local/lib/python2.7/site-packages/epics/ca.py", line 444, in wrapper
    return PySEVCHK( fcn.__name__, status)
  File "/usr/local/lib/python2.7/site-packages/epics/ca.py", line 432, in PySEVCHK
    raise CASeverityException(func_name, message(status))
CASeverityException:  attach_context returned 'Thread is already attached to a client context'

The same code works fine on Linux. I have tried the other approaches from the "How to work with CA and Threads" documentation such as the withInitialContext decorator or calling epics.ca.use_initial_context() and these result in the same issue or cause a segmentation fault.

Am I doing something wrong or is threading broken for pyepics on OS X?

pv.timestamp (calculated with dbr.make_unixtime) loses precision which should be retained for those that need sub-microsecond timing.

Adding attributes of seconds and nanoseconds that come directly from the DBR timestamp structure would fix that.

Hey,

for me it seems like pyepics has some trouble parsing strings with special characters.
Example:
caget('String_PV') returns "J\303\274rgen" which translates to "Jürgen", since \303 \274 is utf8 encoded "ü" in octal (hexadecimal would be "c3 bc")

Doing the same with pyepics returns neither the octal version nor the "ü", but instead: "Jürgen". Is there a way to get the same result as for the epics base caget()? Or even the correctly translated special character?

via bluesky/ophyd#135 (comment)

One thing @dchabot and I found yesterday was that epics.PV().units returns a byte string in 3.x. Need to get this fixed upstream in pyepics.

Moved here so it does not get lost.

I recently pulled into master #80 -- an attempt to distribute libCom and libca for more platforms (linux, darwin) that are not built with readline support. This is especially to avoid conflicts between system readline and Anaconda readline.

The PR has a problem for linux64 that libca.so looks for "libCom.so.3.15.5" -- this can be fixed with a symlink.

But it also has a larger issue in that the install with setuptools installs these libs inside the "pyepics....egg" and the libca library still cannot find libCom.so.*.

I believe the solution is to revert from setuptools to distutils.core so that the installation from the source kit goes to {ANACONDA}/lib/pythonX.Y/epics, and the "data" dlls go to {ANACONDA}/lib. I would be very happy to give up the eggs thing.

Does this seem reasonable to others? If not, how do others handle running on linux with anaconda?

As per a suggestion Matt made to me, get_ctrlvars() should be more robust if we convert it to use a callback.

(I don't have time to do this now, but I'm listing it as an Issue assigned to me because I think it's a good idea.)

So first, I'm Peter, I'm at the ANU in Canberra working with the EPICS system developed by Angus Gratton. Software Developer is not my primary role, so please be patient with me!

On to the issue. If I use a PVText on a EPICS record with a defined precision, say 3, and that number is 1e-5 or smaller, then the resulting text is 1e-5 (or similar). This is fine. However, if that number is 1e-4, then the python conversion to string results in 0.0001. If the EPICS record precision is set to 3, then the PVText is 0.000, dropping the one. We don't want that.

We are basically tracking beam currents and I've set the precision to 3 so that normal range nanoamp currents come up at e.g. 2.34e-9 (as per user request). However, at the source, they occasionally get to >100e-6. This then shows up as 0.000 A.

I'm still learning about overriding methods in python, still can't figure out how to override _pvEvent which is where the formatting occurs (as far as I can tell).

Cheers
Peter

Initializing a epics.devices.scan.Scan() device erases NPTS and all triggers. This should be moved to the reset() method.

The lines 111-112 in device.py
cause all internal attributes like _prefix, _delim
to be hidden, if a nonpvs option is added to the constructor.

The reason for these lines (introduced in commit 3989b59 )
is not clear, as meachnisim to append nonpvs to is already present in lines 120-123.

Thus, the following code is broken in version 3.2.3:

class MyDevice(epics.Device):
    def __init__(self):
        epics.Device.__init__(self,prefix='BBQR',delim=':',nonpvs=('extravar',))
        self.extravar=1
        print self._prefix
md=MyDevice()

In version 3.2.1 this code was still working.

In an environment where channel access rights (read/write) can change dynamically, pyepics does not allow for this, and PV objects will always convey whatever their read/write access rights were on connection. Note, a PV that has had its read/write access revoked will still raise if a read/write is attempted.

Access Rights can change dynamically due to CA Security rules going in and out of effect, for example (guess how I found this problem??).

See test case

I would suggest that the way to deal with this is to have all PV objects register an access_rights_handler.

When using the logging module the messages are not printed to the standar output.

logging.info('My message')

does not show anything.

pyepics version 3.2.3

Red Hat 6.9, EPICS base 3.14.12.5, python 3.4.5, pyepics 3.2.7

[henrique.almeida@sol13-linux ~]$ python3
Python 3.4.5 (default, Jun 1 2017, 13:52:39)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-18)] on linux
Type "help", "copyright", "credits" or "license" for more information.

import epics
epics.caget('LNLS:ANEL:corrente')
177.617

Traceback (most recent call last):
File "/usr/lib/python3.4/site-packages/epics/ca.py", line 251, in initialize_libca
File "/usr/lib/python3.4/site-packages/epics/ca.py", line 213, in find_libCom
File "/usr/lib/python3.4/site-packages/epics/ca.py", line 178, in _find_lib
File "/usr/lib64/python3.4/ctypes/util.py", line 233, in find_library
File "/usr/lib64/python3.4/ctypes/util.py", line 203, in _findSoname_ldconfig
File "", line 2237, in _find_and_load
File "", line 2222, in _find_and_load_unlocked
File "", line 2155, in _find_spec
TypeError: 'NoneType' object is not iterable

cf - #92

if wx is not installed, all devices cannot be loaded. This is true only because ordereddict.py is in epics.wx.

This is a bad place for it, and it is only legacy code anyway.

For back compatibility but to preserve code working with py2.6, ordereddict.py should not be removed from epics.wx/, but should also be in epics.devices/, and epics.devices.mca should use this only if it cannot import OrderedDict from collections.

eventually, ordereddict should be removed from epics.wx

Hi,

Although EPICS base is installed and epics.ca.find_libca() returns the right path, when using systemd to start a script at startup, it seems to fails as ca.current_context() returns None:

Loaded: loaded (/lib/systemd/system/myscript.service; enabled)
Active: failed (Result: exit-code) since Sun 2017-09-17 20:33:47 UTC; 981ms ago
Process: 1658 ExecStart=/usr/bin/python /home/pi/myscript/myscript.py --option option1 (code=exited, status=1/FAILURE)
Main PID: 1658 (code=exited, status=1/FAILURE)

Sep 17 20:33:47 raspberrypi python[1658]: if ca.current_context() is None:
Sep 17 20:33:47 raspberrypi python[1658]: File "/usr/local/lib/python2.7/dist-packages/epics/ca.py", line 362, in wrapper
Sep 17 20:33:47 raspberrypi python[1658]: initialize_libca()
Sep 17 20:33:47 raspberrypi python[1658]: File "/usr/local/lib/python2.7/dist-packages/epics/ca.py", line 209, in initialize_libca
Sep 17 20:33:47 raspberrypi python[1658]: dllname = find_libca()
Sep 17 20:33:47 raspberrypi python[1658]: File "/usr/local/lib/python2.7/dist-packages/epics/ca.py", line 179, in find_libca
Sep 17 20:33:47 raspberrypi python[1658]: raise ChannelAccessException('cannot find Epics CA DLL')
Sep 17 20:33:47 raspberrypi python[1658]: epics.ca.ChannelAccessException: cannot find Epics CA DLL
Sep 17 20:33:47 raspberrypi systemd[1]: myscript.service: main process exited, code=exited, status=1/FAILURE
Sep 17 20:33:47 raspberrypi systemd[1]: Unit myscript.service entered failed state.

The systemd configuration is as such:

[Unit]
Description=myscript script service
After=multi-user.target

[Service]
Type=idle
ExecStart=/usr/bin/python /home/pi/myscript/myscript.py --option option1
User=pi

[Install]
WantedBy=multi-user.target

Any idea what is wrong here?

Hi,
I am running a server for a camera which can be triggered by sending a pv1.put(frames) and is sending data via a second pv using pv2.get(). This is working without any problem even with long numpy arrays (currently 493*659 pixel) if the environmental variable EPICS_CA_MAX_ARRAY_BYTES is set accordingly high.
It would be nice to monitor the pv2 to just send a pv1.put(frames) and get the picture with the shortest delay, but I tried with a few array sizes and it seems like camonitor just doesn't work if the array size exceeds some value between 57600 and 78400. There is no error message and no crash of the programm, it just doesn't react on value changes of pv2.
I hope someone can reproduce this.
Thanks and best regards
p

@dchabot Once #98 is merged, I think we're ready to call this v 3.3.0rc1. Honestly, we probably should have called 3.2.7 3.3.0rc1 as it introduced the new management (mismanagement?) of the ca shared libraries.

I think that your work on this makes the current master much better for the desires of people managing multi-user Linux systems.

Any objections to pushing this release candidate out and aiming for a full release of 3.3.0 next week?

Set-up:

• IOC with a waveform record (name WF_SP) in the db:
  record(waveform, "WF_SP") {
  field(DESC, "Sequence")
  field(FTVL, "LONG")
  field(NELM, "10")
  }
• no operation is done on the waveform after the IOC starts so it is totally uninitialized

Issue description:

When creating a PV as the following:
pv1= PV(pvname = 'WF_SP', callback = onChanges, auto_monitor=True, form='ctrl');

The following error is reported and callback is not called:

Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/local/lib/python2.7/dist-packages/pyepics-3.2.1-py2.7.egg/epics/ca.py", line 436, in _onMonitorEvent
tmpv = value[0]
IndexError: invalid index

This means that callback should be called with no value(?) and count set to 0(?), but one could still use other information from other callback arguments and kwargs.

The same issue does not happen when this is done on an uninitialized longout record (I guess this is true also for other types)

Best Regards,
Rok

I've found that PV's that are created, connected, then disconnected are never garbage collected and long-running applications get slower with time. This appears to be a different issue than the chid's being cached.

Using some forensics with the gc module and gc.get_referrers, I've found that the __on_connect object is never free'd after a disconnect. Turns out that _monref is never deleted or replaced, so a rogue reference to a bound __on_connect and thus to a "self" lives in it. I think the GC can't see the cycle through the ctypes.py_object.

    def disconnect(self):
        "disconnect PV"
        self.connected = False
        if self._monref is not None:
            cback, uarg, evid = self._monref
            ca.clear_subscription(evid)
            ctx = ca.current_context()
            if self.pvname in ca._cache[ctx]:
                ca._cache[ctx].pop(self.pvname)

            del cback
            del uarg
            del evid
            self._monref = None

        ca.poll(evt=1.e-3, iot=1.0)
        self.callbacks = {}

Hi,
Firstly, thanks for the great library! I've been using it lately to talk to the scan record. However, I've just discovered a bug if I'm trying to set a single value in one of the scan record Table Mode Position Arrays e.g., P1PA.

Errors:

Python 2.7.4 (default, Apr 6 2013, 19:04:04)
[GCC 4.8.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.

from epics import caput
caput('SCANPV:scan1.P1PA',1)
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python2.7/site-packages/epics/init.py", line 81, in caput
return thispv.put(value, wait=wait, timeout=timeout)
File "/usr/lib/python2.7/site-packages/epics/pv.py", line 277, in put
callback_data=callback_data)
File "/usr/lib/python2.7/site-packages/epics/ca.py", line 377, in wrapper
return fcn(_args, *_kwds)
File "/usr/lib/python2.7/site-packages/epics/ca.py", line 1214, in put
count = min(len(value), count)
TypeError: object of type 'int' has no len()
caput('SCANPV:scan1.P1PA',[1])

Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python2.7/site-packages/epics/init.py", line 81, in caput
return thispv.put(value, wait=wait, timeout=timeout)
File "/usr/lib/python2.7/site-packages/epics/pv.py", line 277, in put
callback_data=callback_data)
File "/usr/lib/python2.7/site-packages/epics/ca.py", line 377, in wrapper
return fcn(_args, *_kwds)
File "/usr/lib/python2.7/site-packages/epics/ca.py", line 1235, in put
data[0] = type(data[0])(value)
TypeError: float() argument must be a string or a number

So its seems the code here doesn't check for a non-array? I get the same error if I use the PV object.

Thanks,

Stephen Mudie
SAXS/WAXS Beamline
Australian Synchrotron

The following files were installed with pyepics, but they shouldn't have been:

/usr/lib/libCom.so
/usr/lib/libca.so

Instead, pyepics should use the system provided EPICS libraries.

Extra: these libraries caused problems when installing other unrelated packages with yum. Example follows attached.

yum-error-log.txt

I started getting this in linux using a pretty heavily multithreaded program. I don't know when it started happening, but it occures in 3.2.4.

running

import epics
import threading
epics.ca.use_initial_context()
t = threading.Thread(target = lambda : epics.ca.use_initial_context())
t.start()

causes a segfault.

I determined the cause was from the return value of current_context() as stored in initial_context isn't properly being interpreted as a pointer in ca_attach_context.

The fix is to include

libca.ca_attach_context.argtypes = [ctypes.c_void_p]

along with all of the other ctypes annotations in initialize_libca. This is also likely related to the segfault mentioned in issue #37

Hi Matt & others,

I noticed some of my long-running EPICS apps were leaking memory, seemed to be any that re-allocated PV objects on a regular basis.

Callbacks are added to ca._cache, and any callback that is a bound method holds a reference back to the object it's bound to. For any PV object, this is PV.__on_connect and PV.__on_changes. As a result, no PV object with callbacks is ever garbage collected even if it's otherwise unreachable.

This probably has other weird effects when those callbacks are called, I've noticed occasional wxPython "a dead C++ object was referenced here" errors before that can probably be traced back to this.

I've checked in 70b42fc to a branch, which stores any bound method as a weakreference (not as easy as it sounds because you can't make a weak reference directly to a bound method, hence the additional code that splits it into its bound object and the unbound method, and weakreferences the bound object.)

I contemplated expanding that further so all callbacks (bound & unbound methods) are stored as weakreferences, but it's not as obvious what is desirable circumstances for cleanup of unbound callbacks. For instance, if you call ca.create_channel() with a closure callback and then the closure falls out of scope, do you expect it to stay alive or become GCed? Unbound callbacks are a bit of a niche case regardless, though, as PV objects seem to be the usual way to go.

Please let me know what you think. This commit will break any existing code that relies on being able to create PV objects and then discard them, but have the callbacks called anyhow. I'd argue that's a bad practice to begin with, though.

• Angus

Since only the top-level path which contains setup.py is added to the search path during a source install, and the library source directory is named lib, import epics fails:

$ python setup.py develop
$ ipython

In [1]: import epics
ImportError: No module named epics

In [2]: import lib

In [3]: lib.__file__
Out[3]: '/Users/klauer/Repos/pyepics/lib/__init__.pyc'

Easy solution: rename lib to epics.

If a context is created in the main process and a subprocess is spawned, attempts to use channel access in the subprocess fail.

As a workaround, detaching the context in the main process, spawning the subprocess, and reattaching post-launch works (in addition to wiping some pyepics global state variables that were copied over when the process was forked).

This workaround unfortunately still leaves a possibility for missed monitor callbacks while loading the subprocess. If another thread were to be polling via some busy loop (for example) with the same context, they would effectively have their context pulled from underneath without some additional synchronization.

If the reason for detach/attach being necessary can be fixed:

• CA can use the process id as an additional key in the cache -- maybe the tuple (pid, context)
• PID can be checked when making wrapped ca calls, automatically clearing the cache
• Or a clear_state() function to wipe the pyepics state at the start of the new process can be called manually

If it can't be fixed:

• Block with a per-context 'wait-for-reattach' mutex around wrapped ca calls [and monitors are just lost regardless] -- but this may have performance implications
• Write documentation/examples to show how to best mitigate the problems associated with detach/attach
• ?

Example showing the issue:

from __future__ import print_function
import epics
import time
import multiprocessing as mp
import threading

PV = 'IOC:m1'
DETACH_REATTACH = 0


def caget(pv):
    return epics.caget(pv, timeout=1.)


def subprocess_thread():
    epics.ca.use_initial_context()
    print('-> subprocess thread %s=%s' % (PV, caget(PV)))


def subprocess():
    print('----subprocess--------')
    print('    clearing cache, monitors, etc.')
    epics._CACHE_.clear()
    epics._MONITORS_.clear()
    epics.ca._cache.clear()
    epics.ca._put_done.clear()
    epics.ca.libca = None
    epics.ca.initial_context = None

    print('    context is %s' % epics.ca.current_context())
    print('    %s=%s' % (PV, caget(PV)))
    try:
        print('    %s=%s' % (PV, caget(PV)))
        thread = threading.Thread(target=subprocess_thread)
        thread.start()
        thread.join()
    except Exception as ex:
        print('**', ex.__class__.__name, ex)


def main_process():
    def monitor(value=0, **kwargs):
        print('Monitor: value is', value)

    print('----main process-----')
    print('    context is %s' % epics.ca.current_context())
    print('    %s=%s' % (PV, caget(PV)))
    epics.camonitor(PV, callback=monitor)
    context = epics.ca.current_context()
    if DETACH_REATTACH:
        epics.ca.detach_context()

    try:
        p1 = mp.Process(target=subprocess)
        p1.start()
        p1.join()
    except KeyboardInterrupt:
        print('Killing subprocess')
        p1.terminate()

    if DETACH_REATTACH:
        epics.ca.attach_context(context)

    print('----main process-----')
    print('    context is %s' % epics.ca.current_context())
    print('    after process %s=%s' % (PV, caget(PV)))
    print('    context is', epics.ca.current_context())
    print()
    print()

    time.sleep(0.1)
    # after reattaching, monitors still work
    epics.caput(PV, 0)
    epics.caput(PV, 1)
    time.sleep(1.0)

if __name__ == '__main__':
    main_process()

If DETACH_REATTACH==0:

$ python mp_pyepics.py
----main process-----
    context is 156308488
    IOC:m1=1.0
----subprocess--------
    clearing cache, monitors, etc.
    context is 156308488
cannot connect to IOC:m1
    IOC:m1=None
cannot connect to IOC:m1
    IOC:m1=None
cannot connect to IOC:m1
-> subprocess thread IOC:m1=None
----main process-----
    context is 156308488
    after process IOC:m1=1.0
    context is 156308488


Monitor: value is 0.0
Monitor: value is 1.0

If DETACH_REATTACH==1:

$ python mp_pyepics.py
----main process-----
    context is 173302792
    IOC:m1=1.0
----subprocess--------
    clearing cache, monitors, etc.
    context is 171936344
    IOC:m1=1.0
    IOC:m1=1.0
-> subprocess thread IOC:m1=1.0
----main process-----
    context is 173302792
    after process IOC:m1=1.0
    context is 173302792


Monitor: value is 0.0
Monitor: value is 1.0

An epics.PV of a waveform with automonitor enabled can yield this error when first trying to use get(count=x) with this exception:

Traceback (most recent call last):
  File "mca_test.py", line 14, in <module>
    print(mca, mca.get(count=10))
  File "/home/nanopos/ecli/epics/pv.py", line 226, in get
    (count is not None and count > len(self._args['value']))):
TypeError: object of type 'int' has no len()

Here's an example script:

import epics

print(epics.caget('scaler1:mca1')[:10])

mca_records = [epics.PV('scaler1:mca%d' % i, auto_monitor=False) for i in range(1, 5)]

for mca in mca_records:
    print(mca, mca.get(count=10))

mca_records = [epics.PV('scaler1:mca%d' % i) for i in range(1, 5)]

for mca in mca_records:
    print(mca, mca.get(count=10))

And the raw output:

[0 0 0 0 0 0 0 0 0 0]
(<PV 'scaler1:mca1', count=10000/10000, type=long, access=read/write>, array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0]))
(<PV 'scaler1:mca2', count=10000/10000, type=long, access=read/write>, array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0]))
(<PV 'scaler1:mca3', count=10000/10000, type=long, access=read/write>, array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0]))
(<PV 'scaler1:mca4', count=10000/10000, type=long, access=read/write>, array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0]))
Traceback (most recent call last):
  File "mca_test.py", line 14, in <module>
    print(mca, mca.get(count=10))
  File "/home/nanopos/ecli/epics/pv.py", line 226, in get
    (count is not None and count > len(self._args['value']))):
TypeError: object of type 'int' has no len()

A possible fix would be to add a _cached_length property and compare the count to this:

      @property
      def _cached_length(self):
          """
          Length of cached value
          """
          try:
              return len(self._args['value'])
          except:
              return 1

As a note, though, this does not happen on all waveform records. This only appears to happen when used with slower CA servers such as those on VME, though using wait_for_connection and such had no effect.

Hi,
I'm monitoring some records and I was wondering why I got a status value of 1 in the callback while a caget on the command line gave me no alarm status. When creating a PV, form='time' and form='ctrl' give different values (with different meanings) in the pv.status attribute.

After digging around in the code I found the problem. In the file ca.py in the function _onMonitorEvent the status keyword is assigned the value of the status argument (Line 465). According to the EPICS Channel Access manual (http://www.aps.anl.gov/epics/base/R3-14/12-docs/CAref.html#ca_add_event) this is the status of the call (1 for sucess in caerr.h). The pyepics manual (http://pyepics.github.io/pyepics/pv.html#attributes) states that 1 is the ok value but this is not the alarm status of the PV.

Later in the function if someone uses form='time', the status keyword is overwritten with the record alarm status value (kwds['status'] = tmpv.status). This is what I would expect in all circumstances.

I propose to change line 465 to
kwds = {'ftype':args.type, 'count':args.count, 'chid':args.chid, 'pvname': pvname}
and line 475 to:
for attr in dbr.ctrl_limits + ('precision', 'units', 'status', 'severity'):
And change the manual accordingly.

The pv.status and pv.severity attributes (and the keywords in the callbacks) should always be the alarm values.

Hi, good to see there is still active development on this project.

Isee very rare exceptions coming from the ca thread due to a rare race condition between unregistering callbacks in other threads and calling all of the callbacks in a ca-thread loop:

    def run_callbacks(self):
        """run all user-defined callbacks with the current data

        Normally, this is to be run automatically on event, but
        it is provided here as a separate function for testing
        purposes.
        """
        for index in sorted(list(self.callbacks.keys())):
            self.run_callback(index)

Occasionally in this ca-thread loop, the index can have been unregistered from the program thread. Either a lock needs to be put around accesses to self.callbacks, or since most operations are atomic-enough, you can actually fix this with

    def run_callback(self, index):
        """run a specific user-defined callback, specified by index,
        with the current data
        Note that callback functions are called with keyword/val
        arguments including:
             self._args  (all PV data available, keys = __fields)
             keyword args included in add_callback()
             keyword 'cb_info' = (index, self)
        where the 'cb_info' is provided as a hook so that a callback
        function  that fails may de-register itself (for example, if
        a GUI resource is no longer available).
        """
        try:
            fcn, kwargs = self.callbacks[index]
        except KeyError:
            #key suddenly stopped existing, must have been unregistered
            return
        kwd = copy.copy(self._args)
        kwd.update(kwargs)
        kwd['cb_info'] = (index, self)
        if hasattr(fcn, '__call__'):
            fcn(**kwd)

For personal interest, I'm actually using this library through UChicago physics. I should come say Hi sometime as it looks like the main development is though the university?

doing
p = PV(name)
print p.units

results in None. Waiting a short time gives the correct value.

We should wait around for CTRL attributes to be returned. This might be related to issue 4 (using a get_callback for CTRL fields).

Got this error today:

Exception in thread Thread-630:
Traceback (most recent call last):
  File "C:\DATEN\Programme\WinPython-32bit-3.3.5.0\python-3.3.5\lib\threading.py", line 901, in _bootstrap_inner
    self.run()
  File "C:\DATEN\Programme\WinPython-32bit-3.3.5.0\python-3.3.5\lib\threading.py", line 858, in run
    self._target(*self._args, **self._kwargs)
  File "wfh.py", line 1126, in to_epics
    #print(i)
  File "C:\DATEN\Programme\WinPython-32bit-3.3.5.0\python-3.3.5\lib\site-packages\pyepics-3.2.3-py3.3.egg\epics\pv.py", line 282, in put
    callback_data=callback_data)
  File "C:\DATEN\Programme\WinPython-32bit-3.3.5.0\python-3.3.5\lib\site-packages\pyepics-3.2.3-py3.3.egg\epics\ca.py", line 404, in wrapper
    chid, timeout))
TypeError: %d format: a number is required, not c_long

I think there is an issue with the data type "c_long" while formating the exception string.

Anyway I think there is a missing parameter (chid) in the fmt-string (ca.py, line 397ff):

    timeout = kwds.get('timeout', DEFAULT_CONNECTION_TIMEOUT)
    fmt ="%s: timed out waiting for chid to connect (%d seconds)"
    if not connect_channel(chid, timeout=timeout):
        raise ChannelAccessException(fmt % (fcn.__name__,
                                            chid, timeout))

If I connect to a PV, and print the control limits within a callback, it works correctly the first time I do it:

import epics
def show_lims(upper_ctrl_limit=None, lower_ctrl_limit=None, units=None, *args, **kws):
    print(upper_ctrl_limit)
    print(lower_ctrl_limit)
    print(units)
pv = epics.PV("MOTOR:1:VAL", callback=show_lims, form='ctrl')

As soon as the PV connects, the callback fires for the first time, and I get the right info:
180.0
-180.0
deg

But the next time the callback fires, it reports that the limits and units have changed:
0.0
0.0

Using the caget command line utility shows that the control limits and units haven't actually changed. Is this the expected behavior?

callbacks for access_rights on Python2 + Window64 are wrong. I think this is probable due to problems similar to #31 and probably requires a special case at epics/dbr.py#L348

There is a missing functionality that would allow user to disable monitors on a specific CA channel/PV. In general this rarely needed, but it is very important when dealing with fast sources that generate large arrays (for example cameras and fast digitizers) very often (for us ~7MB large arrays at 10Hz). If the monitors are not disabled when not in use a lot of unnecessary network load as well as sources CPU load are generated.

A work-around is to disconnect from the channel completely when it is not in use, however that is precisely what spawned the issue #68 . In addition this work around has a side effects in terms of generating additional broadcast searches as well larger latency compared to having channels persistently connected...

Hallo,

we are using pyepics for monitoring large waveforms (25Msamples, 16bit, @ 1Hz or 20Msampels, 16bit @ 2Hz). In general we get good performance by activate auto monitoring and adding a callback to receive the data.
But in order to reduce the network load it was beneficial to tell the subscription how many samples are actually required.

In fact the calib already supports this via the count argument

    ret = libca.ca_create_subscription(ftype, count, chid, mask,
                                       _CB_EVENT, uarg, ctypes.byref(evid))

So we modified ca.py and pv.py to support this argument.

Would it be possible to add this to an official release?
patch_pyepics3.2.6_count.txt

This is really a reminder for myself: The continuous integration with Travis-CI does not test on Windows. And Windows, especially with 64-bit Python, always, always needs separate testing.

PR #57 introduced a severe bug with 64-bit Python on Windows that I didn't catch.

When used in a multithreaded application where one thread takes the global interpreter lock (GIL) right after the start of a caget() request, thread scheduling can cause the request to incorrectly timeout. In the caget thread in pv.__init__, when various ca CDLL calls are made, the GIL is automatically released as per the ctypes documentation.

This is certainly a corner case currently with a simple mitigation technique (but no simple solution that I can come up with, without some locks to complicate things). Setting start_time after __init__.py:58, to start counting after the non-blocking connect() call should suffice:

51 def __create_pv(pvname, timeout=5.0):          
52     "create PV, wait for connection: "         
53     if pvname in _CACHE_:                      
54         return _CACHE_[pvname]                 
55                                                
56     start_time = time.time()                   
57     thispv = PV(pvname)                        
58     thispv.connect()                           

Although I have not tested it, a simple search indicates that this may be an issue as well in ca.py:connect_channel (and its call to current_context).

Now a thread holding the GIL for 5 seconds (the default connection timeout) is rather unlikely. This is really a topic for another issue, but I’d like to propose a ‘connection_timeout’ kwarg for caget/caput, as there is currently no way of doing that without resorting to using your own PV instance. I made this a local modification, lowering the connection timeout, making this issue much more apparent.

test_xarray2 & test_xarray3 seem to fail on master. I thought this was caused by my latest spate of changes but it seems it might have been something else.

I'm happy to fix, just wanted to check desired behaviour.

It seems like what's expected is that calls to ca.get(pv, count=0) that return array types then the returned array should be truncated to only the non-zero values. Is this right? So "0" is a special standin value for ca.get()

Do we want to keep this behaviour? I'm happy to make whatever changes are desirable, just want to understand the use case.

When I reload all my modules in an embedded Python interpreter, including the epics module, then I get following errors:

Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/local/lib/python2.7/dist-packages/pyepics-b7d26fe9a057be74348eee93a3cab7ab51636d51-py2.7.egg/epics/ca.py", line 456, in _onMonitorEvent
if dbr.PY64_WINDOWS: args = args.contents
AttributeError: 'NoneType' object has no attribute 'PY64_WINDOWS'

Hence, the global dbr object as imported in ca.py becomes None for some reason.

The channel values that I am monitoring still come through. But the control vars don't.
The pv.get_ctrlvars(timeout=0.1) returns None, while orginally after the initial load of the modules this works well.

Anyone a clue?