It would be nice if MongoClient aliased getattr to getitem to better support dynamic db/collection naming.

with mongo[database][collection].find() as cursor:

vs.

with getattr(getattr(mongo, database), collection).find() as cursor:

Issue Type

[x] Bug (Typo)

Steps to Replicate and Expected Behaviour

• Examine diesel/core.py and observe specfied, however expect to see specified.
• Examine doc/patterns.rst and observe respones, however expect to see response.
• Examine diesel/core.py and observe othewise, however expect to see otherwise.
• Examine doc/patterns.rst and observe explictly, however expect to see explicitly.
• Examine diesel/core.py and observe delimitted, however expect to see delimited.
• Examine doc/intro.rst and observe concurreny, however expect to see concurrency.
• Examine diesel/protocols/wsgi.py and observe bootsraps, however expect to see bootstraps.
• Examine doc/intro.rst and observe avaiable, however expect to see available.

Notes

Semi-automated issue generated by
https://github.com/timgates42/meticulous/blob/master/docs/NOTE.md

To avoid wasting CI processing resources a branch with the fix has been
prepared but a pull request has not yet been created. A pull request fixing
the issue can be prepared from the link below, feel free to create it or
request @timgates42 create the PR. Alternatively if the fix is undesired please
close the issue with a small comment about the reasoning.

https://github.com/timgates42/diesel/pull/new/bugfix_typos

Thanks.

The reddit group is not used/useful and everything that happens on irc is lost in the ether... hard for a community to build up when there's limited means of communication.

Hi there,

Diesel seems a really interesting project, such a shame it is not available for python3 !
So I've forked the repo (https://github.com/touilleMan/diesel/tree/py3k) and started to port the code ;-)

So far, all the tests are passing for Python 2.6, 2.7, 3.3 and 3.4 using the same codebase, but it seems those tests are not really exhaustive and there is still some work to do !

Now I'm trying to make work all the examples available.

My main point of concern is how to handle the strings:

• in Python2, str is an array of bytes (bytes type is an alias of str), to use unicode we must is unicode type
• in Python3, str is an array of unicodes characters and only bytes an array of bytes

Considering Diesel, all the code is using classical str types (i.e. array of bytes), this seems pretty logical to me given there is no standard way to determine the encoding of data coming from a socket.

The downside of this is to make the code compatible with Python3, all the strings needs to be explicitly converted to bytes, leading to a lot of small changes, an code more complex (numerous conversions unicode => bytes).

Example:

# Python2 only
send('AUTH', self.password)
send("HTTP/%s %s %s\r\n" % (('%s.%s' % version), resp.status_code, resp.status))
# Python2&3
send(b'AUTH', self.password.encode())
send(("HTTP/%s %s %s\r\n" % (('%s.%s' % version), resp.status_code, resp.status)).encode())

I've started to see if the core.send/core.receive function could be used to handle encoding, but - although it would be a nice solution - I don't believe this could lead to a easy drop-in replacement.

def send(data, priority=5, encoding='ascii'):
    if isinstance(data, builtins.str):
        data = data.encode(encoding)
    return current_loop.send(data, priority=priority)

def receive(spec=None, encoding='ascii'):
    data = current_loop.input_op(spec)
    if isinstance(data, builtins.bytes):
        data = data.decode(encoding)
    return data

Do you have any advice/idea about the best way to handle this ?

The constructor makes it a defaultdict, but parse makes it a dict. Should that latter be a defaultdict as well?

https://github.com/jamwt/diesel/blob/master/diesel/protocols/http.py#L99

at least that I can see.

Unless it's going to be built out further, we should remove diesel.convoy. It's confusing for it to be there in sort of a half-baked state.

Just a note for upstream fix. On mac os x 10.7.3 with python 2.7.1,

pip install pyev

fails with a permissions error on configure. The fix is:

chmod 755 configure

This will prevent pyev from installing cleanly, and subsequently diesel from running.

Ok, I can add a new service and loop into Application, but and if I want to remove just one service?

# cd "$(mktemp -d)"                        

# virtualenv fooo

# python --version
Python 3.6.6

# source fooo/bin/activate

# pip install https://github.com/dieseldev/diesel/archive/master.tar.gz
[..]
Successfully installed Jinja2-2.10 MarkupSafe-1.1.0 Werkzeug-0.14.1 asn1crypto-0.24.0 cffi-1.11.5 click-7.0 cryptography-2.4.2 diesel-3.0.24 dnspython-1.16.0 flask-1.0.2 greenlet-0.4.15 http-parser-0.8.3 idna-2.8 itsdangerous-1.1.0 pycparser-2.19 pyopenssl-18.0.0 six-1.12.0 twiggy-0.4.7

# python -c 'import diesel'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/tmp.3TqpDbqaeU/fooo/lib/python3.6/site-packages/diesel/__init__.py", line 2, in <module>
    from logmod import log, levels as loglevels, set_log_level
ModuleNotFoundError: No module named 'logmod'

The from logmod would ne to be from diesel.logmod I believe.

Related: #95

I'm running a local mirror of several python packages, and the latest diesel tarball appears to cause our mirror to fail with the following error:

ValueError(u'No PKG-INFO in archive: /opt/packages/private/diesel-3.0.24.tar.gz',)

It seems all the other packages we mirror have this file, including an older version -- diesel-3.0.16.tar.gz

Seems this manifest is missing from the current tarball?

The PKG-INFO is present in the python.org zip, but not the github tarball

$ wget https://pypi.python.org/packages/source/d/diesel/diesel-3.0.24.zip
$ unzip -l diesel-3.0.24.zip | grep PKG
     1274  2015-03-14 20:11   diesel-3.0.24/PKG-INFO
     1274  2015-03-14 20:11   diesel-3.0.24/diesel.egg-info/PKG-INFO
$ wget https://github.com/dieseldev/diesel/archive/diesel-3.0.24.tar.gz
$ tar -tzf 3.0.23.tar.gz | grep PKG
$

Hi,

from diesel import response
ImportError: cannot import name response

I am getting an exception while working through the examples. Has request moved or renamed?

Regards,
Rahul

from twiggy import log as olog, addEmitters, levels, outputs, formats, emitters
ImportError: cannot import name addEmitters

in file /usr/local/lib/python2.7/dist-packages/diesel/logmod.py

need to be changed to add_emitters to match logmod namespaces
works fine.

Steps:

extquery_prepare("SELECT count(*) FROM foo WHERE bar IN $1", "foo")
extquery(([1,2,3],), "foo")

Traceback:

File ".../diesel/protocols/pgsql.py", line 276, in <genexpr>
    params = ''.join(struct.pack('!I%ss' % len(p), len(p), p) for p in self.params)
error: argument for 's' must be a string

Waiting on background events doesn't work in dpython or idpython. I'm pretty sure that both are calling raw_input() and blocking the event loop.

Need to investigate using diesel.util.stream and fitting that into the diesel.interactive code (which dpython and idpython use).

the http examples don't work with pyparser, you must have had Cython when you install http_parser for them to work

I couldn't find the flask example on diesel.io homepage in git or I'd have diffed it and issued a pull request. At any rate, it's missing:

from diesel.util.queue import deque

at the beginning

and

app.run(debug=True)

at the end

Diesel website is not opening.

E:>python test.py
Traceback (most recent call last):
File "test.py", line 1, in
from diesel import Application, Service, until_eol, send
File "E:\Python\python-2.7.6\lib\site-packages\diesel__init__.py",
line 7, in
from app import Application, Service, UDPService, quickstart, quickstop, Thu
nk
File "E:\Python\python-2.7.6\lib\site-packages\diesel\app.py", line
10, in
from diesel.hub import EventHub
File "E:\Python\python-2.7.6\lib\site-packages\diesel\hub.py", line
18, in
import fcntl
ImportError: No module named fcntl

Where can I find the documentation of this project? dieselweb.org redirects to github.com/jamwt/diesel/

I am seeing a problem where if I do a lot of reads and updates at the same time, a read will fail with an empty deque error in MongoIter.next.

hi I use this code
https://bitbucket.org/sjl/diesel/src/4f9dff1bb14679d55f96d46dc58c7543d39b5174/examples/chat.py?at=default
but when I try telnet the Connection closed by foreign host. I think It happen because of yield . when I remove It from code telnet work fine . but the aplication dosen't work right!
can you give me a simple example of chat server with diesel?

(Using Ubuntu 12.10 and packages from the official apt repositories)

When using WSGIResource/Site in order to run a DieselFlask app (in these examples, "app"), you'll always end up with getting some exception from Diesel and Twisted's internal code.

from twisted.internet import reactor
from twisted.web.wsgi import WSGIResource
from twisted.web.server import Site

resource = WSGIResource(reactor, reactor.getThreadPool(), app)
site = Site(resource)

reactor.listenTCP(8080, site)
reactor.run()

With this method (app is a simple "hello world" route on /), going to / throws this exception:

WSGI application error
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 167, in _worker
    result = context.call(ctx, function, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
  File "/usr/lib/python2.7/dist-packages/twisted/web/wsgi.py", line 332, in run
    self.reactor.callFromThread(wsgiError, self.started, *exc_info())
--- <exception caught here> ---
  File "/usr/lib/python2.7/dist-packages/twisted/web/wsgi.py", line 315, in run
    appIterator = self.application(self.environ, self.startResponse)
  File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1518, in __call__
    return self.wsgi_app(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1502, in wsgi_app
    with self.request_context(environ):
  File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1463, in request_context
    return RequestContext(self, environ)
  File "/usr/lib/python2.7/dist-packages/flask/ctx.py", line 87, in __init__
    self.url_adapter = app.create_url_adapter(self.request)
  File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1348, in create_url_adapter
    return self.url_map.bind_to_environ(request.environ,
exceptions.AttributeError: 'dict' object has no attribute 'environ'

After several tries, @dontcare4free managed to make a WSGI middleware that fixes that case. More specifically:

from twisted.internet import reactor
from twisted.web.wsgi import WSGIResource
from twisted.web.server import Site

from flask.wrappers import Request
from UserDict import UserDict

class _RequestDict(UserDict, Request):
    def __init__(self, environ):
        Request.__init__(self, environ)
        UserDict.__init__(self)

    @property
    def data(self):
        return self.environ

    @data.setter
    def data(self, data):
        pass  # Bad, but required for this

def environFixerMiddleware(f):
    def wrapper(environ, start_response):
        return f(_RequestDict(environ), start_response)

    return wrapper

resource = WSGIResource(reactor, reactor.getThreadPool(), environFixerMiddleware(app))
site = Site(resource)

reactor.listenTCP(8080, site)
reactor.run()

This works for a simple "hello world" example. However, when throwing more complicated stuff like WebSockets and such (taken from examples) into the cocktail, another exception arises, which ends up in the actual code that handles exceptions:

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 167, in _worker
    result = context.call(ctx, function, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
  File "/usr/lib/python2.7/dist-packages/twisted/web/wsgi.py", line 332, in run
    self.reactor.callFromThread(wsgiError, self.started, *exc_info())
--- <exception caught here> ---
  File "/usr/lib/python2.7/dist-packages/twisted/web/wsgi.py", line 315, in run
    appIterator = self.application(self.environ, self.startResponse)
  File "avenir-api/hack.py", line 19, in wrapper
    return f(_RequestDict(environ), start_response)
  File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1518, in __call__
    return self.wsgi_app(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1506, in wsgi_app
    response = self.make_response(self.handle_exception(e))
  File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1194, in handle_exception
    self.log_exception((exc_type, exc_value, tb))
  File "/usr/local/lib/python2.7/dist-packages/diesel/web.py", line 61, in log_exception
    self._logger.trace().error('Exception on {0} [{1}]',
    exceptions.AttributeError: 'NoneType' object has no attribute 'trace

Not really sure whether this is an issue with Diesel or Twisted, though.

Would be awesome :)

Love the look of the framework, coming from other languages & frameworks, and I know a bit of python, but I'm starting my first full python project & learning the diesel framework at the same time and more framework documentation would be great!

The examples are definitely helping though

It would be very useful if you could use the same style for both connection methods.

Locally I'm emulating this by using a queue for every "connection" and a udp datagram send/receive loop that will create a new loop if a datagram is received from an unknown host, and then queue messages to that loop when further messages are received. Those loops then can send back data using a queue that the send/receive loop checks, and a modified diesel.send that allows sending already constructed datagrams (ie: not setting the address to self.parent.remote_addr)

it would be great if the framework abstracted that away for me so that a loop is created when an unknown host sends a udp datagram and within that loop I could use something along the lines of diesel.receive(diesel.datagram) and diesel.send(message), and just know the datagram is from that current "connection" and that diesel.send will respond to the correct addr/socket.

This would be especially useful when the server may be sending messages that are not in direct response to a received message.

A dieselweb.org website would be great, actually. There's a lot of competition in the async Python library space. I would feel better about diesel if it were more ... presentable.

It's trying to typecast to int, e.g.

And I can't turn it off.

This line in hub.py raises an AttributeError.

AttributeError: 'module' object has no attribute 'version'

I am using pyev==0.9.0 which was released this year (after 2 years of being stale) so I imagine this is a breaking change on their part. Quickest fix was to just hard code the version on my local install.

Traceback:

File ".../diesel/protocols/mongodb.py", line 302, in __exit__
    raise RuntimeError("need to cleanup cursor!")
RuntimeError: need to cleanup cursor!

The PyPI entry has no PyPI-hosted downloads of Diesel -- https://pypi.python.org/pypi/diesel/3.0.16

Currently it's using a download URL to an external site, but pip seems to be deprecating that.

vagrant@ubuntu:~/dev/memcacheliked$ .tox/py26/bin/pip install diesel
Downloading/unpacking diesel
  You are installing an externally hosted file. Future versions of pip will default to disallowing externally hosted files.
  You are installing a potentially insecure and unverifiable file. Future versions of pip will default to disallowing insecure files.
  Error <urlopen error [Errno -3] Temporary failure in name resolution> while getting http://jamwt.com/diesel/diesel-3.0.16.tar.gz (from https://pypi.python.org/simple/diesel/)
...

Bonus points for uploading wheels and/or eggs.

pip installation log:

Downloading/unpacking diesel==3.0.16 (from -r /opt/nagios-api/requirements.txt (line 5))
  Could not fetch URL https://pypi.python.org/simple/diesel/3.0.16: HTTP Error 404: Not Found
  Will skip URL https://pypi.python.org/simple/diesel/3.0.16 when looking for download links for diesel==3.0.16 (from -r /opt/nagios-api/requirements.txt (line 5))
  Ignoring link https://pypi.python.org/packages/source/d/diesel/diesel-2.1.0.tar.gz#md5=bf52eb9c2d51fedc667348af613f3250 (from https://pypi.python.org/simple/diesel/), version 2.1.0 doesn't match ==3.0.16
  Ignoring link https://pypi.python.org/packages/source/d/diesel/diesel-2.1.1.tar.gz#md5=0184b336fea8f5d4482befde5716ce14 (from https://pypi.python.org/simple/diesel/), version 2.1.1 doesn't match ==3.0.16
  Ignoring link https://pypi.python.org/packages/source/d/diesel/diesel-3.0.5.tar.gz#md5=c70f7e096e1722b9b4eecd60af1fdf1a (from https://pypi.python.org/simple/diesel/), version 3.0.5 doesn't match ==3.0.16
  Ignoring link http://download.dieselweb.org/diesel-0.9.0b.tar.gz (from https://pypi.python.org/simple/diesel/), version 0.9.0b doesn't match ==3.0.16
  Ignoring link http://download.dieselweb.org/diesel-0.9.1b.tar.gz (from https://pypi.python.org/simple/diesel/), version 0.9.1b doesn't match ==3.0.16
  Ignoring link http://jamwt.com/diesel/diesel-2.0.0.tar.gz (from https://pypi.python.org/simple/diesel/), version 2.0.0 doesn't match ==3.0.16
  Ignoring link http://jamwt.com/diesel/diesel-2.1.0.tar.gz (from https://pypi.python.org/simple/diesel/), version 2.1.0 doesn't match ==3.0.16
  Ignoring link http://jamwt.com/diesel/diesel-2.1.1.tar.gz (from https://pypi.python.org/simple/diesel/), version 2.1.1 doesn't match ==3.0.16
  Ignoring link http://jamwt.com/diesel/diesel-3.0.5.tar.gz (from https://pypi.python.org/simple/diesel/), version 3.0.5 doesn't match ==3.0.16
  HTTP error 403 while getting http://jamwt.com/diesel/diesel-3.0.16.tar.gz (from https://pypi.python.org/simple/diesel/)
  Could not install requirement diesel==3.0.16 (from -r /opt/nagios-api/requirements.txt (line 5)) because of error HTTP Error 403: Forbidden
Could not install requirement diesel==3.0.16 (from -r /opt/nagios-api/requirements.txt (line 5)) because of HTTP error HTTP Error 403: Forbidden for URL http://jamwt.com/diesel/diesel-3.0.16.tar.gz (from https://pypi.python.org/simple/diesel/)

Looks like problem is caused by user agent set by urlib.

Next request works:
curl -L -o /dev/null -v http://jamwt.com/diesel/diesel-3.0.16.tar.gz

This one doesn't work:
curl -H "User-Agent: Python-urllib" -v http://jamwt.com/diesel/diesel-3.0.16.tar.gz

<title>Access denied | jamwt.com used CloudFlare to restrict access</title>
...
<h2 data-translate="what_happened">What happened?</h2>
            <p>The owner of this website (jamwt.com) has banned your access based on your browser's signature (171b9664133c03d6-ua48).</p>

Title says it all. I couldn't think of other ways to contact you so

Hi Diesel,
How can I have client user's information on Diesel and ofcourse in server side.
I want informations like: ip,port,user,...
I want to write user's presence for chat application.

The title is self explanatory... I think it would be cool to mention the github repo for issues and people that want to follow the development/download a dev version.

$ dpython
[2013/03/19 05:40:35] {diesel} WARNING:Starting diesel <hand-rolled select.epoll>
Python 2.7.1+ (r271:86832, Sep 27 2012, 21:12:17)
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import diesel
>>> diesel.wait('for-something')
^C[2013/03/19 05:40:57] {diesel} WARNING:-- KeyboardInterrupt raised.. exiting main loop --
[2013/03/19 05:40:57] {diesel} INFO:Ending diesel application
$

This doesn't happen in the standard Python interpreter when a blocking operation is in progress. It just interrupts the operation and you can continue on your merry way.

$ python
Python 2.7.1+ (r271:86832, Sep 27 2012, 21:12:17)
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import time
>>> time.sleep(10)
^CTraceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyboardInterrupt
>>>

In the tutorial the Queues section has the following code in each consumer before the while loop.

item = queue.get()

This item doesn't get processed and confused me. After running the code, realized it was just a typo.

http://dieselweb.org is not working and everywhere that mention diesel points to it.

If I pass a unicode object to simplequery, I get an error from line 111 of pgsql.py. Traceback:

File ".../diesel/protocols/pgsql.py", line 111, in send
  tosend = self.idbyte + struct.pack('!I', len(allbytes)+4) + allbytes
UnicodeDecodeError: 'ascii' codec can't decode byte 0xa1 in position 4: ordinal not in range(128)

This is because the struct.pack() output is being implicitly coerced to ascii by the concatenation with the unicode allbytes. Presumably the rest of the module should be checked for this bug as well. The workaround is to encode sql to str before passing to simplequery, like so:

sql = sql.encode('ascii')

I am trying to install diesel in widows xp but I faced following error:
cc1.exe : error : unrecognized command line option ' -mon-cygwin'
error: setup script exited with error : command 'gcc' failed with exit status 1
my friends have already installed it easily in linux and believed that it is because of windows. Have I to go to linux?

Might actually be more than the example ...

$ python examples/http_client.py
[2013/03/20 06:22:32] {diesel} WARNING:Starting diesel <hand-rolled select.epoll>
[2013/03/20 06:22:32] {http-client} INFO:<Response 34645 bytes [200 OK]>
[2013/03/20 06:22:32] {diesel} ERROR:-- Unhandled Exception in local loop <<function req_loop at 0x2e0fb18>> --
Traceback (most recent call last):
  File "/home/christian/src/bump/server/contrib/diesel/diesel/core.py", line 189, in run
    self.loop_callable(*self.args, **self.kw)
  File "examples/http_client.py", line 15, in req_loop
    log.info(str(client.request('GET', '/', heads)))
  File "/home/christian/src/bump/server/contrib/diesel/diesel/core.py", line 141, in __call__
    raise ClientConnectionClosed(str(e), addr=self.client.addr, port=self.client.port)
ClientConnectionClosed: Connection closed by remote host (addr=www.jamwt.com, port=80)
^C[2013/03/20 06:22:34] {diesel} WARNING:-- KeyboardInterrupt raised.. exiting main loop --
[2013/03/20 06:22:34] {diesel} INFO:Ending diesel application

Currently the TCP transport supports the concept of priority when sending. This bubbles up into the diesel.send function. Other transports have a dummy/unused priority kwarg to comply with diesel.send/the TCP implementation.

Instead of each transport being forced to support options that don't map directly to it, each should support receiving a dictionary of kwargs from diesel.send and acting on values that may exist there.

The PostgresClient constructor doesn't require an explicit port. It'd be nice if the MongoClient constructor did the same.