beanstalkd / pybeanstalk Goto Github PK
View Code? Open in Web Editor NEWPython beanstalkd client implementations (single-threaded/select/twisted)
Python beanstalkd client implementations (single-threaded/select/twisted)
PyBeanstalk: Copyright 2008-2015 A client library for beanstalkd. beanstalkd is a lightweight job queuing daemon based on libevent. It is meant to be used across multiple systems, and takes inspiration from memcache. More details on beanstalk can be found at: http://xph.us/software/beanstalkd/ This client library aims to be simple and extensible. It provides both a single thread, single connection serialized (select-based) beanstalk connection with optional, simple thread pool implementation, and a basic Twisted client. They can be used directly, or be used as basis for for more sophisticated client applications. Please see the examples directory for usage examples. To install, just run python setup.py install from this directory. For the Twisted client, install the 'twisted' extra (python setup.py install [twisted]). Please see the examples directory for usage examples. The package home is at https://github.com/beanstalkd/pybeanstalk, with an issue tracker and a wiki. Issue reports and pull requests most welcome. Copyright notice: This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program (In file named COPYING). If not, see <http://www.gnu.org/licenses/>.
As suggested by @mahmoudimus; see https://pypi.python.org/pypi/tox.
When importing the pybeanstalk
package in any possible manner, the following exception is raised:
Traceback (most recent call last):
File "local_server.py", line 1, in <module>
from war import app
File "C:\war\war.py", line 3, in <module>
from beanstalk import job
File "C:\{SomeFolders}\Python35\lib\site-packages\beanstalk\__init__.py", line 1, in <module>
import serverconn
ImportError: No module named 'serverconn'
By "any possible manner", I'm mean I tried the following import
statements in my C:\war\war.py
file:
import beanstalk
from beanstalk import job, serverconn
from beanstalk import serverconn
I don't understand what is the issue. Maybe this is related to relative imports in Python 3?
Hi,
I get an error when trying to run the example simple_clients.py using beanstalkd 1.4.4:
kangaroo:examples cbare$ python simple_clients.py producer 127.0.0.1
handling args
setting up connection
No handlers could be found for logger "beanstalk.serverconn"
starting producer loop
This is data to be consumed (0)!
usage: example.py TYPE server [port]
TYPE is one of: [producer|consumer]
Traceback (most recent call last):
File "simple_clients.py", line 53, in
main()
File "simple_clients.py", line 41, in main
producer_main(connection)
File "simple_clients.py", line 15, in producer_main
data.Queue()
File "/Library/Python/2.6/site-packages/beanstalk/job.py", line 107, in Queue
self.Server.put(self._serialize(), self.pri, self.delay, self.ttr)
File "/Library/Python/2.6/site-packages/beanstalk/serverconn.py", line 51, in caller
return self._do_interaction(_res(_args, **kw))
File "/Library/Python/2.6/site-packages/beanstalk/serverconn.py", line 99, in _do_interaction
return self._get_response(handler)
File "/Library/Python/2.6/site-packages/beanstalk/serverconn.py", line 90, in _get_response
res = handler(recv)
File "/Library/Python/2.6/site-packages/beanstalk/protohandler.py", line 133, in call
return self.h(val)
File "/Library/Python/2.6/site-packages/beanstalk/protohandler.py", line 153, in handler
checkError(response)
File "/Library/Python/2.6/site-packages/beanstalk/errors.py", line 37, in checkError
raise err
beanstalk.errors.ExpectedCrlf: Server returned: EXPECTED_CRLF
Exception beanstalk.errors.UnexpectedResponse: UnexpectedResponse('Response was: WN_COMMAND ',) in <bound method Job.__del of <beanstalk.job.Job object at 0x101163890>> ignored
These are supported in beanstalkd since a long time ago:
The test test_MultiServerConn.py
produces the following output a number several times (on Debian stable, newest Python 2.x.x with beanstalkd from debian pkg):
Exception ServerInUse: ServerInUse('AsyncServerConn(127.0.0.1:11303#[Open][Waiting]) is currently in use!', <0x139986747569216 AsyncServerConn(127.0.0.1:11303#[Open][Waiting])>) in method __del__ of <beanstalk.job.Job object at 0x0000000001ea3298> ignored
It's always the connection to 11303, with a different Job object each time.
There's also one error and one failure:
ERROR: test_MultiServerConn.test_tube_operations
----------------------------------------------------------------------
Traceback (most recent call last):
File "/opt/pythonapps/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/media/psf/Code/thirdparty/pybeanstalk/tests/test_MultiServerConn.py", line 296, in test_tube_operations
"Was expecting %s, got %s" % (expecting,
NameError: global name 'expecting' is not defined
======================================================================
FAIL: test_MultiServerConn.test_reserve_timeout_works
----------------------------------------------------------------------
Traceback (most recent call last):
File "/opt/pythonapps/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/media/psf/Code/thirdparty/pybeanstalk/tests/test_MultiServerConn.py", line 323, in test_reserve_timeout_works
assert conn.stats()['data']['current-jobs-ready'] == 0, "The server is not empty "\
AssertionError: The server is not empty of jobs so test behaviour cannot be guaranteed. Bailing out.
How about https://travis-ci.org ? @kr, to set it up for pybeanstalk, Travis CI needs permission to access beanstalk organization repositories; hope you're ok with that.
Currently, the Twisted client tests cannot be run using nose. Help would be appreciated; it probably just needs some debugging by someone who knows Twisted and nose better than I.
Another alternative would be switching to pytest for all tests, under which I know I can make the Twisted client tests run.
Currently, none exist.
Looks like this repository is not that much maintained, and there are many forks with some improvements.
It would be good if we could roll the pybeanstalk improvements into one "canonical" repository and publish the package on PyPI.
@sophacles , would it be ok for you to move the repository to https://github.com/beanstalkd and encourage forkers maybe join the organization as well? I could ask Keith Rarick @kr (author of beanstalkd) if it would be ok for him.
Twisted 15.2.0 introduced a new improved logging framework. Make the pybeanstalk Twisted client use it.
In addition to people already here, there are forks by @dustin, @pragnesh, @AlekSi, @rubenk, @gerry, @jasiek @labjack, @kaulie, @wuguobao and @ZoomQuiet.
This issue is an attempt to reach them via GitHub mentions and let them know that pybeanstalk has been transferred to the beanstalkd GitHub organization, and, that issues and pull requests are most welcome!
Please, if your GitHub username is on the above list and you see this, comment on this issue and let us know what you'd like to see happen to pybeanstalk - and whether you'd perhaps even like to join the GitHub organization to help improve pybeanstalk.
In _get_response, error msg refers to server.ip and server.port, which do not exist. They should instead refer to self.server and self.port.
Special method del is called by Python when the last reference of the item is being destroyed. See https://docs.python.org/2/reference/datamodel.html#object.__del__.
Given that, I don't understand why there's del implementation at https://github.com/beanstalkd/pybeanstalk/blob/master/beanstalk/job.py#L65-L66 that calls self.Finish, in other words the Job is trying to delete itself from beanstalk server???
I can understand someone might somewhat mistakenly try to support deleting a job by del job
, but this is dangerous: an application may have more references hanging around when del is called, and on the other hand, there are cases where the reference ends up destroyed inadvertently.
Apparently this has been noticed before as well...
I am not aware of pybeanstalk using the __del__
implementation internally. I will remove the __del__
implementation unless someone objects for a good reason.
The BeanStalk
client class uses __getattr__
to call protocol implementations. This however obfuscates/breaks access to Python __nonzero__
and zope.interface
__conform__
special methods of the BeanStalk
instance passed as the conn
parameter to the job.Job
constructor. This results in an error when trying to use job.Job with Twisted.
It would be convenient if the tests could be run without having to descend to tests/ . Right now it's not possible because of how config.py looks for the tests.cfg file.
Could you please help in documenting how the tests are to be run? I just tried to run the tests by going to the tests directory and running nosetests. It looks like that is the way to go, but I got some errors.
I am not really familiar with nose, but taking a look at some of the files, I understand some of the tests require a live beanstalkd server (or more than one server). That caused at least some of the errors, perhaps even all of them.
Perhaps we could split the tests into three groups (at least in the documentation):
So we could ask pull request submitters to provide and run tests without forcing them to set up multiple beanstalk servers just because some tests require them.
This should be easy by using nose to run just some of the tests, either manually by module or test name, or perhaps using some nose-specific criteria for grouping?
update package structure, switch to setuptools, etc.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.