Giter VIP home page Giter VIP logo

aiomysql's Introduction

aiomysql

https://github.com/aio-libs/aiomysql/actions/workflows/ci-cd.yml/badge.svg?branch=master Code coverage Latest Version Documentation Status Chat on Gitter

aiomysql is a "driver" for accessing a MySQL database from the asyncio (PEP-3156/tulip) framework. It depends on and reuses most parts of PyMySQL . aiomysql tries to be like awesome aiopg library and preserve same api, look and feel.

Internally aiomysql is copy of PyMySQL, underlying io calls switched to async, basically yield from and asyncio.coroutine added in proper places)). sqlalchemy support ported from aiopg.

Documentation

https://aiomysql.readthedocs.io/

Basic Example

aiomysql based on PyMySQL , and provides same api, you just need to use await conn.f() or yield from conn.f() instead of calling conn.f() for every method.

Properties are unchanged, so conn.prop is correct as well as conn.prop = val.

import asyncio
import aiomysql


async def test_example(loop):
    pool = await aiomysql.create_pool(host='127.0.0.1', port=3306,
                                      user='root', password='',
                                      db='mysql', loop=loop)
    async with pool.acquire() as conn:
        async with conn.cursor() as cur:
            await cur.execute("SELECT 42;")
            print(cur.description)
            (r,) = await cur.fetchone()
            assert r == 42
    pool.close()
    await pool.wait_closed()


loop = asyncio.get_event_loop()
loop.run_until_complete(test_example(loop))

Example of SQLAlchemy optional integration

Sqlalchemy support has been ported from aiopg so api should be very familiar for aiopg user.:

import asyncio
import sqlalchemy as sa

from aiomysql.sa import create_engine


metadata = sa.MetaData()

tbl = sa.Table('tbl', metadata,
               sa.Column('id', sa.Integer, primary_key=True),
               sa.Column('val', sa.String(255)))


async def go(loop):
    engine = await create_engine(user='root', db='test_pymysql',
                                 host='127.0.0.1', password='', loop=loop)
    async with engine.acquire() as conn:
        await conn.execute(tbl.insert().values(val='abc'))
        await conn.execute(tbl.insert().values(val='xyz'))

        async for row in conn.execute(tbl.select()):
            print(row.id, row.val)

    engine.close()
    await engine.wait_closed()


loop = asyncio.get_event_loop()
loop.run_until_complete(go(loop))

Requirements

aiomysql's People

Contributors

albertyw avatar alexlisovoy avatar anton-ryzhov avatar asvetlov avatar bbigras avatar clarkevans avatar clelland avatar cleure avatar dependabot[bot] avatar ikhlestov avatar jamadden avatar jcotton1123 avatar jettify avatar jvankoten avatar lecram avatar mateuszczubak avatar methane avatar nan-guo avatar nickolai-dr avatar nothing4you avatar notsosuper avatar petehunt avatar pyup-bot avatar reonaydo avatar rfk avatar terricain avatar timothyfitz avatar tvoinarovskyi avatar vlanse avatar webknjaz avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

aiomysql's Issues

Exception with pool

Hello!
I used connection pool to reuse connections. All was ok.
But after some time(about 24 hours) I get exception "index out of range" on each second request.
Also rollback also throw exception.
Here traceback:

Aug 24 15:20:18 justinstalledpanel python[930]: Traceback (most recent call last):                                                                                                                                                       
Aug 24 15:20:18 justinstalledpanel python[930]: File "/var/www/autodomain/app.py", line 86, in create_domain                                                                                                                             
Aug 24 15:20:18 justinstalledpanel python[930]: if (yield from is_record_exists(name, conn)):                                                                                                                                            
Aug 24 15:20:18 justinstalledpanel python[930]: File "/var/www/autodomain/app.py", line 48, in is_record_exists                                                                                                                          
Aug 24 15:20:18 justinstalledpanel python[930]: yield from cur.execute(query, (name,))                                                                                                                                                   
Aug 24 15:20:18 justinstalledpanel python[930]: File "/var/www/autodomain-venv/lib/python3.4/site-packages/aiomysql/cursors.py", line 238, in execute                                                                                    
Aug 24 15:20:18 justinstalledpanel python[930]: yield from self._query(query)                                                                                                                                                            
Aug 24 15:20:18 justinstalledpanel python[930]: File "/var/www/autodomain-venv/lib/python3.4/site-packages/aiomysql/cursors.py", line 459, in _query                                                                                     
Aug 24 15:20:18 justinstalledpanel python[930]: yield from conn.query(q)                                                                                                                                                                 
Aug 24 15:20:18 justinstalledpanel python[930]: File "/var/www/autodomain-venv/lib/python3.4/site-packages/aiomysql/connection.py", line 377, in query                                                                                   
Aug 24 15:20:18 justinstalledpanel python[930]: yield from self._read_query_result(unbuffered=unbuffered)                                                                                                                                
Aug 24 15:20:18 justinstalledpanel python[930]: File "/var/www/autodomain-venv/lib/python3.4/site-packages/aiomysql/connection.py", line 518, in _read_query_result                                                                      
Aug 24 15:20:18 justinstalledpanel python[930]: yield from result.read()                                                                                                                                                                 
Aug 24 15:20:18 justinstalledpanel python[930]: File "/var/www/autodomain-venv/lib/python3.4/site-packages/aiomysql/connection.py", line 736, in read
Aug 24 15:20:18 justinstalledpanel python[930]: yield from self._read_result_packet(first_packet)
Aug 24 15:20:18 justinstalledpanel python[930]: File "/var/www/autodomain-venv/lib/python3.4/site-packages/aiomysql/connection.py", line 806, in _read_result_packet
Aug 24 15:20:18 justinstalledpanel python[930]: yield from self._get_descriptions()
Aug 24 15:20:18 justinstalledpanel python[930]: File "/var/www/autodomain-venv/lib/python3.4/site-packages/aiomysql/connection.py", line 889, in _get_descriptions
Aug 24 15:20:18 justinstalledpanel python[930]: FieldDescriptorPacket)
Aug 24 15:20:18 justinstalledpanel python[930]: File "/var/www/autodomain-venv/lib/python3.4/site-packages/aiomysql/connection.py", line 499, in _read_packet
Aug 24 15:20:18 justinstalledpanel python[930]: packet = packet_type(buff, self._encoding)
Aug 24 15:20:18 justinstalledpanel python[930]: File "/var/www/autodomain-venv/lib/python3.4/site-packages/pymysql/connections.py", line 382, in __init__
Aug 24 15:20:18 justinstalledpanel python[930]: self.__parse_field_descriptor(encoding)
Aug 24 15:20:18 justinstalledpanel python[930]: File "/var/www/autodomain-venv/lib/python3.4/site-packages/pymysql/connections.py", line 390, in __parse_field_descriptor
Aug 24 15:20:18 justinstalledpanel python[930]: self.db = self.read_length_coded_string()
Aug 24 15:20:18 justinstalledpanel python[930]: File "/var/www/autodomain-venv/lib/python3.4/site-packages/pymysql/connections.py", line 331, in read_length_coded_string
Aug 24 15:20:18 justinstalledpanel python[930]: length = self.read_length_encoded_integer()
Aug 24 15:20:18 justinstalledpanel python[930]: File "/var/www/autodomain-venv/lib/python3.4/site-packages/pymysql/connections.py", line 312, in read_length_encoded_integer
Aug 24 15:20:18 justinstalledpanel python[930]: c = self.read_uint8()
Aug 24 15:20:18 justinstalledpanel python[930]: File "/var/www/autodomain-venv/lib/python3.4/site-packages/pymysql/connections.py", line 282, in read_uint8
Aug 24 15:20:18 justinstalledpanel python[930]: result = self._data[self._position]
Aug 24 15:20:18 justinstalledpanel python[930]: IndexError: index out of range
Aug 24 15:20:18 justinstalledpanel python[930]: During handling of the above exception, another exception occurred:
Aug 24 15:20:18 justinstalledpanel python[930]: Traceback (most recent call last):
Aug 24 15:20:18 justinstalledpanel python[930]: File "/var/www/autodomain-venv/lib/python3.4/site-packages/aiohttp/server.py", line 266, in start
Aug 24 15:20:18 justinstalledpanel python[930]: yield from self.handle_request(message, payload)
Aug 24 15:20:18 justinstalledpanel python[930]: File "/var/www/autodomain-venv/lib/python3.4/site-packages/aiohttp/web.py", line 90, in handle_request
Aug 24 15:20:18 justinstalledpanel python[930]: resp = yield from handler(request)
Aug 24 15:20:18 justinstalledpanel python[930]: File "/var/www/autodomain/app.py", line 41, in middleware
Aug 24 15:20:18 justinstalledpanel python[930]: return (yield from handler(request))
Aug 24 15:20:18 justinstalledpanel python[930]: File "/var/www/autodomain/app.py", line 96, in create_domain
Aug 24 15:20:18 justinstalledpanel python[930]: yield from conn.rollback()
Aug 24 15:20:18 justinstalledpanel python[930]: File "/var/www/autodomain-venv/lib/python3.4/site-packages/aiomysql/connection.py", line 327, in rollback
Aug 24 15:20:18 justinstalledpanel python[930]: yield from self._read_ok_packet()
Aug 24 15:20:18 justinstalledpanel python[930]: File "/var/www/autodomain-venv/lib/python3.4/site-packages/aiomysql/connection.py", line 298, in _read_ok_packet
Aug 24 15:20:18 justinstalledpanel python[930]: raise OperationalError(2014, "Command Out of Sync")
Aug 24 15:20:18 justinstalledpanel python[930]: pymysql.err.OperationalError: (2014, 'Command Out of Sync')

Connection of any kind Issues (1043 Bad Handshake)

Trying to connect to the database and getting a really weird error. Tried doing this without aiomysql but rather directly with PyMySQL and the connection worked fine. Would greatly appreciate some help.

Environment:
Mysql: 5.7.12
python: 3.5.1+
PyMySQL: 0.7.4
aiomysql: 0.0.7

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/aiomysql/connection.py", line 447, in _connect
    yield from self._request_authentication()
  File "/usr/local/lib/python3.5/dist-packages/aiomysql/connection.py", line 615, in _request_authentication
    auth_packet = yield from self._read_packet()
  File "/usr/local/lib/python3.5/dist-packages/aiomysql/connection.py", line 500, in _read_packet
    packet.check_error()
  File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 393, in check_error
    err.raise_mysql_exception(self._data)
  File "/usr/local/lib/python3.5/dist-packages/pymysql/err.py", line 120, in raise_mysql_exception
    _check_mysql_exception(errinfo)
  File "/usr/local/lib/python3.5/dist-packages/pymysql/err.py", line 115, in _check_mysql_exception
    raise InternalError(errno, errorvalue)
pymysql.err.InternalError: (1043, 'Bad handshake')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.5/code.py", line 91, in runcode
    exec(code, self.locals)
  File "<input>", line 22, in <module>
  File "/usr/lib/python3.5/asyncio/base_events.py", line 373, in run_until_complete
    return future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/lib/python3.5/asyncio/tasks.py", line 240, in _step
    result = coro.send(None)
  File "<input>", line 11, in test_example
  File "/usr/local/lib/python3.5/dist-packages/aiomysql/utils.py", line 57, in __await__
    resp = yield from self._coro
  File "/usr/local/lib/python3.5/dist-packages/aiomysql/pool.py", line 28, in _create_pool
    yield from pool._fill_free_pool(False)
  File "/usr/local/lib/python3.5/dist-packages/aiomysql/pool.py", line 163, in _fill_free_pool
    **self._conn_kwargs)
  File "/usr/local/lib/python3.5/dist-packages/aiomysql/utils.py", line 52, in __iter__
    resp = yield from self._coro
  File "/usr/local/lib/python3.5/dist-packages/aiomysql/connection.py", line 76, in _connect
    yield from conn._connect()
  File "/usr/local/lib/python3.5/dist-packages/aiomysql/connection.py", line 467, in _connect
    self._host) from e
pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on 'XXX.XXX.XXX.XXX'")

Why I got ignored exception when I use aiomysql in python 3.5

Basic Info:

  • python 3.5
  • run the code with SublimeREPL

The exception traceback is as follows.

Exception ignored in: <bound method Connection.__del__ of <aiomysql.connection.Connection object at 0x00000030F8080B38>>
Traceback (most recent call last):
  File "C:\software\development\python3.5\lib\site-packages\aiomysql\connection.py", line 689, in __del__
  File "C:\software\development\python3.5\lib\site-packages\aiomysql\connection.py", line 261, in close
  File "C:\software\development\python3.5\lib\asyncio\selector_events.py", line 569, in close
  File "C:\software\development\python3.5\lib\asyncio\base_events.py", line 447, in call_soon
  File "C:\software\development\python3.5\lib\asyncio\base_events.py", line 456, in _call_soon
  File "C:\software\development\python3.5\lib\asyncio\base_events.py", line 284, in _check_closed
RuntimeError: Event loop is closed

I implement a simple ORM framework with some functions to process SQL.Some related codes in orm.py(just ignore the chinese annotation) as follows. The update or findAll method in class Model works well and really give the results correctly but every time after I running the test_method,it gives the exception.

@asyncio.coroutine
def create_pool(loop, **kw):        # 引入关键字后不用显示import asyncio了
    # 该函数用于创建连接池
    global __pool  # 全局变量用于保存连接池
    __pool = yield from aiomysql.create_pool(
        host=kw.get('host', 'localhost'),  # 默认定义host名字为localhost
        port=kw.get('port', 3306),      # 默认定义mysql的默认端口是3306
        user=kw['user'],                # user是通过关键字参数传进来的
        password=kw['password'],        # 密码也是通过关键字参数传进来的
        db=kw['db'],                    # 数据库的名字
        charset=kw.get('charset', 'utf8'),  # 默认数据库字符集是utf8
        autocommit=kw.get('autocommit', True),  # 默认自动提交事务
        maxsize=kw.get('maxsize', 10),      # 连接池最多同时处理10个请求
        minsize=kw.get('minsize', 1),       # 连接池最少1个请求
        loop=loop       # 传递消息循环对象loop用于异步执行
    )
@asyncio.coroutine
def execute(sql, args, autocommit=True):
    # execute方法只返回结果数,不返回结果集,用于insert,update这些SQL语句
    log(sql)
    with (yield from __pool) as conn:
        if not autocommit:
            yield from conn.begin()
        try:
            cur = yield from conn.cursor()
            # 执行sql语句,同时替换占位符
            # pdb.set_trace()
            yield from cur.execute(sql.replace('?', '%s'), args)
            affected = cur.rowcount     # 返回受影响的行数
            yield from cur.close()       # 关闭游标
            if not autocommit:
                yield from conn.commit()
        except BaseException as e:
            if not autocommit:
                yield from conn.rollback()
            raise e  # raise不带参数,则把此处的错误往上抛;为了方便理解还是建议加e吧
        return affected


class Model(dict, metaclass=ModelMetaclass):
    # 继承dict是为了使用方便,例如对象实例user['id']即可轻松通过UserModel去数据库获取到id
    # 元类自然是为了封装我们之前写的具体的SQL处理函数,从数据库获取数据

    def __init__(self, **kw):
        # 调用dict的父类__init__方法用于创建Model,super(类名,类对象)
        super(Model, self).__init__(**kw)

    def __getattr__(self, key):
        # 调用不存在的属性时返回一些内容
        try:
            return self[key]  # 如果存在则正常返回
        except KeyError:
            raise AttributeError(
                r"'Model' object has no attribute '%s'" % key)      # r表示不转义

    def __setattr__(self, key, value):
        # 设定Model里面的key-value对象,这里value允许为None
        self[key] = value

    def getValue(self, key):
        # 获取某个具体的值,肯定存在的情况下使用该函数,否则会使用__getattr()__
        # 获取实例的key,None是默认值,getattr方法使用可以参考http://kaimingwan.com/post/python/pythonzhong-de-nei-zhi-han-shu-getattr-yu-fan-she
        return getattr(self, key, None)

    def getValueOrDefault(self, key):
        # 这个方法当value为None的时候能够返回默认值
        value = getattr(self, key, None)
        if value is None:       # 不存在这样的值则直接返回
            # self.__mapping__在metaclass中,用于保存不同实例属性在Model基类中的映射关系
            field = self.__mappings__[key]
            if field.default is not None:  # 如果实例的域存在默认值,则使用默认值
                # field.default是callable的话则直接调用
                value = field.default() if callable(field.default) else field.default
                logging.debug('using default value for %s:%s' %
                              (key, str(value)))
                setattr(self, key, value)
        return value


# --------------------------每个Model类的子类实例应该具备的执行SQL的方法比如save------
    @classmethod    # 类方法
    @asyncio.coroutine
    def findAll(cls, where=None, args=None, **kw):
        sql = [cls.__select__]  # 获取默认的select语句
        if where:   # 如果有where语句,则修改sql变量
            # 这里不用协程,是因为不需要等待数据返回
            sql.append('where')  # sql里面加上where关键字
            sql.append(where)   # 这里的where实际上是colName='xxx'这样的条件表达式
        if args is None:    # 什么参数?
            args = []

        orderBy = kw.get('orderBy', None)    # 从kw中查看是否有orderBy属性
        if orderBy:
            sql.append('order by')
            sql.append(orderBy)

        limit = kw.get('limit', None)    # mysql中可以使用limit关键字
        if limit is not None:
            sql.append('limit')
            if isinstance(limit, int):   # 如果是int类型则增加占位符
                sql.append('?')
                args.append(limit)
            elif isinstance(limit, tuple) and len(limit) == 2:   # limit可以取2个参数,表示一个范围
                sql.append('?,?')
                args.extend(limit)
            else:       # 其他情况自然是语法问题
                raise ValueError('Invalid limit value: %s' % str(limit))
            # 在原来默认SQL语句后面再添加语句,要加个空格

        rs = yield from select(' '.join(sql), args)
        return [cls(**r) for r in rs]   # 返回结果,结果是list对象,里面的元素是dict类型的

    @classmethod
    @asyncio.coroutine
    def findNumber(cls, selectField, where=None, args=None):
        # 获取行数
        # 这里的 _num_ 什么意思?别名? 我估计是mysql里面一个记录实时查询结果条数的变量
        sql = ['select %s _num_ from `%s`' % (selectField, cls.__table__)]
        # pdb.set_trace()
        if where:
            sql.append('where')
            sql.append(where)   # 这里不加空格?
        rs = yield from select(' '.join(sql), args, 1)  # size = 1
        if len(rs) == 0:  # 结果集为0的情况
                return None
        return rs[0]['_num_']   # 有结果则rs这个list中第一个词典元素_num_这个key的value值

    @classmethod
    @asyncio.coroutine
    def find_by_key(cls, pk):
        # 根据主键查找
        # pk是dict对象
        rs = yield from select('%s where `%s`=?' % (cls.__select__, cls.__primary_key__), [pk], 1)
        if len(rs) == 0:
            return None
        return cls(**rs[0])

    # 这个是实例方法
    @asyncio.coroutine
    def save(self):
        # arg是保存所有Model实例属性和主键的list,使用getValueOrDefault方法的好处是保存默认值
        # 将自己的fields保存进去
        args = list(map(self.getValueOrDefault, self.__fields__))
        args.append(self.getValueOrDefault(self.__primary_key__))
        # pdb.set_trace()
        rows = yield from execute(self.__insert__, args)  # 使用默认插入函数
        if rows != 1:
            # 插入失败就是rows!=1
            logging.warn(
                'failed to insert record: affected rows: %s' % rows)

    @asyncio.coroutine
    def update(self):
        # 这里使用getValue说明只能更新那些已经存在的值,因此不能使用getValueOrDefault方法
        args = list(map(self.getValue, self.__fields__))
        args.append(self.getValue(self.__primary_key__))
        # pdb.set_trace()
        rows = yield from execute(self.__update__, args)    # args是属性的list
        if rows != 1:
            logging.warn(
                'failed to update by primary key: affected rows: %s' % rows)

    @asyncio.coroutine
    def remove(self):
        args = [self.getValue(self.__primary_key__)]
        # pdb.set_trace()
        rows = yield from execute(self.__delete__, args)
        if rows != 1:
            logging.warn(
                'failed to remove by primary key: affected rows: %s' % rows)

The test file is as follows.

from models import User
import asyncio
import sys
import orm
import pdb
import time

# import pdb

# 测试插入


@asyncio.coroutine
def test_save(loop):
    yield from orm.create_pool(loop, user='kami', password='kami', db='pure_blog')
    u = User(name='hi', email='[email protected]',
             passwd='hi', image='about:blank')
    # pdb.set_trace()
    yield from u.save()

# 测试查询


@asyncio.coroutine
def test_findAll(loop):
    yield from orm.create_pool(loop, user='kami', password='kami', db='pure_blog')
    # 这里给的关键字参数按照xxx='xxx'的形式给出,会自动分装成dict
    rs = yield from User.findAll(email='[email protected]')      # rs是一个元素为dict的list
    # pdb.set_trace()
    for i in range(len(rs)):
        print(rs[i])

# 查询条数?


@asyncio.coroutine
def test_findNumber(loop):
    yield from orm.create_pool(loop, user='kami', password='kami', db='pure_blog')
    count = yield from User.findNumber('email')
    print(count)

# 根据主键查找,这里试ID


@asyncio.coroutine
def test_find_by_key(loop):
    yield from orm.create_pool(loop, user='kami', password='kami', db='pure_blog')
    # rs是一个dict
    # ID请自己通过数据库查询
    rs = yield from User.find_by_key('0014531826762080b29033a78624bc68c867550778f64d6000')
    print(rs)

# 根据主键删除


@asyncio.coroutine
def test_remove(loop):
    yield from orm.create_pool(loop, user='kami', password='kami', db='pure_blog')
    # 用id初始化一个实例对象
    u = User(id='0014531826762080b29033a78624bc68c867550778f64d6000')
    yield from u.remove()


# 根据主键更新
@asyncio.coroutine
def test_update(loop):
    yield from orm.create_pool(loop, user='kami', password='kami', db='pure_blog')
    # 必须按照列的顺序来初始化:'update `users` set `created_at`=?, `passwd`=?, `image`=?,
    # `admin`=?, `name`=?, `email`=? where `id`=?' 注意这里要使用time()方法,否则会直接返回个时间戳对象,而不是float值
    u = User(id='00145318300622886f186530ee74afabecedb42f9cd590a000', created_at=time.time(), passwd='test',
             image='about:blank', admin=True, name='test', email='[email protected]')  # id必须和数据库一直,其他属性可以设置成新的值,属性要全
    # pdb.set_trace()
    yield from u.update()


loop = asyncio.get_event_loop()  # 获取消息循环对象
loop.run_until_complete(test_update(loop))  # 执行协程
loop.close()

python version for the __del__ method

@asvetlov, for now we use __del__ method with python sys.version_info >= (3, 4) but aiohttp.TCPConnector uses sys.version_info >= (3, 4, 1). I am little bit confused should aiomysql have restriction >= (3, 4, 1) too?

Windows exception when using ProactorEventLoop

On Windows, using ProactorEventLoop raises an exception when trying to create a pool or a connection:

import asyncio
import aiomysql
import sys


async def mysql_example( loop ):
    async with aiomysql.create_pool(host='ensembldb.ensembl.org', port=3306, user='anonymous', password='', db='danio_rerio_vega_78_9', loop=loop) as pool:
        async with pool.get() as conn:
            async with conn.cursor() as cur:
                await cur.execute("SELECT 42;")
                value = await cur.fetchone()
                print(value)



def main():
    loop = asyncio.get_event_loop()
    if sys.platform == 'win32':
        loop = asyncio.ProactorEventLoop()
        asyncio.set_event_loop( loop )

    loop.run_until_complete( mysql_example( loop ) )

if __name__ == '__main__':
    main()

The db is a public accessible mysql server I found on the internet just for writing a quick test.

The TraceBack is:

Exception in callback _ProactorReadPipeTransport._loop_reading(<_OverlappedF...op_reading()]>)
handle: <Handle _ProactorReadPipeTransport._loop_reading(<_OverlappedF...op_reading()]>)>
Traceback (most recent call last):
File "D:\Programs\Anaconda\envs\siope\lib\asyncio\events.py", line 125, in _run
self._callback(_self._args)
File "D:\Programs\Anaconda\envs\siope\lib\asyncio\proactor_events.py", line 183, in _loop_reading
data = fut.result() # deliver data later in "finally" clause
File "D:\Programs\Anaconda\envs\siope\lib\asyncio\futures.py", line 268, in result
raise InvalidStateError('Result is not ready.')
asyncio.futures.InvalidStateError: Result is not ready.
Exception in callback _ProactorReadPipeTransport._loop_reading(<OverlappedF...S&a;?is\x00'>)
handle: <Handle _ProactorReadPipeTransport._loop_reading(<_OverlappedF...S&a;?is*\x00'>)>
Traceback (most recent call last):
File "D:\Programs\Anaconda\envs\siope\lib\asyncio\events.py", line 125, in _run
self._callback(*self._args)
File "D:\Programs\Anaconda\envs\siope\lib\asyncio\proactor_events.py", line 181, in _loop_reading
self._closing)
AssertionError

I'll investigate myself in the near future but maybe someone else has a hunch why this is happening?

aiomysql not support mysql 5.1

In my test,aiomysql not support mysql 5.1, select statement not fetch result return.What can I do?

async def aio_test():
con=await aiomysql.connect(host='instance_51',port=3306,user='user',password='passwd')
cur=await con.cursor()
await cur.execute('select 1')
res=await cur.fetchone()
print(res)
await cur.close()
con.close()

if name == 'main':
loop = asyncio.get_event_loop()
loop.run_until_complete(aio_test())

'trans.rollback()' is not working correct

I've tried to use standart code from docs:

with (yield from engine) as conn:
    trans = yield from conn.begin()
    try:
        yield from conn.execute("insert into x (a, b) values (1, 2)")
    except Exception:
        yield from trans.rollback()
    else:
        yield from trans.commit()

but in case if trans.rollback() was called for some reason then during closing the connection we will receive an exception:

  File "/home/legat/Projects/ecoisme/env/lib/python3.5/site-packages/aiomysql/sa/engine.py", line 191, in __exit__
    self._engine.release(self._conn)
  File "/home/legat/Projects/ecoisme/env/lib/python3.5/site-packages/aiomysql/sa/engine.py", line 116, in release
    raise InvalidRequestError("Cannot release a connection with "
aiomysql.sa.exc.InvalidRequestError: Cannot release a connection with not finished transaction

I think that _rollback_impl() method form SAConnection instance should be updated the same as it was at aiopg library at this commit

A little bit later I will make the same/test and provide pull request.

Add ssl support

  1. add ssl support by passing ssl.SSLContext to asyncio.open_connection.
  2. start additional mysql instance at travis ci and try to pass test suite.

Allow pulling down MetaData via reflection

Is there a way to pull down Table data with the aiomysql.sa API? When I try either of the following I get an error.

meta = MetaData()
yield from meta.reflect(bind=engine)

or

meta = MetaData()
Table('test_table', meta, autoload=True, autoload_with=engine)

File "/anaconda/envs/asap/lib/python3.5/asyncio/base_events.py", line 387, in run_until_complete
return future.result()
File "/anaconda/envs/asap/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/anaconda/envs/asap/lib/python3.5/asyncio/tasks.py", line 239, in _step
result = coro.send(None)
File "/Users/user/Documents/test/loader/loader_asyncio.py", line 39, in get_vrts
yield from get_data(engine, vrt)
File "/Users/user/Documents/test/loader/loader_asyncio.py", line 53, in get_data
yield from _get_compound_data(engine, message, vrt_number)
File "/Users/user/Documents/test/loader/loader_asyncio.py", line 90, in _get_compound_data
yield from meta.reflect(bind=engine)
File "/anaconda/envs/asap/lib/python3.5/site-packages/sqlalchemy/sql/schema.py", line 3652, in reflect
with bind.connect() as conn:
AttributeError: 'Engine' object has no attribute 'connect'
Exception ignored in: <bound method Connection.del of <aiomysql.connection.Connection object at 0x1039a8940>>
Traceback (most recent call last):
File "/anaconda/envs/asap/lib/python3.5/site-packages/aiomysql/connection.py", line 694, in del
File "/anaconda/envs/asap/lib/python3.5/site-packages/aiomysql/connection.py", line 260, in close
File "/anaconda/envs/asap/lib/python3.5/asyncio/selector_events.py", line 573, in close
File "/anaconda/envs/asap/lib/python3.5/asyncio/base_events.py", line 497, in call_soon
File "/anaconda/envs/asap/lib/python3.5/asyncio/base_events.py", line 506, in _call_soon
File "/anaconda/envs/asap/lib/python3.5/asyncio/base_events.py", line 334, in _check_closed
RuntimeError: Event loop is closed

Wrong ArgumentError message

There is a copy-paste error message mistake into aiomysql/sa/connection.py:

dp = _distill_params(multiparams, params)
if len(dp) > 1:
    raise exc.ArgumentError("aiopg doesn't support executemany")    # aiopg => aiomysql

rename Connection.wait_closed()

asyncio.AbstractServer uses the following idiom:

  • close() closes connection asynchronously, it's regular function.
  • wait_closed() is a coroutine that waits for actual closing.

Connection.wait_closed() has different behavior: it sends disconnection signal to server and after that closes connection.

I guess rename it to .ensure_closed().

Query returns result from another query

Here is my problem. I use aiomysql in aiohttp handlers. Something like this:

async def handler(request):
    data = await get_data(request.app['db_pool'])

async def get_data(db_pool):
    # prepare query q and params
    async with db_pool.acquire() as conn:
        async with conn.cursor() as cur:
            await cur.execute(q, params)
            rows = await cur.fetchall()
    # do something with rows
    # and return result

And some times rows are not the result for the given query q but the result from other query. There are no errors. It just silently returns wrong result. The only way i could determine that situation was that queries had different number of fields so there was an index out of bounds error in the log some times.

So the question is - do i run it in a right way? Or should i use any locks somewhere? Or is it a bug in the library?

Release 0.0.4

I am going to release aiomysql with following changes:

CHANGES

  • Allow to call connection.wait_closed twice.
  • Fixed sqlalchemy 1.0.0 support.
  • Fix #11: Rename Connection.wait_closed() to .ensure_closed()
  • Raise ResourceWarning on non-closed Connection
  • Rename Connection.connect to _connect

'DeprecationWarning: no_delay option is deprecated' raised even if 'no_delay' argument was not provided

After update to aiomysql 0.0.5 I've receive such warning:
'/env/lib/python3.5/site-packages/aiomysql/connection.py:154: DeprecationWarning: no_delay option is deprecated'
It was very strange because I've not provide such argument anywhere.

After diving deep a little bit I've discovered that such warning was added in this commit - link to commit

There on line 147 was added restriction

        if no_delay is not None:
            warnings.warn("no_delay option is deprecated", DeprecationWarning)
            no_delay = bool(no_delay)
        else:
            no_delay = True

But at the same time default argument at init is no_delay=False.
So with that settings it raise DeprecationWarning any time you create some connection.
I'd like to fix it, but I cannot figure out what right default 'no_delay' value should be.

Versions:
aiomysql: 0.0.5
python: 3.5.0
MariaDB: mysql Ver 15.1 Distrib 10.1.8-MariaDB
Ubuntu 14.04

'timeout' attribute not exist

Hello, I have conversation regarding issue at aio-libs google group here but there brief expanation again in any case:
aiomysql.sa.create_engine() method does not allow timeout parameter.
aiomysql.sa.Engine have no timeout attrubute.
aiomysql.Pool have no timeout attribute.
But all this situation were mentioned in docs.
I've provide pull request with docs updated(just remove this missleading 'timeouts') #42 if required.

I see that aiomysql.connect() method allow connect_timeout parameter and same we can provide to aiomysql.sa.create_engine() method without exception. But in any case we cannot get timeout attribute later from sa.Engine instance.

Also it seems that sa.create_engine() method and aiomysql.Pool class can receive all parameters that low level aiomysql.connect() method can. Maybe it reasonable to note some way regarding this in docs?

`create_pool` can not work with `async/await` feature

test script

import asyncio
import aiomysql


loop = asyncio.get_event_loop()


async def test_example():
        pool = await aiomysql.create_pool(host='127.0.0.1', port=3306,
                                               user='root', password='',
                                               db='mysql', loop=loop)
        with (await pool) as conn:
            cur = await conn.cursor()
            await cur.execute("SELECT 10")
            # print(cur.description)
            (r,) = await cur.fetchone()
            assert r == 10
        pool.close()
        await pool.wait_closed()

loop.run_until_complete(test_example())

Error Traceback

Traceback (most recent call last):
  File "aiomysql_pool.py", line 21, in <module>
    loop.run_until_complete(test_example())
  File "/usr/local/var/pyenv/versions/3.5.0/lib/python3.5/asyncio/base_events.py", line 342, in run_until_complete
    return future.result()
  File "/usr/local/var/pyenv/versions/3.5.0/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/local/var/pyenv/versions/3.5.0/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(value)
  File "aiomysql_pool.py", line 12, in test_example
    with (await pool) as conn:
  File "/Users/Eric/Projects/Python/aiomysql/aiomysql/utils.py", line 121, in __enter__
    assert self._conn
AssertionError
...

Follow PyMySQL

We made lot of improvements to PyMySQL after this fork is born.
Could you merge it?

Use loop.create_future() instead direct instantiation

Recent asyncio added new function loop.create_future() for Future object creation, so third party loops like uvloop can create own faster future implementations. We need to use this approach to benefit from speed ups.

Ping should not ignore CancelledError

This part:

        try:
            yield from self._execute_command(COMMAND.COM_PING, "")
            yield from self._read_ok_packet()
        except Exception:

In ping function will result in strange behavior on cancelling coroutine. CancelledError should be propagated outside of this try/except without any reconnect.

How to use in construct

import sqlalchemy as sa

metadata = sa.MetaData()

android_push = sa.Table(
    'android_push',
    metadata,
    sa.Column('uid', sa.Integer, primary_key=True),
    sa.Column('reg_id', sa.CHAR(256)),
)

if __name__ == '__main__':
    print(android_push.delete().where(android_push.c.uid == [1, 2, 3]))

This is the result I want.
DELETE FROM android_push WHERE android_push.uid in (1,2,3)

But the result is:
DELETE FROM android_push WHERE android_push.uid = :uid_1

AttributeError: 'NoneType' object has no attribute 'affected_rows'

requirements:

PyMySQL<=0.6.7
aiomysql

code:

async def test(conn):
    async with conn.cursor() as cur:
        q = """SELECT 1;"""
        cur.execute(q)
Traceback (most recent call last):
  File "main.py", line 129, in test
    cur.execute(q)
  File "/usr/local/lib/python3.5/site-packages/pymysql/cursors.py", line 146, in execute
    result = self._query(query)
  File "/usr/local/lib/python3.5/site-packages/pymysql/cursors.py", line 297, in _query
    self._do_get_result()
  File "/usr/local/lib/python3.5/site-packages/pymysql/cursors.py", line 347, in _do_get_result
    super(DictCursorMixin, self)._do_get_result()
  File "/usr/local/lib/python3.5/site-packages/pymysql/cursors.py", line 306, in _do_get_result
    self.rowcount = result.affected_rows
AttributeError: 'NoneType' object has no attribute 'affected_rows'

but it works with bare pymysql

AttributeError: 'Engine' object has no attribute 'contextual_connect'

I don't understand what going wrong when I try to use sqlalchemy.orm with aiomysql.sa. Code:

import asyncio

from aiomysql.sa import create_engine

from sqlalchemy import Column, Integer, String
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker

Base = declarative_base()


class User(Base):
    __tablename__ = 'users'

    id = Column(Integer, primary_key=True)
    login = Column(String(50))


@asyncio.coroutine
def init_db(loop):
    engine = yield from create_engine(
        user='root', password='password',
        db='mydb', host='localhost',
        loop=loop
    )
    return engine


@asyncio.coroutine
def get_users(engine):
    Session = sessionmaker()
    Session.configure(bind=engine)

    s = Session()
    res = s.query(User).all()
    for r in res:
        print(r.id, r.login)


if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    engine = loop.run_until_complete(init_db(loop))
    loop.run_until_complete(get_users(engine))
....
  File "/home/konst/develop/ex/my.py", line 35, in get_users
    res = s.query(User).all()
  File "/home/konst/develop/ex/env/lib/python3.4/site-packages/sqlalchemy/orm/query.py", line 2423, in all
    return list(self)
  File "/home/konst/develop/ex/env/lib/python3.4/site-packages/sqlalchemy/orm/query.py", line 2571, in __iter__
    return self._execute_and_instances(context)
  File "/home/konst/develop/ex/env/lib/python3.4/site-packages/sqlalchemy/orm/query.py", line 2584, in _execute_and_instances
    close_with_result=True)
  File "/home/konst/develop/ex/env/lib/python3.4/site-packages/sqlalchemy/orm/query.py", line 2575, in _connection_from_session
    **kw)
  File "/home/konst/develop/ex/env/lib/python3.4/site-packages/sqlalchemy/orm/session.py", line 893, in connection
    execution_options=execution_options)
  File "/home/konst/develop/ex/env/lib/python3.4/site-packages/sqlalchemy/orm/session.py", line 898, in _connection_for_bind
    engine, execution_options)
  File "/home/konst/develop/ex/env/lib/python3.4/site-packages/sqlalchemy/orm/session.py", line 334, in _connection_for_bind
    conn = bind.contextual_connect()
AttributeError: 'Engine' object has no attribute 'contextual_connect'

Versions:

  • python-3.4
  • asyncio==3.4.3
  • aiomysql==0.0.5
  • SQLAlchemy==1.0.9

Can you help?

Inconsistant results after insert using sa

Using aiomysql.sa with MySQL 5.5.49 I find that when I insert columns into a database then subsequently query the it, the result I get can sometimes include the inserted value and sometimes not.

If I essentially disable the connection pool by setting the minsize and maxsize to 1 then all works as expected. This is however not any use for production.

Attached is a simple test program and schema which should handily reproduce the problem:
test.py.txt
test.sql.txt

Impossible to install via PyPI

Hi,

I can't install aiomysql via PyPI with pip.
It's ok via Git repository.
You should add a Manifest.in file to include RST files.

Log message:

(pyvenv) lg@steroids:~/tmp$ pip install aiomysql
Downloading/unpacking aiomysql
  Downloading aiomysql-0.0.1.tar.gz (44kB): 44kB downloaded
  Running setup.py (path:/home/lg/tmp/pyvenv/build/aiomysql/setup.py) egg_info for package aiomysql
    Traceback (most recent call last):
      File "<string>", line 17, in <module>
      File "/home/lg/tmp/pyvenv/build/aiomysql/setup.py", line 54, in <module>
        long_description='\n\n'.join((read('README.rst'), read('CHANGES.rst'))),
      File "/home/lg/tmp/pyvenv/build/aiomysql/setup.py", line 20, in read
        return open(os.path.join(os.path.dirname(__file__), f)).read().strip()
    FileNotFoundError: [Errno 2] No such file or directory: '/home/lg/tmp/pyvenv/build/aiomysql/CHANGES.rst'
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 17, in <module>

  File "/home/lg/tmp/pyvenv/build/aiomysql/setup.py", line 54, in <module>

    long_description='\n\n'.join((read('README.rst'), read('CHANGES.rst'))),

  File "/home/lg/tmp/pyvenv/build/aiomysql/setup.py", line 20, in read

    return open(os.path.join(os.path.dirname(__file__), f)).read().strip()

FileNotFoundError: [Errno 2] No such file or directory: '/home/lg/tmp/pyvenv/build/aiomysql/CHANGES.rst'

Release 0.0.1

Following features are not implemented or not finished:

  1. documentation (#3)
  2. ssl support
  3. more examples
  4. ???
    So my question, should I make initial release without docs and ssl support?

LOAD LOCAL INFILE statement not work

Mysql statment "LOAD LOCAL INFILE ... " not working

changelog PyMySQLt ## 0.6.4

  • Support "LOAD LOCAL INFILE".

I try to use this LOAD LOCAL INFILE, using param client_flag: LOCAL_FILES I turn on support local files for connection. But aiomysql return Exception on LOAD LOCAL INFILE statement. Please check and add this feature.

Thank you

ERROR:aiohttp.web:Error handling request

in my case,i use aiomysql for my httpserver,everything is running OK,but it raised a error after a day, more details as fallow:

INFO:root:SQL: select `id`, `rmdposter` from `video` where rmdposter<>''
ERROR:aiohttp.web:Error handling request
Traceback (most recent call last):
  File "/root/.pyenv/versions/3.4.3/lib/python3.4/site-packages/aiomysql/connection.py", line 466, in _read_packet
    packet_header = yield from self._reader.readexactly(4)
  File "/root/.pyenv/versions/3.4.3/lib/python3.4/asyncio/streams.py", line 482, in readexactly
    raise IncompleteReadError(partial, len(partial) + n)
asyncio.streams.IncompleteReadError: 0 bytes read on a total of 4 expected bytes

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/.pyenv/versions/3.4.3/lib/python3.4/site-packages/aiohttp/server.py", line 272, in start
    yield from self.handle_request(message, payload)
  File "/root/.pyenv/versions/3.4.3/lib/python3.4/site-packages/aiohttp/web.py", line 84, in handle_request
    resp = yield from handler(request)
  File "./app.py", line 30, in logger
    return (yield from handler(request))
  File "./app.py", line 38, in response
    r = yield from handler(request)
  File "/root/.pyenv/versions/3.4.3/lib/python3.4/asyncio/coroutines.py", line 143, in coro
    res = yield from res
  File "/diskb/dxb/code/python-webapp/httpserver/coroweb.py", line 139, in __call__
    r = yield from self._func(**kw)
  File "/root/.pyenv/versions/3.4.3/lib/python3.4/asyncio/coroutines.py", line 143, in coro
    res = yield from res
  File "/diskb/dxb/code/python-webapp/httpserver/handlers.py", line 89, in api_get_media_recommend
    dataList = yield from Media.getAssetRecommend("rmdposter<>'' ",args=None, type='index')
  File "/diskb/dxb/code/python-webapp/httpserver/orm.py", line 239, in getAssetRecommend
    rs = yield from select(' '.join(sql), args)
  File "/diskb/dxb/code/python-webapp/httpserver/orm.py", line 36, in select
    yield from cur.execute(sql.replace('?', '%s'), args or ())
  File "/root/.pyenv/versions/3.4.3/lib/python3.4/site-packages/aiomysql/cursors.py", line 223, in execute
    yield from self._query(query)
  File "/root/.pyenv/versions/3.4.3/lib/python3.4/site-packages/aiomysql/cursors.py", line 444, in _query
    yield from conn.query(q)
  File "/root/.pyenv/versions/3.4.3/lib/python3.4/site-packages/aiomysql/connection.py", line 357, in query
    yield from self._read_query_result(unbuffered=unbuffered)
  File "/root/.pyenv/versions/3.4.3/lib/python3.4/site-packages/aiomysql/connection.py", line 497, in _read_query_result
    yield from result.read()
  File "/root/.pyenv/versions/3.4.3/lib/python3.4/site-packages/aiomysql/connection.py", line 688, in read
    first_packet = yield from self.connection._read_packet()
  File "/root/.pyenv/versions/3.4.3/lib/python3.4/site-packages/aiomysql/connection.py", line 477, in _read_packet
    raise OperationalError(2006, msg % (exc,)) from exc
pymysql.err.OperationalError: (2006, 'MySQL server has gone away (0 bytes read on a total of 4 expected bytes)')

it seems that because of aiomysql , after i restart my case, everythig is OK, how can i fix it?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.