Use redis to manage queues, locks, etc. across processes and multiple hosts.

Problem

When two or more clients try to connect to the same IRC network simultaneously, for the first client hitting the server, we create a bot instance to connect to the IRC server, and for the subsequent clients, we try to reuse the existing bot and start sending messages to the IRC server. But there's a time, when the IRC bot is instantiated, but not yet connected to the IRC, and the other clients try to send message to the IRC server, and it fails.

[2015-12-01 17:50:02][   bouncer   ERROR] get_bot_handle error: 'IrcbBot' object has no attribute 'protocol'
Traceback (most recent call last):
  File "ircb/bouncer.py", line 156, in get_bot_handle
    bot.raw('NAMES %s' % channel.name)
  File "/home/rtnpro/waartaa/ircb/ircb/irc/__init__.py", line 160, in raw
    self.send_line(message)
  File "/home/rtnpro/.virtualenvs/ircb/lib/python3.4/site-packages/irc3/__init__.py", line 162, in send_line
    self.send(data.replace('\n', ' ').replace('\r', ' '))
  File "/home/rtnpro/.virtualenvs/ircb/lib/python3.4/site-packages/irc3/__init__.py", line 166, in send
    self._send(data)
  File "/home/rtnpro/.virtualenvs/ircb/lib/python3.4/site-packages/irc3/__init__.py", line 169, in _send
    self.protocol.write(data)
AttributeError: 'IrcbBot' object has no attribute 'protocol'

Solution

We need to use some kind of mutex to block other client requests until the IRC bot is connected to the IRC server.

• convert data to serializable dict when publishing it from stores
• allow loading model objects from serialized data in storeclient

Allow running the bouncer server to accept SSL connections, to secure traffic from the IRC clients to bouncer server. This should ease up setting on SSL support for bouncer server with depending on external load balancers, however this should not stop people from implementing SSL support using 3rd party applications.

In 0.2 release, with server side flux implementation, we went multiprocess, i.e., separate store, bouncer, cli and web processes. With the basic server side flux framework in place, it's time to go multiple bouncer instances this time. The goal is to:

• distribute IRC bot instances across multiple bouncer instances
• to be fault tolerant, i.e., if a bouncer instance dies, the system must be able to re create the deceased bots in the failed bouncer in the current running bouncers
• implement load balancer for the bouncers

TODOS:

• Implement dynamic bouncer registration and discovery.
• Implement a strategy to distribute IRC bots across multiple bouncers (Round Robin would work for now)
• Implement a registry for IRC bots running in bouncers
• When a bouncer dies, distribute bots running on it across working bouncers and connect to IRC
• Load balance across bouncers using 005 RPL_BOUNCE IRC message as mentioned in https://tools.ietf.org/html/rfc2812

Let's create separate issues for each of the above TODOs and start hacking

irc3 IRC bot supports SSL connection to remote IRC servers. All we need to do is make the fields:

• ssl
• ssl_cert_check

fields to Network table, so that we can use the info during bot creation.

Issue

We need redis to setup pubsub between stores and storeclient, however, we do not throw any error if we are not able to connect to redis. As a result, the services keep running without able to access data from or publish data to stores.

Expected behavior

Services should error out with meaningful log message when it's not able to connect to redis.

Related code

• https://github.com/waartaa/ircb/blob/develop/ircb/lib/dispatcher/__init__.py#L43
• https://github.com/waartaa/ircb/blob/develop/ircb/lib/dispatcher/__init__.py#L79

Let Network chose whether it wants to IRC using SSL or not, similar to typical IRC clients, like Xchat.

I have fedora22 with python 3.4 so pip3.5 is not available on my pc.
when I do try with pip3.4 : pip3.4 install -r requirements.txt
I get:
gcc -pthread -Wno-unused-result -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -DUSE__THREAD -I/usr/include/ffi -I/usr/include/libffi -I/usr/include/python3.4m -c c/_cffi_backend.c -o build/temp.linux-x86_64-3.4/c/_cffi_backend.o

c/_cffi_backend.c:15:17: fatal error: ffi.h: No such file or directory

compilation terminated.

error: command 'gcc' failed with exit status 1

----------------------------------------
Command "/usr/bin/python3 -c "import setuptools, tokenize;__file__='/tmp/pip-build-qslojink/cffi/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-vpyemxb6-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-qslojink/cffi

this type of error, how do I solve this.

Currently call to self.away() is breaking in

TypeError: away() takes from 1 to 2 positional arguments but 3 were given

Support filter, order_by, limit, sort queries when fetching Channel entries from stores.

root@deepak-K56CA:~/waartaa/app# meteor
'/home/deepak/.meteor' exists, but '/home/deepak/.meteor/meteor' is not executable.

Remove it and try again.

Doing periodical release for a project drives the project faster. Let's start releasing a new version of ircb, every month. The policy would be to release a new version with whatever stable code we have in develop branch.

@sayanchowdhury @kushaldas
What do you think?

Python 3.5 incorporated updated the the asyncio.

PEP 492 - Coroutines with async and await syntax

Lets consider upgrading to the latest version of python asyncio and syntax going forward and possible back-porting these updates into the current code.

We need to add docs on how to expose IRCB bouncer over SSL using stunnel.

'networks create' from ircb command line tool does not work. Which means:
(1)the network creation is not completed and (2)the bouncer crashes and returns the following traceback :

 [   bouncer   ERROR] get_bot_handle error: 'NoneType' object has no attribute 'id'
Traceback (most recent call last):
  File "ircb/bouncer.py", line 108, in get_bot_handle
    key = network.id
AttributeError: 'NoneType' object has no attribute 'id'

Steps to replicate

After setting up ircb :

Run:
ircb run stores

Another terminal try to create user by the given command.

The following traceback occurs

Traceback (most recent call last):
File "/home/fhackdroid/.virtualenvs/python3/lib/python3.5/site-packages/aiozmq-0.7.1-py3.5.egg/aiozmq/rpc/base.py", line 236, in try_log
fut.result()
File "/usr/lib64/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/home/fhackdroid/.virtualenvs/python3/lib/python3.5/site-packages/aiozmq-0.7.1-py3.5.egg/aiozmq/rpc/pubsub.py", line 223, in process_call_result
if fut.result() is not None:
File "/usr/lib64/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/lib64/python3.5/asyncio/tasks.py", line 239, in _step
result = coro.send(None)
File "/home/fhackdroid/waarta/ircb/ircb/lib/dispatcher/init.py", line 49, in register_sub
self._lock.release()
File "/usr/lib64/python3.5/asyncio/locks.py", line 201, in release
raise RuntimeError('Lock is not acquired.')
RuntimeError: Lock is not acquired.
[2016-09-30 15:33:19][ stores DEBUG] UserStore CREATE: store-user-create {'password': '', 'username': 'farhaan', 'email': '[email protected]'} 1d0db124-86f5-11e6-b9e4-$
0fd61ea064c
[2016-09-30 15:33:19][ stores DEBUG] UserStore CREATED: <ircb.models.user.User object at 0x7f8a3edc07b8>
[2016-09-30 15:33:19][dispatcher DEBUG] SEND: store-user-create {'password': '', 'username': 'farhaan', 'email': '[email protected]'} 1d0db124-86f5-11e6-b9e4-90fd61ea064c
[2016-09-30 15:33:19][dispatcher DEBUG] PUBLISH from stores: ('store-user-created', {'last_name': '', 'created': 1475209999.469223, 'id': 1, 'confirmed_at': None, 'first_name': $
', 'email': '[email protected]', 'username': 'farhaan', 'reset_password_token': '', 'last_updated': 1475209999.469235, 'is_active': False}, '1d0db124-86f5-11e6-b9e4-90fd61e$
064c')
An exception RuntimeError('Lock is not acquired.',) from method 'register_sub' call occurred.

Currently, we do not autojoin previously joined channels during connecting to IRC server, but we let the IRC client initiating the connection dictate which channels are to be joined.

We need to join the previously connected channels, as well as, new channels requested by the IRC client to join.

TODO:

• update signature: https://github.com/waartaa/ircb/blob/develop/ircb/stores/network.py#L34
• update call to Network: https://github.com/waartaa/ircb/blob/develop/ircb/stores/network.py#L39
• Add CLI option for ssl: default=False, ssl_verify: default="CERT_NONE" at https://github.com/waartaa/ircb/blob/develop/ircb/cli/network.py#L22 and update call at: https://github.com/waartaa/ircb/blob/develop/ircb/cli/network.py#L27

Have a list of predefined networks so that the user can directly connect

I have been looking into the files and found that a lot functions don't have docstring written which makes it difficult to figure out what that function does.

https://github.com/waartaa/ircb/blob/develop/ircb/lib/dispatcher/__init__.py

Proposed features

• Auto join previously connected IRC networks on bouncer start. Create a coroutine for it
• Disconnect client requests for a IRC network as long as it's bot has not started

ssl_verify in IrcbBot config expects a string and not a ChoiceType field.

So, IRC raw commands usually follow the pattern: [target] :[some message]. Currently we are passing :[some message] as is without removing the prefix :.

Bug location: https://github.com/waartaa/ircb/blob/develop/ircb/irc/__init__.py#L55

Problem

Currently, we store in-memory the IRC joining messages for the first client for a bot, to spit out to the subsequent clients connecting to the same bot. We need a better way to send joining messages to clients so that IRC clients understand that connection to network is established.

Problem

Migrating to use irc3.IrcBot broke our support for supporting multiple IRC clients for the same IRC network connection, or, a single instance of a bot.

Probable solution

May be we should reject the initial NICK message sent by the IRC client when reusing an existing bot. Look at https://github.com/waartaa/ircb/blob/develop/ircb/bouncer.py#L141 part.

We need to have a client side ircb cli so that a user can:

• set ircb endpoint
• register, login, logout
• get/create/update/delete network
• connect/disconnect network

We can use requests module on client to interact with the basic ircb api (work in progress at #51)

Currently, we subscribe to all messages over message bus, which works pretty well for WRITE operations to keep the systems updated about the latest changes. But, this does not work well for READs as these are many in number and other's don't need to know about it, but the client requesting the data. Also, we need to find out a way to subscribe (unsubscribe) to(from) TOPICS without latency in setting up the duplex communication channel.

• Write Dockerfiles
• Write docker compose file

Problem statement

Currently we add data: user, network data in ircb using ircb cli. This can prove tedious sometimes. It'd be great if we could load this data from a YAML file. This'd allow ircb service providers to maintain this file in a git repository, and adopt a simple git workflow to grant users access.

Implementation details

Possible file content could have the following schema:

---
username1:
  email: [email protected]
  networks:
    network1:
      nick: some-nick
      host: ...
      port: ...
      realname: ...
      username: ...
      password: ...
      usermode: ...
    network2:
      ...
username2:
  ...

I just came across this cool Python library: autobahn 1 for building realtime webapps with support for pub/sub, rpc, etc.

Consider adding in type annotations to the code going forward, adding it into the existing code base, and adding the mypy check to the tox/travisci integration

Reading through the code I see that the runserver command is running using the asyncio library but still using sqlalchemy to the database requests. This seems like an incompatible configuration based on the discussion on the sqlalchemy repo (link below).

asyncio-and-sqlalchemy

I also found some interesting discussion happening here

How to make SQLAlchemy in Tornado to be async?

I'm still reading through the system's configuration so please let me know if I've missed something.

Implement basic publisher for chat logs to allow clients to subscribe to realtime chat logs.

The Readme for the project says:

IRC client should have the following settings enabled:

    Use SSL for all server on this network
    Accept invalid SSL certificate

I am wondering what is the exact limitation in the project that needs 2nd setting to be enabled? Is it because ircb uses self generated & signed messages or something else?

I got the following error when I tried to open 0.0.0.0:9000

[2015-12-26 02:07:40][   bouncer   ERROR] get_bot_handle error: 'NoneType' object has no attribute 'id'
Traceback (most recent call last):
  File "/home/yasoob/Desktop/ircb/ircb/bouncer.py", line 110, in get_bot_handle
    key = network.id
AttributeError: 'NoneType' object has no attribute 'id'

I followed the following steps:

$ python3 setup.py develop
$ ircb users create yasoob [email protected] <password>
$ ircb networks create yasoob Freenode irc.freenode.net 6665 yasoob
$ ircb runserver

bcrypt/bcrypt_python.c:18:20: fatal error: Python.h: No such file or directory
#include "Python.h"

import MySQLdb
db=MySQLdb.connect(host="localhost",user="deepak",passwd="",db="hanuman")
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python2.7/dist-packages/MySQLdb/init.py", line 81, in Connect
return Connection(_args, *_kwargs)
File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 187, in init
super(Connection, self).init(_args, *_kwargs2)
_mysql_exceptions.OperationalError: (1045, "Access denied for user 'deepak'@'localhost' (using password: NO)")

This is contain the list of guidelines to keep in mind before sending out a PR.

https://help.github.com/articles/setting-guidelines-for-repository-contributors/

Currently, marking user as not AWAY does not work.

This is because we are not able to:

verb, message = line.split(" ", 1)

when line == 'AWAY at

Implement identd server for ircb.

Benefits

• Interprocess communication, e.g., allow CLI interact with running bouncer