mining / mining Goto Github PK

View Code? Open in Web Editor NEW

1.3K 118.0 231.0 27.64 MB

Business Intelligence (BI) in Python, OLAP

Home Page: https://github.com/mining/mining

License: MIT License

Makefile 0.91% Python 98.16% Shell 0.93%

bi business-intelligence data-analytics olap olap-cube python

mining's Introduction

Open Mining

Business Intelligence (BI) Application Server written in Python

Requirements

Python 2.7 (Backend)
Lua 5.2 or LuaJIT 5.1 (OML backend)
MongoDB (Admin)
Redis (Queue and DataWarehouse)
Bower (Install frontend libs, NodeJS depends)

Install dependencies -------

$ sudo apt-get install mongodb-10gen redis-server nodejs nodejs-dev npm
$ npm install bower

If you use Mac OSX you can install all dependencies using HomeBrew.

Install Open Mining -------

Clone the repository

$ git clone [email protected]:mining/mining.git

Install python and bower dependencies using make command

$ make build

FAQ

If mongodb or redis-server problems

Install mongodb and redis-server, make sure it running

Supported databases

For example, to connect to a PostgreSQL database make sure you install a driver like psycopg2. OpenMining supports all databases that the underlying ORM SQLAlchemy supports.

See the SQLAlchemy documentation for more info about drivers and connection strings.

Run

python manage.py runserver
python manage.py celery
python manage.py scheduler

Running Demo

Make sure runserver still running when run 'build_demo' command.

python manage.py runserver
python manage.py build_demo

And now you can login with: username 'admin' and password 'admin'.

Screenshots

Dashboard OpenMining

Dashboard Charts OpenMining

Dashboard Widgets OpenMining

Late Scheduler and running Cubes OpenMining

Contribute

Join us on IRC at #openmining on freenode (web access).

Credits

Authors: Avelino and UP! Essência

Many thanks to all the contributors!

License

Licensed under the MIT license (see the (LICENSE file).

mining's People

Contributors

Stargazers

Watchers

Forkers

zennro yuripiratello yupbank srkama fth-ship uddhavpgautam pombredanne rmoorman pengjia prodigeni lnx1337 douglas-larocca liyosi jeanmask henriquejensen yenchih rogeriofalcone hlarndt ayr-ton webmaven xunyou michaeltcoelho richwu stone5495 ljdawn allanvieira python-devs-brasil frac drlatech horacioibrahim shrutikohli multitic charlespang why-not-sky aimas inzaghi2012 oelesin jgabriellima huulk daniellins songfang likaiguo chrisdamba andrzejr sayiho hwl-bi gotoc kentchun33333 jbimat jianoll ares7 netok99 ervilis yuhuofei saran-io jiekechoo solarisan2 andrew8305 textmining123 eternonq markrey iamjakob ruanaragao zjw0358 fashtimedotcom shuaiyin arfu2016 gbmoretti aaronzhangl socomsun jeremialcala venkatsethu lhxcyz euwen dpylr khaledto ferrero-zhang edwardwbli perryhau foreverjay marcososhiro leezqcst ddbs johnsonc nuaadot 3rawkz yasiral lu828 sawon1234 awesome-python zhaochl diemesleno mshtyusuf derek-zl vibhormehta mikewlange qwaszx102938 amarsubedi haoybl chesterwang

mining's Issues

Manage patterns

--server, default: 127.0.0.1
--port, default: 8888

any sample data available?

that would be nice to have a taste of this tool with sample data provided 👍

order by to element render

look paginate and filters
implement via back end (pandas)

Used MongoDB (Motor async drive) in Admin

drive: https://github.com/mongodb/motor/

Interface

Used SB-Admin: http://startbootstrap.com/templates/sb-admin/index.html

Delete handler removed hierarchy objects

If delete connection remove all related objects!

support on hadoop storage (map reduce)

Refactoring in chart javascript

breaking fields description on admin selected field

<select name="categories" ng-model="fields" class="ng-valid ng-dirty"><option value="? string:creditos ?"></option>
<option value=""></option>
<!-- ngRepeat: f in fields --><option value="c" ng-repeat="f in fields" class="ng-scope ng-binding">c</option><!-- end ngRepeat: f in fields --><option value="r" ng-repeat="f in fields" class="ng-scope ng-binding">r</option><!-- end ngRepeat: f in fields --><option value="e" ng-repeat="f in fields" class="ng-scope ng-binding">e</option><!-- end ngRepeat: f in fields --><option value="d" ng-repeat="f in fields" class="ng-scope ng-binding">d</option><!-- end ngRepeat: f in fields --><option value="i" ng-repeat="f in fields" class="ng-scope ng-binding">i</option><!-- end ngRepeat: f in fields --><option value="t" ng-repeat="f in fields" class="ng-scope ng-binding">t</option><!-- end ngRepeat: f in fields --><option value="o" ng-repeat="f in fields" class="ng-scope ng-binding">o</option><!-- end ngRepeat: f in fields --><option value="s" ng-repeat="f in fields" class="ng-scope ng-binding">s</option><!-- end ngRepeat: f in fields -->
  </select>

Render more than one element item crash chart

Not render chart on dashboard when more than one element!

crash import, used 12G of memory and kill script

Crash in long loop SQL fetchall convert to dict (Python)

CLEAN MEMCACHE/RIAK: test

CONNECT IN RELATION DATA BASE: test

LOAD DATA ON DATAWAREHOUSE: test

Traceback (most recent call last):
File "./bin/mining", line 48, in
df.columns = resoverall.keys()
File "/Users/yupbank/.virtualenvs/mining/lib/python2.7/site-packages/pandas/core/generic.py", line 1613, in setattr
object.setattr(self, name, value)
File "properties.pyx", line 59, in pandas.lib.AxisProperty.set (pandas/lib.c:30413)
File "/Users/yupbank/.virtualenvs/mining/lib/python2.7/site-packages/pandas/core/generic.py", line 394, in _set_axis
self._data.set_axis(axis, labels)
File "/Users/yupbank/.virtualenvs/mining/lib/python2.7/site-packages/pandas/core/internals.py", line 1999, in set_axis
cur_axis, value = self._set_axis(axis, value, check_axis)
File "/Users/yupbank/.virtualenvs/mining/lib/python2.7/site-packages/pandas/core/internals.py", line 1992, in _set_axis
len(value)))
ValueError: Length mismatch: Expected axis has 0 elements, new values have 5 elements

http://pandas.pydata.org/pandas-docs/stable/comparison_with_sql.html

[E 140210 13:26:56 web:1305] Uncaught exception GET / (127.0.0.1)
HTTPRequest(protocol='http', host='localhost:8888', method='GET', uri='/', version='HTTP/1.1', remote_ip='127.0.0.1', headers={'Connection': 'keep-alive', 'Accept-Language': 'en-US,en;q=0.8,pt-BR;q=0.6,pt;q=0.4', 'Accept-Encoding': 'gzip,deflate,sdch', 'Host': 'localhost:8888', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8', 'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36'})
Traceback (most recent call last):
  File "/deploy/pyenvs/mining/local/lib/python2.7/site-packages/tornado/web.py", line 1218, in _when_complete
    callback()
  File "/deploy/pyenvs/mining/local/lib/python2.7/site-packages/tornado/web.py", line 1239, in _execute_method
    self._when_complete(method(*self.path_args, **self.path_kwargs),
  File "/home/yuri/Documentos/Projetos/mining/mining/views.py", line 25, in get
    self.render('index.html', dashboard=dashboard)
  File "/deploy/pyenvs/mining/local/lib/python2.7/site-packages/tornado/web.py", line 615, in render
    html = self.render_string(template_name, **kwargs)
  File "/deploy/pyenvs/mining/local/lib/python2.7/site-packages/tornado/web.py", line 722, in render_string
    return t.generate(**namespace)
  File "/deploy/pyenvs/mining/local/lib/python2.7/site-packages/tornado/template.py", line 278, in generate
    return execute()
  File "index_html.generated.py", line 5, in _tt_execute
    for d in dashboard:  # index.html:5
TypeError: 'NoneType' object is not iterable

If not exist return text: "Does not exist dashboard!"