The minion from alibaba

minion's Introduction

Introduction

Minion, which achieves an approach to maximize utilization of broadband traffic like BT, implemented by python and based on HTTP protocol

Minion can be easily used to dispatch data, similar to curl/wget. It also provides python API libs.

Installtion

minion cli need python 2.7, requests

minion tracker need python 2.7, django mysql

git clone git://github.com/alibaba/minion
cd minion
pip install -r requirements.txt
python setup.py install

Usage

deploy your tracker server

tracker need django mysql

python tracker/manage.py syncdb
python tracker/manage.py runserver # or use wsgi/nginx

try your minion cli

minion get http://foo.bar/testfile \
     --tracker some.tracker.server \
     --dest-path=/tmp/tops1 \
     --upload-rate 10M \
     --download-rate 100M \
     --callback delete \
     --upload-time 5 \
     --fallback

--tracker specify tracker
--dest-path specify the path data will be wrote to, default to current dir
--upload-time specify time for upload after download complete, default to 60s
--download-rate --upload-rate specify rate for upload/download can used with unit, like 10M
--fallback option, if no resource exists in tracker download directly from origin url
--callback
--verbose for more log

Framework

Minion work as picture below

PEER: The host ready to download SOURCE TRACKER: The host manage PEERs and SOURCEs SOURCE: URL of some data

STEP

PEER0 access TRACKER to get peers that was uploading SOURCE, TRACKER return null
PEER0 get SOURCE from origin url directly
PEER0 upload SOURCE, telling that to TRACKER
PEER1 do like step 1，return a list contain PEER0
PEER1 get SOURCE from PEER0
PEER1 do like step 3

License

Minion as a whole is released under the GNU General Public License version 2.

中文

minion's People

Contributors

Stargazers

Watchers

minion's Issues

why do not generate the file named "minion"

In the shell of ubuntu, after i run "python setup.py install", then i exect the cmd of "minion", but can not find it, could you tell me the reason.
Thank You.

项目简介，一句话两处 typo

Minion, which achives an approach to broadbrand traffic like BT, implemented by python and based on HTTP protocol

认真点，好不好？

执行 python tracker/manage.py syncdb 遇到问题

$python tracker/manage.py syncdb
Traceback (most recent call last):
  File "tracker/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/lib/python2.7/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
    utility.execute()
  File "/usr/lib/python2.7/site-packages/django/core/management/__init__.py", line 377, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/lib/python2.7/site-packages/django/core/management/base.py", line 288, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/usr/lib/python2.7/site-packages/django/core/management/base.py", line 338, in execute
    output = self.handle(*args, **options)
  File "/usr/lib/python2.7/site-packages/django/core/management/base.py", line 533, in handle
    return self.handle_noargs(**options)
  File "/usr/lib/python2.7/site-packages/django/core/management/commands/syncdb.py", line 27, in handle_noargs
    call_command("migrate", **options)
  File "/usr/lib/python2.7/site-packages/django/core/management/__init__.py", line 115, in call_command
    return klass.execute(*args, **defaults)
  File "/usr/lib/python2.7/site-packages/django/core/management/base.py", line 338, in execute
    output = self.handle(*args, **options)
  File "/usr/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 63, in handle
    executor = MigrationExecutor(connection, self.migration_progress_callback)
  File "/usr/lib/python2.7/site-packages/django/db/migrations/executor.py", line 17, in __init__
    self.loader = MigrationLoader(self.connection)
  File "/usr/lib/python2.7/site-packages/django/db/migrations/loader.py", line 48, in __init__
    self.build_graph()
  File "/usr/lib/python2.7/site-packages/django/db/migrations/loader.py", line 183, in build_graph
    self.applied_migrations = recorder.applied_migrations()
  File "/usr/lib/python2.7/site-packages/django/db/migrations/recorder.py", line 59, in applied_migrations
    self.ensure_schema()
  File "/usr/lib/python2.7/site-packages/django/db/migrations/recorder.py", line 49, in ensure_schema
    if self.Migration._meta.db_table in self.connection.introspection.get_table_list(self.connection.cursor()):
  File "/usr/lib/python2.7/site-packages/django/db/backends/__init__.py", line 165, in cursor
    cursor = self.make_debug_cursor(self._cursor())
  File "/usr/lib/python2.7/site-packages/django/db/backends/__init__.py", line 138, in _cursor
    self.ensure_connection()
  File "/usr/lib/python2.7/site-packages/django/db/backends/__init__.py", line 133, in ensure_connection
    self.connect()
  File "/usr/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/usr/lib/python2.7/site-packages/django/db/backends/__init__.py", line 133, in ensure_connection
    self.connect()
  File "/usr/lib/python2.7/site-packages/django/db/backends/__init__.py", line 122, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/usr/lib/python2.7/site-packages/django/db/backends/mysql/base.py", line 472, in get_new_connection
    conn = Database.connect(**conn_params)
  File "/usr/lib64/python2.7/site-packages/MySQLdb/__init__.py", line 81, in Connect
    return Connection(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 193, in __init__
    super(Connection, self).__init__(*args, **kwargs2)
django.db.utils.OperationalError: (1049, "Unknown database 'tracker'")