Giter VIP home page Giter VIP logo

spore's Introduction

spore

一个简易的传输工具

背景

本人在工作中遇到大数据(20G左右)批量部署的问题,刚开始通过scp或者rsync限速80M串行分发,大约每台机器的耗时为5分钟, 极大的增加了大批量部署的时间和难度,各种难产。

思路

将传输完成的目标机器转换成数据源为其他机器提供数据源服务。(相当于《行尸走肉》中的僵尸一样,每一个被感染的个体都是传染源)

语言选型

对于数据传输来说,个人认为语言不是瓶颈而是传输的方法和(串行、并行还是其他),之所以选择go是因为go可以编译后扔到机器上直接执行(python需要安装各种依赖包增加了部署的难度)

依赖

由于公司机器都已安装ftp服务且已配置限速下载,所以工具本身不提供下载服务,下载限速也是写死在client端的。

架构

C/S架构,即大家熟知的master和client结构。 Master:负责分发控制客户端的下载以及接受客户端的回传信息。 Client:等待接受master下载任务并执行。

效果

第1次传输1台,第2次传输2台,第3次传输4台……第n次传输2(n-1) 台,也就是说传输1+2+4=7台数据的时间相当于串行传输3台机器的时间,并且传输的数量越多,相对串行传输时间越少。 服务执行完自动退出,主要目的:防止误操作,切断服务器后门。

spore's People

Contributors

auxten avatar

Watchers

James Cloos avatar 李巨龙 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.