Giter VIP home page Giter VIP logo

bowenislandsong / distributivecom Goto Github PK

View Code? Open in Web Editor NEW
3.0 4.0 2.0 162.47 MB

EC 601 Group Distributive Computing

Home Page: http://medusapys.site

Python 11.07% Jupyter Notebook 6.83% PHP 0.98% Swift 0.21% ASP 0.06% C# 0.05% JavaScript 22.08% HTML 34.11% CSS 24.46% C++ 0.08% Shell 0.08%
distributed-computing python3 website client-server mongodb split-files iot boinc cross-platform cluster

distributivecom's Introduction

Distributivecom

A Distributive Computing Project.

Project deployed on http://medusapys.site

Project Sprint at https://trello.com/b/i4B9QjWY/sprint

Project User Story at https://trello.com/b/RPGMAzTq/user-stories

Claim

This project is devoted to create a Distributive Computing Environment for the public via web application. (Project proposed to fulfill class requirement for BU EC 601)

Project Description

A cross platform software aiming at creating a distributive computing environment for demanding program. Client either provides computers for helping executing program to obtain credits or propose a program task to distribute amongst available units by paying credits. (Can be substituted by watching commercials.)

Motivation

We are motivated in developing this project because of our interest in distributive system among IoTs, potential financial benefits, and needs for such service from our own experience. The cooperation between IoTs is an absolute trend towards smart grid and this project, we believe, will prepare us for a general understanding. we do acknowledge the difference between distributive system and computing, but we believe distributive computing will help studying the distributive system model.

Users and Consumers

Users are people who are not using their computers for a long period of time and have constant stable Internet access. These users would be interested in providing their computers for this service for credits that can be used to for our service or potentially exchanged to their local currency. In other words, users are people who are interested in signing up their computers for work with fixed amount of credit income depending on their computing powers.

The consumers, who will be paying for the service, can execute their program on multiple computers to speed up the program execution process.

Alt text

Competitors

Current leading competitors are BOINC and MATLAB. BOINC only allows well founded super labs be the proposing side while volunteering clients are only motivated for the contribution for big topic science. General public is not allowed to propose a program for the service. MATLAB only supports MATLAB scripted programs.

Minimum Value Product (MVP)

The Minimum Value Product is the version of a new product with a collection of essential features that satisfy the early users. Our MVP is a User Interface with Low Latency response to Real-Time machine learning and other Iterative tasks.

Current Project Structure

The project is currently split into three components: Server Control, Local Web UI, and Website.

As their name suggested, Server Control runs on server and is in charge of temporary file storage, client information reserving, and file splitting.

Local Web UI is built for customers interested in leasing their computers for financial benefits. This Local Web UI, acting as a cross platform application, after activation, will automatically interact with server and perform tasks.

The Website, currently hosted at http://medusapys.site, is a client side UI for customers looking for hiring computers on Medusa.pys clusters. Clients will upload their file for computing on other computers that are part of our cluster nodes.

To communicate with peer and master computers, the project uses Swarm to host Redis clusters, commuincation between client and server with Web Sockets, user information stored in MongoDB, and file transfer with FTP powered by vsftpb.

Alt text

Testing & Result

Manual Testing Result avaliable at https://docs.google.com/spreadsheets/d/15eJNxu7yRpKBnOrxvpmXRAIFQRdWiJ12TazPb3c-FlM/edit?usp=sharing

Automated Website testing result available at http://www.webpagetest.org/result/171208_8Z_78ef41e76aa9df6ef1cd5b6e9a5aa3b1/

distributivecom's People

Contributors

bowenislandsong avatar chenxiaoxi729 avatar dishantpandya avatar hariharan-m avatar helloimlixin avatar marlon666 avatar yzy123 avatar

Stargazers

 avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

distributivecom's Issues

Code Review by Abhi Vora

  1. The start of any outsider into your Github Repo is your ReadME file and y'all have done a great job in that regard. The File is very well organized and has detailed description on how to access this repository . This also would help others who would like to contribute to this project.

  2. Coming to The UI i.e. the Webpage, http://medusapys.site/index.html , it is very nicely made. It is user friendly and very accessible . The only issue I had with it, was that when I tried uploading a file to run your application, I get sent to a DeadEnd where I receive a confirmation that my file is uploaded. It would be more useful, if that can redirect me to a page into the Website.

  3. The python files written by y'all seem to be very well maintained. There are comments when required and there is a good use of variable names which makes it easier for outsiders like me to understand your code. The only suggestion I would give is that y'all should run the codes through PEP8 or PYLINT which could make the code look more presentable although that need not be a priority, considering the amount of work to be done.

  4. I do not see a license in the Master Branch of your code. Maybe its a good idea to do so.

  5. The Achieve folder seems some what a mystery to me. There was no Read ME file as well as there were many formats of files, which got me very confused.

  6. MAJOR PROBLEM: In your database directory, I can see your Database credentials, which is not a good idea at all. I Can go at this second and make malicious changes in your DB. Please look into this ASAP.

  7. As we heard Prof. Osama say, try avoiding arbitrary numbers in the code, since it may differ from architecture to architecture.

Finally I would like to say, that y'all have done a good job and keep it up!!

code review 2 by Fengjun Li

Congratulation for u guys finish this huge work finally. Well, there are some problems I found and would like to talk.

  1. I could not sign in or sing up on the webpage while I really need to do that to enter the webpage

2.Sometimes it is difficult for me to understand the code, so may be u guys should update your python codes based on PEP8 and Pylint.

  1. There are some files u guys not use, maybe u can consider to delete them to make ur folder look clear.

4.In summary, I think u guys have already do a lot of work on project while some more work should be done for testing.

Code Review 2 by Jiahao Zhang

Code Error

When I ran Distributivecom/Archive/Aggrigation.py, it shows "SyntaxError: Missing parentheses in call to 'print' "which means you are using python 2. That doesn't matter, but it showed synax error for EXCEPT part.

Register Failure

I noticed that in Distributivecom/WebSite file, there is register.html indeed. But when I entered your website https://distributive-computing.firebaseapp.com/index.html, the "Register Now" part can not work still. I felt confused. Possibly you achieved that part but you should upload your latest webpages screenshot at least

PEP8 Test

I ran some codes on http://pep8online.com/. Some codes have some problems. For example, here is the check results of register.html:result_20171205_063649.txt

File Classification

I think you just put every file you had for your projects on the github, and they are classified into different folders. It can showed your hard work on project obviously. But there still exists three more separate documents .DS_Store, .firebaserc and firebase.json in the main page. Why? .

Folder Description

It's good to see that in Distributivecom/webUI_Networking/ file, you wrote a new readme file owing to itself. But for other files, there is no brief description which may consume much more time to figure out some specific file's meaning.

Redundant Files

Some files are unnecessary since some are too short to show any functionality and some can not even be opened. You should eliminate some useless file and just show the significant files.

Code Review by Arshitha Basavaraj

  1. The README page of your repository is informative and structured. It helped me jog my memory about your project and understand it's purpose and motivation. The folders have a clear naming convention and each of their purposes are detailed in the README which makes it easier for the contributors interested in working on your project.

  2. The website user interface (http://medusapys.site/index.html) is particularly impressive considering you haven't made use of templates. And that it has already been hosted. However, except the provider.html page, the other pages aren't interactive yet. The "register here" and login forms aren't functional yet.

  3. In Archive/Data_Splitting_Sample, the readme.txt file contains the following text which wasn't readable. I couldn't figure out its purpose or functionality. Requires more clarity.

    {\rtf1\ansi\ansicpg1252\cocoartf1561
    {\fonttbl\f0\fswiss\fcharset0 Helvetica;}
    {\colortbl;\red255\green255\blue255;}
    {\*\expandedcolortbl;;}
      \margl1440\margr1440\vieww10800\viewh8400\viewkind0
      \pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\
      tx7200\tx7920\tx8640\pardirnatural\partightenfactor0
    

    The python code included in the same folder is time-consuming to understand for a novice in
    python. So, comments would make it more readable. Also, I'd suggest adding comments and
    description to the python code instead of creating a separate readme.txt file since reading
    comments along with the code would give it context.

  4. In the local_Web_UI folder, I noticed that you guys haven't used any bootstrap templates. Bootstrap templates make it more effective and interactive. It makes it easier for the developers to focus on funtionality of the application instead of the reinventing the wheel.

  5. When the browser is resized, the UI does change dynamically but the navbar displays all of it's content which can be compressed into bootstrap sandwich menu. These small features are easy to add using bootstrap. I'd recommend using bootstrap more extensively in your project. Link: https://getbootstrap.com/docs/3.3/components/#navbar

  6. To increase visitors conversion rate of your web application, its better to use Google or Facebook user authentication instead of creating your own which would not only be time-consuming, it'd also hurt the user conversion rates.

  7. .html files are poorly indented and sparsely commented.

Code Review 2 - Anish Gupta

My team was supposed to a code review of another project but we couldn't find the repository of the project. So I will do a code review of this project.

You have come a long way since I have my last review, I looked through the code and here is my review.

  • I can see that you have moved a lot of things to docker. Which is an excellent tool when it comes to scaling and deploying a distributed application. I noticed in a lot of DockerFile's the RUN command is on a higher layer as compared to the EXPOSE and EXPORT commands. Functionally doing this is ok, but when you are debugging the docker image, you want the RUN command to be in the higher layer as you can use the lower layers as the Base image. I would also recommend you to break down the RUN into smaller and more discrete commands, that way you will have more layers that would be reused when you update the Dockerfile.

  • A few python files have not been properly formatted, they have links, random strings, etc. If these are redundant, you could delete them or add them to the GIT LFS.

  • Most of the directories in the repositories house separate projects. It would make more sense to have separate repositories for each project. It would be much easier to maintain and scale the code base.

  • I observe a lack of build files and I can guess that you are building this project manually. I would recommend you to try certain build frameworks like Make or TUP.

Code Review 2 - Arshitha Basavaraj

  1. The issues raised in the previous code review haven't been closed and relevant changes need to be made.

  2. Distributivecom/Archive/Distributive Computing/uploads/bigint.cpp Why does this file exist? As far as, I remember this belongs to EC602. How is this relevant to your project?

  3. Distributivecom/Archive/Distributive Computing/Code.txt This is basically the .txt version of Code.php . Since there are no comments in the code or a readme file for the folder. It's difficult to understand which particular functionality is being implemented.

  4. Even though this folder Distributivecom/Archive/Local Web UI/webUI/templates/ contains the required register/login .html files. On the deployed website, I'm unable to register our login. The submit button aren't taking the query and responding appropriately.

  5. Distributivecom/Archive/Local Web UI/webUI/connect.py inappropriate comments and doesn't seem to have been used anywhere.

code review1 by Fengjun Li

You team did good job on the whole project! You got all file structured and I can not wait to see your final achievement. But there are still some problems that you could solve to make your project better.

The readme is good and it really shows many detail information about your project. While I think it is not easy to understand your project especially for users who know nothing at all. They may need time to search something via google. I think it is better to briefly explain your project in one sentence. Uploading some pictures and videos to explain may also be a good method.

For the structure, all files are perfect ordered. Users can easily find or check what they want.

as for the code,i think it is perfect for you guys to add some commits. It makes user can fully understand what you had done. While I think maybe you can make effort on codes's style to make it easier to understand.

Code review 1 by Jiahao Zhang

  • Readme file
    At the beginning page, your readme file is very clearly structured.
  • Website
    The website is attractive as it looks like you guys wanna do something big, not only for a course project👍. However, there is no registration part and when I upload a file, it died. I believe you've just finished yet.
  • Files
    There are so many files in your repositories. Actually some files are not necessary added. And It will be better to rename some files so that it is easier to find. There is a good habit that in most of code files, you have comments. But they are not enough clear to review.
  • Code
    Your code is in a good manner.
  • Thanks.

code review by Hongtao Zhao

At first, I can view all your codes and blocks in a clear way, that's good. And I found you have commented every file when you upload it, which makes it easier for code reviewers to see what is that file or program for. I think our team should learn the document management ability/method from your team.

Although you have made it clear for the light pass, I think there should be more comments in the programs. I haven't learned how to set up webpage before so in your webUI_Networking/client&server_simulation/client_v2.py, I really don't know the function and usage of Fernet.generate_key() in the def encryption(message), could you make a small comment for that? Is this function created by yourself or contained in the python...

Also in the webUI_Networking/client&server_simulation/client_v2.py, I have read your code several times. I realized in your product, the user's information could be cracked easily. Could you use more advanced encryption algorithm? For the sake of not being familiar with cybersecurity, my idea is wrong maybe...

I found you guys have done a lot for your project, from user sign up, log in, data communication, firebase and google api. If you can make these things run well I hope you can add more fancy objects to your project. How to realize your idea in a more efficient way? Could it be applied to business issues? Think more about that.

Xintong Hao

Nice job! It seems like that you guys have successfully built your website and initialized some basic functions. Meanwhile, I also found some problems in your GitHub and I hope my suggestions would make your project better!

  1. Web_UI: Webpage is nice and clear. But it needs more instructions to help users know how it works. You can add some examples and give some tutorials.

  2. Server Control: It seems that you use different kind of tools to link your database. I am just a little confused with your different "client" and "server" files. Do they have different jobs? Or some of them are new version and some are test version? It is better to have a more precise name and comment in it.

  3. Networking: No idea about the test.zip (At first I think it is the sample of the test file..). And in "client&server_simulation", I cannot open the file.zip. It just extracts again and again... In your client_v2 and server_v1, if it has already tested the files, it is better to let users know what the process looks like . Because I cannot test any file through your webpage, I totally have no idea how your app works.

4.And if you have database where store the user's profile and upload information, where is it? The login page did not return anything in the console, and you don't have other "sign up" page, so I have no idea how you manage the users. Maybe you have not done that part, that will be OK~

Thank you.

Yu Cao Code Review

Structure

Firstly I would say the file construction is really good, seems everything is in great order. Readme files and comments in code are sufficient. Makes me easy to understand your project's structure and what each file is about. But one suggestion is that in your code, maybe you can add more comments in it to make it more understandable. And also that will help you guys to modify things later .

Also some name of the html files should be modified to clearify its usage.

And you should think of adding license to your project.

Code

Firstly, I think maybe you can check your python codes's style. Follow the instruction of pylint and maybe make the code more elegant.

What's more, when I'm reading the mongodb part, I find out that you are using the local host for now, this maybe the part you can improve.

Issues provided by Peixin Li

This is a new issues report after last issues provided a few weeks ago. I noticed that there are several commits after last code review which is really good , meaning that you are keep working on your project.

There are some checklist for me to do this code review.

- Code formatting

I used online PEP8 checking website to check if your code meet with PEP8 requirement. Results are shown below of login_register.py file:
screen shot 2017-12-03 at 11 21 24 am

So, as you can see, you should do some modification about your code format.

- Architecture

It looks good to me since there are basically three layers that work for each other. But I cannot find what Local_Web_UI really looks like because there isn't any screenshots for me to review. Hope you can add them later to better illustrate your project. Besides When I looked into your websites, I noticed that websites named http://medusapys.site broke down for some reasons, while another one worked good. However, I cannot use your web application without signing up and I only found sign in page which still cannot help me to register and use your websites. I checked code about regiter.html and app.py, then I realized that you didn't have any action linked when you send post request in the first file

screen shot 2017-12-03 at 11 52 54 am

and it lacks some routers about register and login in second file respectively.

screen shot 2017-12-03 at 11 54 00 am

- Coding best practices

There do exist some hard coding in your website pages, please use constants/configuration values.

This is my code review, hope it can help!

Peixin Li

Code Review:

  • Style Standards

Code structure looks good with stander indent and signature. But for the overall folder structure, it is easy to make people confused when reviewer is not familiar with the project contents and goals. I know there are three parts of code, but when I enter the folder there has no README file to explain each code's function.So I think you can add more reasonable explanation to describe your code.

  • Comments

Each code has some comments to let code easier to understand. However, Some arguments should be
commented either. Some comments are bad and unnecessary. Direct transliterations of code into English, for example, do nothing to improve understanding, because you should assume that your reader at least knows Python.

  • Avoid Magic Numbers

When using numbers, you can make it as a CONSTANT, which is more convenient for the further modification.

  • Use Good Names

Good method and variable names are long and self-descriptive. Comments can often be avoided entirely by making the code itself more readable, with better names that describe the methods and variables.

Code Review2 by Hongtao Zhao

  1. Firstly, I saw your team have uploaded quite a lot of files after code review1, and you made the classification more clear, it is a good job.

  2. I just noticed that you updated your css file just three hours ago so let me talk from that first. You have a very standard format so that code reviewer can tell each part's function. I think your webpage is delicate enough because you defined the features for footer, index page, buttons... Our group should learn it from you.

  3. And then I viewed your TestSoftware/Resource/dataset/TrainingFiles, I saw there were many samples for training.

  4. When I want to try your service on your web:https://distributive-computing.firebaseapp.com, it asked me to be your user first and then I can upload files, but I didn't find the register button! So you should fix this first. When I want to leave some messages for you on your CONTACT page, there was no feedback after I submitted. I also tried to login without registration, but there was no reminder to tell me whether I was not the user nor the password was wrong.

  5. When I tried another webpage:http://medusapys.site, this one looks pretty than the other one, but when I try to register, it reminded me invalid all the time.

  6. But in general, I believe your team have a clear goal of your project, so please do more test after you publish that, at least make sure your core function works well.

Code Review 2 : Abhi Vora

Firstly y'all have over 300 commits which is commendable. Hopefully y'all are In the final stretch as its not too long before the Poster Presentation. Best of Luck!

Now coming to the code review:

  1. Y'all havent closed the earlier Code Reviews but y'all have worked on some of the issues so thats good.

  2. This time there was an issue in opening the Web UI and I couldn't dig deeper because I am required to log in before I can do anything on the UI.

  3. On going onto the registration page, I couldn't register.

  4. When I checked the code on why I couldn't, I realized that your form action is NULL, thus y'all need to fix that soon.

  5. We were recently taught about PEP8 and Pylint, and I think y'all should update your python codes based on that ( so should we :p)

  6. There seems to be some unnecessary files in your Github such as Bigint.cpp which doesn't have a function in your project.

  7. Some of the comments are difficult to comprehend in connect.py.

Best of luck for the project!

Code Review - Anish Gupta

My team was supposed to a code review of another project but we couldn't find the repository of the project. So I will do a code review of this project.

Code Review

  1. There are few residual files from the MacOS file system, .DS_Store namely. This file is unnecessary, you might want to remove this file and add this to .gitignore.
  2. Website: It looks like you have a local copy of all the libraries (flexbox grid, font-awesome) you are using in your web pages, although republishing that code may be legal, it's not a good practice to do so. One recommendation is to use package managers like npm, bower or yarn.
  3. LICENCE: I observed that there is a GPLv3 LICENCE in the archive folder copyrighted to FSF. If you are using some other repository that has a GPLv3, you should fork the repository instead of copying the files to your repository, make changes and add it to your repository as a submodule. Also, add a license to the root of your repository.
  4. Python Modules: A python module is a directory containing related python files initialized with a __init__.py file . I observed that there are many stray python files, you could organize them into modules rather than standalone python scripts. If you are implementing your software in python, you need to adhere to the Python style guide. You can use code analyzers like pylint or pep8 module. While creating a python project, you need to specify which python version you are using and create a requirement file, generally named requirement.txt.
  5. Python Code: The python code seems fine. There are comments in almost every file describing specific code decisions. If the code is OpenSource, the python code should have docstrings which is defined in PEP-257. Also when you comment out a part of the code, like in webUI_Networking/server.py, you should add a comment describing the reason for commenting out the code.
  6. Archive: I am curious to know the utility of this directory in the code as the code here makes no sense, there are different languages being used here, PHP, Python, ASPX, Jupyter Notebooks, etc. You should add a README for this directory explaining its purpose.
  7. Security: In file MongoLogin/login_register.py. I can clearly see the password to your MongoDB server, dishantp:[email protected]:57485. You should try to use a configuration file which is not public for this purpose.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.