prasadtalasila / transportscheduler Goto Github PK

Implementation using Maru or any other RESTful API library

Please update Behavior and Polymorphism page.

integration tests for now seem to be done using ExUnit and curl. We can have them run using BATS. For example tests, see functional tests of JavaAutolab.

Tests are:

1. Query from one station to another station.
2. Query from network constructor to a station.
3. Answer from a station to network constructor.
4. Query from network constructor to a station and answer coming back from another station.

a linter and static code analysis tool for Elixir language. A web-based service that we can integrate with the github commits would be good as well.

When NC receives multiple potential itinerary results from destination, these must be collected and best option based on the given constraints must be chosen. In the future, batch queries are to be submitted to the application. The NC will then receive results from multiple destinations, and will need a separate collection of itineraries for each such query. Within each collection, NC will have selection and timeout mechanism separately.

Input parser to load data from Stations.txt, Schedule.txt and Local_Variables.txt

Periodically station process crashes. We are using supervisor to recover from a process crash. We don't know the reason for the crash itself. Use Elixir debugger to investigate the cause of crash.

API call to station controller fetches station schedule.

In order to complete this user story, an appropriate RESTful library must be integrated with station controller. A good way to check this would be to issue the API call from curl of the form http://localhost/schedule/stn_code and check all the output. We must receive all the schedule entries of a station in tuple form.

api_supervisor.ex seems redundant. Remove the same and spawn api.ex from supervisor.ex.

Coveralls tool has support for elixir code coverage reports. Please integrate into the project by following the instructions.

excoveralls seems like a useful library for this task.

Build status of both master and ts-v2.0-alpha branches on the local server.

learn and update the TDD wiki page.

New wiki page tutorials need to be created for the mentioned topics.

Joseph Kain wrote elixir-life project which has the exprof as a mix task. Please see Profile.ex for more details. He uses eflame to draw the performance graphs which will help in looking at the bottlenecks. For a way to include eflame as dependency, see his mix.exs file. An explanation of his code is available profile, profile-2.

As shown in the comments of profile-elixir-2 article, see RickHull's mix task for profiling.

Joseph Kain's bmark tool seems useful as well. An explanation of bmark tool is available.

benchee and benchfella helps benchmark unit tests. This is especially useful for integration kind of unit tests. We were hoping to get performance results at the level of functions of TransportScheduler. Both these tools help us get such performance numbers. Of the two tools, benchee seems more promising.

Ref: http://learningelixir.joekain.com/optimizing-elixir-posts/

a wiki page detailing the installation setup required to make contributions to the project. The page would contain proper installation instructions for Erlang, OTP-19.0, Elixir and then the commands for running the ts app.

The new station module needs to be created with workex + gen_server + fsm. Obviously, the newly created station module must pass all the unit tests written so far.

• Stores given state
• Retrieve the state
• Update the given state
• Receive a itinerary search query
• Send completed search query to neighbours
• Computes the itinerary correctly
• Does not forward itineraries with potential self-loops
• Does not forward stale queries
• Incorrectly received queries are discarded
• Terminated queries are handed over to query collector
• Consumes rapid input stream of mixed queries
  • number of queries - 1000; 10,000; 1 million

When a process crashes, updates to its state are lost. Any updates coming through update controller (UC) must be persisted via ETS / Mnesia so that a recovered station process can reload its state.

References:
amnesia - Mnesia wrapper for Elixir.
Mnesia tutorial

Both sequential and concurrent queries increase the memory consumption of the application. Ideally after query is complete, the application must relinquish the excess memory. But, that does not happen.

Each sequential query increases the memory requirement till the entire available memory is allocated to ts application.

Presently an api call is needed to initialize the application. For auto initialization the code for initialization ( lines 24-33 in api.ex ) has to be moved to the init block of API.
But on doing so an api call is still required for initialization.

New wiki page needs to be created for architecture so that all of us are on the same page.

In case a process abruptly terminates, NC must handle the issue. Only queries that pass through a non-existent process must be affected.
Process needs to be restarted and registry updated with new pid value. Any updates that had been made to local variables of the old process will be lost in such a situation.

If an invalid query is submitted, say invalid station name that does not match any station code, NC will throw an exception. API module must handle these and reply with suitable message to user.

Concurrent requests need to be supported at UQC; each request gets its own timeout value of 0.5sec.

Choose proper data structure for storing query results. We want to send N (=10) itinerary results to the user.

A module representing one station. The detailed description of a station is available on wiki page.

Look for a scalable RESTful API library for elixir and use it to in the Station Controller. Appropriate reference links to be updated on References page.

visualixir project offers path tracing possibility in the process network. We can take advantage of that for visualizing some scenarios for the research paper.

The install scripts assume /home/ubuntu as the fixed parent directory of the project. Use git commands to deduce the project directory and make the scripts generic.

The updated scripts must run satisfactorily in direct host install scenario, in a virtual machine and in vagrant environment.

We can use Escript to continuously generate the bundled binary scripts to be run on BeamVM.
Once a Escript is successfully run manually, we can use Jenkins CI to build every commit.

The NetworkConstructor module keeps track of the active queries (the queries whose results are awaited by the user and the itineraries are currently being searched in the network) in the form a Map data structure, and there is an abstracted client function check_active/2 that returns the status of a given query (true for active, false for inactive). This needs to be implemented in the ETS table at the UQC.

initialization of station must happen in init function. At the moment, the work is being done in update() function.

At the moment, code profiling reveals that 70% of the processing time is spent in message passing. This is leading to a scenario of sequential query execution. If we can solve the problem of list storage in station processes and message passing overhead, truly concurrent query processing can take place.

Checkout the following resources.

1. Edgelixir - elixir graph processing, similar to ours
2. ETS vs Message Passing vs redis vs Mnesia (see references for links
3. Learn from Pregel/Giraph - think like a vertex model. See review
4. See Y-Combinator Question on this topic.
5. Message passing benchmarks - thesis
6. Erlang memory model
7. message passing between processes - sending big lists, on choice of binary messages, message passing steps, ETS vs message passing, Erlang process documentation

The itinerary search queries are all being collected in NC at the moment, making NC a bottleneck. We can change the scenario to the following.

1. A query goes to NC
2. NC spawns a query response collector process
3. Adds the process id of the query response collector to the query structure.
4. Gives the query to source station
5. Destination passes the response to query response collector process
6. Query response collector gives the set of responses to UQC

It would be a good practice to have a ready release / pre-release for the software. That way, any foreseen issues on the dev branch would not effect the people who would want to use the software.

We can have a working commit that is decent, create a release out of it.

Erlang provides pattern matching on mailbox messages 1, 2. Even Elixir has the message receive pattern matching capability.

Also see an explanation on this stackoverflow answer.

We can probably use them to flush out the stale queries faster and to give priority to active queries at a station process.

A scalable way must be found for passing functions betweens elixir processes. Two possible approaches we are exploring are:

1. Have a module of all possible functions and use Map to store references to functions in stations.
2. Create a function in one process, quote the function and pass the function to another process. The receiving process unquotes the function and uses it.

The lib/maru/test.ex and test/ts_test.ex files are either not needed or in wrong location. They need to be either put in correct location or removed.

Input parser tests must pass and input parser module must integrate well with the station module. There would be unit tests for integration as well.

we want a profiler that gives the execution time of each of our processes and the function calls. That way we can talk about optimization of the application in terms of domain logic.

The current tool exprof is giving the execution time results of low-level functions. We need to obtain profiling information on elixir processes.

select an appropriate profiling tool for this job.

When message passing is used for a given query, current implementation checks for all neighbours of a given station node. Hence, if an outgoing vehicle link exists back from a destination to a source node with valid time of departure, an itinerary may be generated that contains a loop. The same station is revisited and the resulting itinerary will never be the optimal one. Such a result must be excluded.

Suggested ones are:
Gen State Machine v2.0.0, Erlang OTP 20.0, Elixir v1.5

The problem is that the station.ex uses both synchronous and asynchronous functions of gen state machine. We need to convert all the synchronous functions to asynchronous before we attempt an upward revision of all the packages.

This issue would also solve #9.

A short summary of Erlang concurrency model as used by Elixir. Also see the Actor concurrency model chapter in Seven Concurrency Models in Seven Weeks book.

We need to have all code commits regression tested and built. The necessary GitHub integrations must be setup.