podigg / podigg-lc Goto Github PK

View Code? Open in Web Editor NEW

3.0 2.0 2.0 16 KB

Linked Connections RDF output generator for PoDiGG

License: MIT License

JavaScript 100.00%

rdf generator population-distribution

podigg-lc's Introduction

PoDiGG Linked Connections

POpulation DIstribution-based Gtfs Generator

A realistic public transport dataset generator, which is serialized as RDF/Turtle. This is an extension of the PoDiGG generator that automatically generates Linked Connections RDF in Turtle from the GTFS data that is generated.

Install

npm install podigg-lc

Usage

This tool will generate a lc.ttl file in the output_data folder, which contains Linked Connections. Additionally, the folder will also contain the raw GTFS files and a visualization of the data.

All generator parameters can be configured using a config file or environment variables, depending on how the generator is invoked. More information about this can be found at https://github.com/PoDiGG/podigg#parameters

For big data generation, the NODE_MEM environment variable can be overridden to increase the maximum memory usage. This can be done by adding the -e NODE_MEM=<memory in MB> option when running the Docker container.

Command line

The easiest way to run the generator is using the command line tool:

podigg-lc [output folder [path to a JSON config file]]

This config file contains parameters for the generator, as explained in the main generator.

Alternatively, the generator can also be configured using environment variables, as explained below. In that case, the generator must be called as follows:

podigg-lc-env [output folder]

Docker

This generator can be run using a Docker container as follows:

Downloading and running the container from the Docker hub:

docker pull podigg/podigg-lc
docker run -it --rm \
-v $(pwd)/output_data:/tmp/output_data \
-e GTFS_GEN_SEED=123 \
podigg/podigg-lc

Building and running the container from this repo:

git clone [email protected]:PoDiGG/podigg-lc.git
cd podigg-lc
docker build -t podigg-lc .
docker run -it --rm \
-v $(pwd)/output_data:/tmp/output_data \
-e GTFS_GEN_SEED=123 \
podigg-lc

Parameters must be passed using environment variables.

License

The PoDiGG generator is written by Ruben Taelman.

This code is copyrighted by Ghent University – imec and released under the MIT license.

podigg-lc's People

Contributors

Stargazers

Watchers

Forkers

cuulee constraintautomaton

podigg-lc's Issues

NODE_MEM doesn't work

The gtfs crashes with a Node out of memory error

info: >> Exporting to GTFS

<--- Last few GCs --->

 1379495 ms: Mark-sweep 1288.8 (1376.1) -> 1288.5 (1377.1) MB, 2426.2 / 0.0 ms [allocation failure] [GC in old space requested].
 1381833 ms: Mark-sweep 1288.5 (1377.1) -> 1288.5 (1340.1) MB, 2337.4 / 0.0 ms [last resort gc].
 1384150 ms: Mark-sweep 1288.5 (1340.1) -> 1288.5 (1337.1) MB, 2317.7 / 0.0 ms [last resort gc].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x22dc1f7cfb51 <JS Object>
    1: writeForStop(aka writeForStop) [/node_modules/podigg/lib/gtfs/GtfsBuilder.js:~147] [pc=0x3ce6ba1895ae] (this=0x22dc1f704381 <undefined>,stop=0x22c6585160f9 <an Object with map 0x5568686ac71>,i=5)
    2: /* anonymous */(aka /* anonymous */) [/node_modules/podigg/lib/gtfs/GtfsBuilder.js:~146] [pc=0x3ce6ba192c54] (this=0x22dc1f704381 <undefined>,connection=0x25ec5956efe9 <an Object with map...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [node]
 2: 0x109838c [node]

despite running with -e NODE_MEM=50000 as indicated in the docs. I believe the issue is that the parameter does not extend into the childProcess.spawn calls

As a dirty fix I replaced /usr/local/bin/node with a script that calls node --max_old_space_size=50000

Data generator exits with runtime error

docker run -it --rm \
-v $(pwd)/output_data:/tmp/output_data \
-e GTFS_GEN_SEED=123 \
podigg/podigg-lc

bails out with this error:

info: Generating with options:  seed=123
info: >> Generating stops
info: >> Generating edges
info: Clustering all stops
info: Clustering border stations
info: Connecting loose stops
info: >> Generating stops post-edges
info: >> Generating routes
info: Creating micro-routes
29 / 30
info: Creating macro-routes
info: >> Generating connections
info: >> Visualizing output
info: >> Exporting to GTFS
GTFS to linked connections converter use --help to discover more functions
Indexing services and trips succesful!
Connections: +29924
Routes: +4500
/node_modules/gtfs2lc/lib/StopsBuilder.js:31
        body['@graph'].forEach(function(element) {
                      ^

TypeError: Cannot read property 'forEach' of undefined
    at Request._callback (/node_modules/gtfs2lc/lib/StopsBuilder.js:31:23)
    at Request.self.callback (/node_modules/request/request.js:186:22)
    at emitTwo (events.js:106:13)
    at Request.emit (events.js:191:7)
    at Request.<anonymous> (/node_modules/request/request.js:1081:10)
    at emitOne (events.js:96:13)
    at Request.emit (events.js:188:7)
    at IncomingMessage.<anonymous> (/node_modules/request/request.js:1001:12)
    at IncomingMessage.g (events.js:291:16)
    at emitNone (events.js:91:20)

Column mismatch and parse errors

Occurs randomly with Docker.

info: >> Visualizing output
info: >> Exporting to GTFS
GTFS to linked connections converter use --help to discover more functions
Error reading calendar_dates.txt: Error: Parse Error: expected: '"' got: ':'. at ':["/node_m
events.js:160
      throw er; // Unhandled 'error' event
      ^
Error: Parse Error: expected: '"' got: ':'. at ':["/node_m
    at parseEscapedItem (/node_modules/fast-csv/lib/parser/parser.js:79:19)
    at ParserStream.parseLine [as parser] (/node_modules/fast-csv/lib/parser/parser.js:174:30)
    at ParserStream._parseLine [as _parse] (/node_modules/fast-csv/lib/parser/parser_stream.js:164:25)
    at ParserStream._transform (/node_modules/fast-csv/lib/parser/parser_stream.js:198:18)
    at ParserStream.Transform._read (_stream_transform.js:167:10)
    at ParserStream.Transform._write (_stream_transform.js:155:12)
    at doWrite (_stream_writable.js:307:12)
    at writeOrBuffer (_stream_writable.js:293:5)
    at ParserStream.Writable.write (_stream_writable.js:220:11)
    at ReadStream.ondata (_stream_readable.js:555:20)

Routes: +44Error reading routes.txt: Error: Unexpected Error: column header mismatch expected: 5 columns got: 7Routes: +110Error reading routes.txt: Error: Unexpected Error: column header mismatch expected: 5 columns got:

With parameters:

GTFS_GEN_SEED=111

GTFS_GEN_REGION__SIZE_X=2000 

GTFS_GEN_REGION__SIZE_Y=2000

GTFS_GEN_REGION__CELLS_PER_LATLON=200

GTFS_GEN_STOPS__STOPS=5000

GTFS_GEN_CONNECTIONS__DELAY_CHANCE=0.01

GTFS_GEN_CONNECTIONS__CONNECTIONS=800000

GTFS_GEN_ROUTES__ROUTES=5000

GTFS_GEN_ROUTES__MAX_ROUTE_LENGTH=50

GTFS_GEN_ROUTES__MIN_ROUTE_LENGTH=10 

GTFS_GEN_CONNECTIONS__ROUTE_CHOICE_POWER=1.3 

GTFS_GEN_CONNECTIONS__TIME_FINAL=63072000000

podigg / podigg-lc Goto Github PK

podigg-lc's Introduction

PoDiGG Linked Connections

Install

Usage

Command line

Docker

License

podigg-lc's People

Contributors

Stargazers

Watchers

Forkers

podigg-lc's Issues

NODE_MEM doesn't work

Data generator exits with runtime error

Column mismatch and parse errors

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent