Giter VIP home page Giter VIP logo

turtle.io's Introduction

turtle.io

build status

turtle.io is very easy to get up and running! All you need to do is install it, and tell it what directory holds your web sites, & which hostnames to answer for.

You can also create complex web applications, with a familiar API.

Getting Started

  1. Install the module with: npm install turtle.io
  2. Create a script to load & start a server. You could use sample.js in the turtle.io directory (./node_modules/turtle.io) as a template, or see the examples below
  3. [Optional] You can override defaults by passing the factory an Object

The following examples assume you've installed turtle.io into /opt/turtleio, if this is not the case you need to edit the applicable file to correct the path.

Upstart

Use the provided upstart recipe: sudo sh -c 'cp node_modules/turtle.io/turtleio.conf /etc/init; initctl reload-configuration; service turtleio start;'

Systemd

Use the provided systemd service: sudo sh -c 'cp node_modules/turtle.io/turtleio.service /etc/systemd/system; systemctl enable turtleio; systemctl start turtleio;'

What about Windows?

It runs great on Windows, but you're on your own to daemonize it!

Examples

turtle.io requires a default virtual host to be specified, because it is the failover when a request can't be routed.

Virtual hosts

Virtual host keys are the hostname, and the value is the directory relative to "root".

const turtleio = require("turtle.io");
const server = turtleio({
    default: "mysite.com",
    port: 80,
    uid: 100,
    root: "/var/www",
    hosts: {
        "mysite.com": "mysite.com",
        "another-domain.com": "another-domain.com"
    }
});

server.start();

Benchmark with express.js

siege was used instead of ab because we want to compare accurate transaction rates.

Specs

  • Machine MacBook Air (Early '14) / Core i7 @ 1.7Ghz / 8GB ram / 512 flash / OS X 10.10.2
  • ulimit 2560
  • express 4.11.2
  • turtle.io 3.2.2
  • benchmark siege -c100 -b -q -H 'Connection: Keep-Alive' -t15S localhost:$@

Test

express.js

Hello World! from a route (content-length: 12), no allow header.

var express = require("express"),
    app = express();

app.get('/', function (req, res) {
  res.send('Hello World!')
})

app.listen(3000);
turtle.io

Hello World! html file streamed from disk (content-length: 53), has accurate allow header.

"use strict";

require("index.js")({
	default: "test",
	root: __dirname + "/sites",
	port: 8000,
	hosts: {
		"test" : "test"
	},
	logging: {
		enabled: false
	}
}).start();

Transactions/s

Transaction rates are similar.

  • turtle.io [1233.78, 1203.2, 1187.44] (1208.14 avg)
  • express [1105.85, 1124.25, 1167.57] (1132.56 avg)

Handling Uploads

The request object is passed to every route handler as the second argument, will have a body property with the payload from the Client. It will not be coerced to another format, so if you expect JSON, you'll have to JSON.parse() it yourself (for now).

API & decoration

canETag

Function (path, method)

Determines if a path can receive an ETag

etags.ignore

Array

Array of paths to ignore for ETag generation

etags.update

Function (state)

Updates the etag cache with new state

request

allow

String

Allowed HTTP methods

ip

Number

Request IP

parsed

Object

Parsed HTTP request

query

String

Parsed query string

server

Object

turtle.io instance

host

String

Virtual host handling the request.

response

error

Function (status, body)

Send an error response.

redirect

Function (url)

Send a redirection.

respond

Function (body[, status, headers])

Send a response.

send

Function (body[, status, headers])

Send a response.

Configuration

Configuration values can be set by passing an Object to the factory, or any time afterward.

address

String (0.0.0.0)

Network address to listen on.

cacheSize

Number (1000)

Size of LRU cache for Etag validation.

catchAll

Boolean (true)

Handle unterminated requests.

compress

Boolean (true)

Compress responses when supported.

default

String

[Required] Default hostname to handle requests which are not specified within vhosts; must be a valid entry within vhosts.

etags

Object

ETag middleware configuration.

To enable state propagation set notify to true, and when receiving new state pass to etags.update().

{
	notify: true,
	ignore: [], // Array of paths to ignore
	onchange: (eventName, serializedCache) => {
	... // serializedCache needs to be passed to other instances `etags.update()`
	},
	update: serializedCache => {
	... // Override if you want to do more than set new state
	}
}

headers

Object

Response headers. CORS is enabled by default.

hosts

Object

[Required] Virtual hosts the server will respond for, key is the hostname & value is the directory relative to root.

index

Array

Files to look for when accessing a directory resource.

json

Number (2)

Default "pretty" ident size

logging

Object

Logging configuration.

logging.enabled

Boolean (true)

Override & disable stdout emitting by setting to false.

logging.format

String (%v %h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-agent}i")

Common Log Format string of tokens, defaulting to standard Virtual Host format.

logging.level

String ("info")

Minimum Common Log Level which is emitted to stdout.

logging.time

String (D/MMM/YYYY:HH:mm:ss ZZ)

Format for the date/time portion of a log message.

maxBytes

Number (1048576)

Maximum request body size; when exceeded a 429 is sent.

port

Number (8000)

Port the server will listen on.

root

String ("")

Relative path to the web root directory.

seed

Number (625)

Seed for hashing of middleware with MurmurHash3.

ssl.cert

String

[Optional] SSL certificate file path

ssl.key

String

[Optional] SSL certificate key/pem file path

ssl.pfx

String

[Optional] SSL certificate pfx file path

uid

Number (null)

[Optional] UID the server runs as.

License

Copyright (c) 2017 Jason Mulligan Licensed under the BSD-3 license.

turtle.io's People

Contributors

avoidwork avatar ergoithz avatar gitter-badger avatar mithgol avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

turtle.io's Issues

Etag can represent multiple URIs

The wrong asset will be returned based on the Etag without verifying the Etag belongs to the URI. Probably ideal to change filename.

Cached items have erroneous `etag` values in `this.registry`, if present at all

Debugging output:

{ etag: undefined, mimetype: 'text/html' }
{ etag: undefined, mimetype: 'text/css' }
{ etag: undefined, mimetype: 'text/css' }
{ mimetype: 'application/javascript' }
{ etag: undefined, mimetype: 'image/png' }
{ etag: undefined, mimetype: 'image/gif' }
{ etag: undefined, mimetype: 'application/javascript' }
{ etag: undefined, mimetype: 'application/octet-stream' }
{ mimetype: 'application/javascript' }
undefined
{ etag: undefined, mimetype: 'application/javascript' }
{ etag: undefined, mimetype: 'application/javascript' }
{ etag: undefined, mimetype: 'application/javascript' }
{ mimetype: 'application/javascript' }
{ mimetype: 'application/javascript' }
{ mimetype: 'application/javascript' }
{ etag: undefined, mimetype: 'application/javascript' }
{ etag: undefined, mimetype: 'application/javascript' }
{ mimetype: 'application/javascript' }
{ etag: undefined, mimetype: 'application/javascript' }
{ mimetype: 'application/octet-stream' }
undefined
{ mimetype: 'application/json' }
{ mimetype: 'text/css' }
{ mimetype: 'image/x-icon' }
{ etag: undefined, mimetype: 'application/javascript' }
{ etag: undefined, mimetype: 'application/javascript' }
{ etag: undefined, mimetype: 'application/javascript' }
{ etag: undefined, mimetype: 'application/javascript' }
{ etag: undefined, mimetype: 'application/javascript' }
{ mimetype: 'application/javascript' }
{ etag: undefined, mimetype: 'application/javascript' }
{ etag: undefined, mimetype: 'application/javascript' }
{ etag: undefined, mimetype: 'application/javascript' }
{ etag: undefined, mimetype: 'application/javascript' }
{ etag: undefined, mimetype: 'application/javascript' }
{ etag: undefined, mimetype: 'application/javascript' }
{ mimetype: 'application/javascript' }
{ mimetype: 'application/javascript' }
{ mimetype: 'application/javascript' }
{ etag: undefined, mimetype: 'application/javascript' }
{ etag: undefined, mimetype: 'application/javascript' }
{ etag: undefined, mimetype: 'text/html' }
{ etag: undefined, mimetype: 'text/html' }
{ mimetype: 'text/html' }
{ etag: undefined, mimetype: 'text/html' }
{ etag: undefined, mimetype: 'text/html' }
{ etag: undefined, mimetype: 'text/html' }
{ etag: undefined, mimetype: 'text/html' }
{ etag: undefined, mimetype: 'text/html' }
{ etag: undefined, mimetype: 'text/html' }
{ mimetype: 'text/html' }
{ mimetype: 'text/html' }
{ mimetype: 'text/html' }
{ etag: undefined, mimetype: 'text/html' }
{ mimetype: 'text/html' }
{ etag: undefined, mimetype: 'text/html' }
{ mimetype: 'text/html' }
{ mimetype: 'text/html' }
{ etag: undefined, mimetype: 'text/html' }
{ etag: undefined, mimetype: 'text/html' }
{ etag: undefined, mimetype: 'text/html' }
{ mimetype: 'text/html' }
{ mimetype: 'text/html' }
{ etag: undefined, mimetype: 'text/html' }
{ etag: undefined, mimetype: 'text/html' }
{ etag: undefined, mimetype: 'text/html' }
{ mimetype: 'text/html' }
{ etag: undefined, mimetype: 'text/html' }
{ mimetype: 'text/html' }
{ mimetype: 'text/html' }
{ mimetype: 'text/html' }
{ mimetype: 'text/html' }
{ etag: undefined, mimetype: 'text/html' }
{ etag: undefined, mimetype: 'application/javascript' }
{ mimetype: 'application/javascript' }
{ mimetype: 'application/javascript' }
{ mimetype: 'application/javascript' }
{ etag: undefined, mimetype: 'application/json' }
{ etag: undefined, mimetype: 'application/json' }
{ mimetype: 'application/octet-stream' }
{ etag: undefined, mimetype: 'application/octet-stream' }
{ mimetype: 'application/octet-stream' }
{ etag: undefined, mimetype: 'application/octet-stream' }

Implement `toobusy`

This module will allow turtle.io to gracefully handle an overload scenario without requiring the dev to write the handler themselves.

Verify status code in logs

In-house testing of apps deployed through jenkins have 200 codes in the log file, where I expect to see 500 & 404.

Determine the cause, and write the appropriate fix, if one is valid.

Fix README.md

The current order of the content does not work well with groc for generating the API documentation.

Proxy JSONP responses are stripped of the "P"

abaaso returns coerced results which are directly usable; this breaks the JSONP expectation on the Client because the response will not be padded with a function, it'll be JSON.

Implement a `domain`

Implementing a domain is just future proofing the design for the intended role. There's no gain, yet.

Move some LRU work from workers to master ps

This refactoring would handle the case where a worker is killed that was watching one or more files.

As is, if workers are terminated the LRU wouldn't be updated when the files being watched change.

There's also a tiny gap where a new ps could answer a request without it's LRU populated because it's not sent on ready.

There's a bottleneck in the design

Comparing express.js vs turtle.io on the same DigitalOcean VPS, with siege, shows that turtle.io is approximately 4-8x slower, when it should be closer to 1:1 performance. There is a massive bottleneck somewhere in the pipe. The concurrency is generally the same, but the time to respond is drastically slower.

Most likely it's due to the constant lookup of permissions for URIs; caching the permissions should reduce latency. I'm sure the routing is also highly inefficient compared to the rest of the code.

expressjs:

# siege -b -t30S -c250 http://127.0.0.1:3000
** SIEGE 2.70
** Preparing 250 concurrent users for battle.
The server is now under siege...
Lifting the server siege...      done.
Transactions:              25278 hits
Availability:             100.00 %
Elapsed time:              29.41 secs
Data transferred:          97.13 MB
Response time:              0.29 secs
Transaction rate:         859.50 trans/sec
Throughput:             3.30 MB/sec
Concurrency:              246.49
Successful transactions:       25278
Failed transactions:               0
Longest transaction:            3.29
Shortest transaction:           0.00

turtle.io

# siege -b -t30S -c250 http://127.0.0.1     
** SIEGE 2.70
** Preparing 250 concurrent users for battle.
The server is now under siege..      done.
siege aborted due to excessive socket failure; you
can change the failure threshold in $HOME/.siegerc
Transactions:               3236 hits
Availability:              75.31 %
Elapsed time:              15.61 secs
Data transferred:           3.34 MB
Response time:              1.18 secs
Transaction rate:         207.30 trans/sec
Throughput:             0.21 MB/sec
Concurrency:              243.61
Successful transactions:        3236
Failed transactions:            1061
Longest transaction:            2.15
Shortest transaction:           0.00

Logging compressed assets sometimes doesn't include the Content-Length

This is a sample from local testing:

localhost:8000 - - [11/Aug/2013:10:13:04 -0400] "GET / HTTP/1.1" 200 4730 - "GraphicalHttpClient 1.0.8 (Macintosh; Mac OS X 10.8.4; en_CA)"
localhost:8000 - - [11/Aug/2013:10:13:05 -0400] "GET / HTTP/1.1" 200 - - "GraphicalHttpClient 1.0.8 (Macintosh; Mac OS X 10.8.4; en_CA)"
localhost:8000 - - [11/Aug/2013:10:13:05 -0400] "GET / HTTP/1.1" 200 4730 - "GraphicalHttpClient 1.0.8 (Macintosh; Mac OS X 10.8.4; en_CA)"
localhost:8000 - - [11/Aug/2013:10:13:05 -0400] "GET / HTTP/1.1" 200 4730 - "GraphicalHttpClient 1.0.8 (Macintosh; Mac OS X 10.8.4; en_CA)"

Find 25-50% performance

This ticket is more important than any other enhancement. The design of turtle.io is still too complex, and incurs an overhead that limits performance by 25-50%. It's likely that the worst offenders will be the functions that decorate the response.

Support middleware

This would allow massaging a response before it's sent to the Client. Useful for I18N localiziation, etc.

Can't get turtle.io to serve default contents.

Trying to run turtle.io with the default script ends up with this error when accessing a page.

TypeError: Cannot use 'in' operator to search for '/.*' in undefined
at Array. (/Users/jgilb13/Documents/node-workspace/jeffreygilbert.com/node_modules/turtle.io/lib/turtle.io.js:97:73)
at Object.array.each (/Users/jgilb13/Documents/node-workspace/jeffreygilbert.com/node_modules/turtle.io/node_modules/abaaso/lib/abaaso.js:415:13)
at Array.prototypes.array.each (/Users/jgilb13/Documents/node-workspace/jeffreygilbert.com/node_modules/turtle.io/node_modules/abaaso/lib/abaaso.js:7567:17)
at TurtleIO.allowed (/Users/jgilb13/Documents/node-workspace/jeffreygilbert.com/node_modules/turtle.io/lib/turtle.io.js:94:30)
at Array. (/Users/jgilb13/Documents/node-workspace/jeffreygilbert.com/node_modules/turtle.io/lib/turtle.io.js:133:13)
at Object.array.each (/Users/jgilb13/Documents/node-workspace/jeffreygilbert.com/node_modules/turtle.io/node_modules/abaaso/lib/abaaso.js:415:13)
at Array.prototypes.array.each (/Users/jgilb13/Documents/node-workspace/jeffreygilbert.com/node_modules/turtle.io/node_modules/abaaso/lib/abaaso.js:7567:17)
at TurtleIO.allows (/Users/jgilb13/Documents/node-workspace/jeffreygilbert.com/node_modules/turtle.io/lib/turtle.io.js:132:8)
at TurtleIO.respond (/Users/jgilb13/Documents/node-workspace/jeffreygilbert.com/node_modules/turtle.io/lib/turtle.io.js:1177:66)
at TurtleIO.error (/Users/jgilb13/Documents/node-workspace/jeffreygilbert.com/node_modules/turtle.io/lib/turtle.io.js:460:14)
at TurtleIO.request (/Users/jgilb13/Documents/node-workspace/jeffreygilbert.com/node_modules/turtle.io/lib/turtle.io.js:1108:10)
at TurtleIO. (/Users/jgilb13/Documents/node-workspace/jeffreygilbert.com/node_modules/turtle.io/lib/turtle.io.js:1622:9)
at op (/Users/jgilb13/Documents/node-workspace/jeffreygilbert.com/node_modules/turtle.io/lib/turtle.io.js:1321:14)
at TurtleIO.auth (/Users/jgilb13/Documents/node-workspace/jeffreygilbert.com/node_modules/turtle.io/lib/turtle.io.js:156:3)
at TurtleIO.route (/Users/jgilb13/Documents/node-workspace/jeffreygilbert.com/node_modules/turtle.io/lib/turtle.io.js:1373:7)
at Server. (/Users/jgilb13/Documents/node-workspace/jeffreygilbert.com/node_modules/turtle.io/lib/turtle.io.js:1650:12)
at Server.EventEmitter.emit (events.js:98:17)
at HTTPParser.parser.onIncoming (http.js:2056:12)
at HTTPParser.parserOnHeadersComplete as onHeadersComplete
at Socket.socket.ondata (http.js:1946:22)
at TCP.onread (net.js:525:27)

Potential crypto flaw around session detection

A co-worker has been able to reproduce a crypto.js bug bubble out from session.get(), but this code shouldn't even be executed. The result is an invalid cipher execution.

Sven_v8 on irc had this suggestion: "could it be that the message you put in has to be padded to a length that is a multiple of the block size?"

Erroneous cache behaviour

If a Client sends an Etag header with a request, and there's a corresponding file on disk, it's sent as a naive short cut. An LRU validation should be done, to ensure the Etag is the most current rep, which should so solve the stale response.

Implement route variables

Match any variable in the parsed path using the same pattern as every other routing framework, ":something".

Proxy routes can trigger a 502 erroneously

This is a side effect of 0.11.12, which points to an underlying design flaw which affects caching proxy responses in the cache server, or at the source. This needs a deep dive to fix.

Log time format string in config.json is wrong

The config.logs.time string ends with "MM:SS ZZ". It should be "mm:ss ZZ". "MM" is the two digit month and "SS" is hundredths of a second.

I'd also like to see "ss.SSS" on that string to give millisecond resolution.

proxying causes problems with javascript files

With javascript files the proxy is mutating every string with a forward slash causing js errors in the end result.

For example in the proxied jquery-1.8.1.js:
hrefNormalized: ( a.getAttribute("href") === "/a" ),
becomes
hrefNormalized: ( a.getAttribute("href") === "/reports/a" ),

Cached etags occassionally have problems

This was logged during work hours at FileTrek:

TypeError: Cannot call method 'replace' of undefined
    at TurtleIO.stale (/Users/kbuchanan/projects/git/ControlPanel/node_modules/turtle.io/lib/turtle.io.js:1505:29)
    at TurtleIO.register (/Users/kbuchanan/projects/git/ControlPanel/node_modules/turtle.io/lib/turtle.io.js:1047:9)
    at TurtleIO.respond (/Users/kbuchanan/projects/git/ControlPanel/node_modules/turtle.io/lib/turtle.io.js:1183:9)
    at TurtleIO.handle (/Users/kbuchanan/projects/git/ControlPanel/node_modules/turtle.io/lib/turtle.io.js:506:11)
    at /Users/kbuchanan/projects/git/ControlPanel/node_modules/turtle.io/lib/turtle.io.js:1106:9
    at Object.oncomplete (fs.js:107:15)

Compare against nginx

How does turtle.io compare to nginx as a proxy? Nginx is usually the proxy of choice for node developers because of its speed. The downside is you still need a webserver. Turtle.io could be the best of best worlds, if it is comparable.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.