jshttp / etag Goto Github PK

View Code? Open in Web Editor NEW

248.0 11.0 32.0 84 KB

Create simple HTTP ETags

License: MIT License

JavaScript 100.00%

javascript nodejs etag

etag's People

Stargazers

Watchers

etag's Issues

Stats-based etags for files

When you have a stats object and you don't want to bother calculating the actually contents

Would you be interested in linking to npmcharts.com for the shields badge?

These currently both link to the npm page

What do you think about having them link to my site npmcharts.com instead 😛 ?

Link - https://npmcharts.com/compare/etag?minimal=true

Would still look the same, just link to the interactive chart

Remove "strong ETags" from Stats objects

Right now, when given a Stats object, we can generate a weak ETag or a strong ETag; the difference between the two is not the correct strong vs. weak difference. We really cannot emit any strong ETags without at least knowing the bytes of the body, so it shouldn't even be an option (stemmed from #11).

Broken when graceful-fs 3.0 loaded in node.js 0.11.13

If I nvm use v0.11.13 and re-run my app, it blows up on etag here:

TypeError: argument entity must be string, Buffer, or fs.Stats
    at etag (/proj/node_modules/express/node_modules/etag/index.js:55:11)
    at SendStream.setHeader (/proj/node_modules/express/node_modules/send/index.js:724:15)
    at SendStream.send (/proj/node_modules/express/node_modules/send/index.js:500:8)
    at onstat (/proj/node_modules/express/node_modules/send/index.js:585:10)
    at Object.oncomplete (fs.js:97:15)

versions in use: [email protected], [email protected]

fs.stat is not an object

from the index.js

function isstats(obj) {
  // not even an object
  if (obj === null || typeof obj !== 'object') {
    return false
  }

Above check fails for SpiderMonkey since it returns 'function' for new function instances. I don't know exactly how this check works. Why do we need the typeof stat is 'object' ?

Consider using XXHash instead of SHA1 for entitytag

Feature Request/Question
non-critical issue

XXHash has a considerable performance gain over using crypto.createHash('sha1')

I've created a gist to demonstrate this better (these benchmarks can take some time to complete). Note that there is some variance that could be addressed in the tests.

The benchmark generates a large random(-ish) string ~97MB's and a smaller string, and runs each node crypto hashing method with each supported encoding format as a test in the suite. It also runs XXHash methods with the same string and various encoding formats as well.

For brevity here is the output on my 2018 Macbook Pro:

  116 tests completed.

  crypto-md4-ascii 97.6572265625MB     x     5,869 ops/sec ±1.00% (90 runs sampled)
  crypto-md4-ascii 0.0986328125MB      x   432,210 ops/sec ±7.83% (63 runs sampled)
  crypto-md4-utf8 97.6572265625MB      x     5,497 ops/sec ±1.21% (86 runs sampled)
  crypto-md4-utf8 0.0986328125MB       x   406,748 ops/sec ±8.37% (62 runs sampled)
  crypto-md4-base64 97.6572265625MB    x     5,576 ops/sec ±1.60% (84 runs sampled)
  crypto-md4-base64 0.0986328125MB     x   393,881 ops/sec ±13.88% (61 runs sampled)
  crypto-md4-latin1 97.6572265625MB    x     5,569 ops/sec ±1.54% (81 runs sampled)
  crypto-md4-latin1 0.0986328125MB     x   432,399 ops/sec ±13.79% (61 runs sampled)
  crypto-md4-binary 97.6572265625MB    x     5,651 ops/sec ±1.30% (88 runs sampled)
  crypto-md4-binary 0.0986328125MB     x   437,956 ops/sec ±14.20% (62 runs sampled)
  crypto-md4-hex 97.6572265625MB       x     5,449 ops/sec ±0.88% (87 runs sampled)
  crypto-md4-hex 0.0986328125MB        x   380,399 ops/sec ±13.02% (64 runs sampled)
  crypto-md5-ascii 97.6572265625MB     x     5,132 ops/sec ±1.10% (90 runs sampled)
  crypto-md5-ascii 0.0986328125MB      x   409,959 ops/sec ±13.71% (63 runs sampled)
  crypto-md5-utf8 97.6572265625MB      x     5,022 ops/sec ±1.08% (88 runs sampled)
  crypto-md5-utf8 0.0986328125MB       x   407,994 ops/sec ±13.21% (62 runs sampled)
  crypto-md5-base64 97.6572265625MB    x     5,034 ops/sec ±1.31% (88 runs sampled)
  crypto-md5-base64 0.0986328125MB     x   400,548 ops/sec ±13.24% (63 runs sampled)
  crypto-md5-latin1 97.6572265625MB    x     5,082 ops/sec ±1.38% (85 runs sampled)
  crypto-md5-latin1 0.0986328125MB     x   439,974 ops/sec ±13.55% (63 runs sampled)
  crypto-md5-binary 97.6572265625MB    x     5,147 ops/sec ±1.58% (88 runs sampled)
  crypto-md5-binary 0.0986328125MB     x   446,293 ops/sec ±13.66% (64 runs sampled)
  crypto-md5-hex 97.6572265625MB       x     4,998 ops/sec ±1.47% (85 runs sampled)
  crypto-md5-hex 0.0986328125MB        x   389,102 ops/sec ±13.14% (63 runs sampled)
  crypto-mdc2-ascii 97.6572265625MB    x       196 ops/sec ±1.25% (82 runs sampled)
  crypto-mdc2-ascii 0.0986328125MB     x   137,596 ops/sec ±7.14% (81 runs sampled)
  crypto-mdc2-utf8 97.6572265625MB     x       200 ops/sec ±1.29% (84 runs sampled)
  crypto-mdc2-utf8 0.0986328125MB      x   137,999 ops/sec ±6.44% (81 runs sampled)
  crypto-mdc2-base64 97.6572265625MB   x       193 ops/sec ±1.43% (80 runs sampled)
  crypto-mdc2-base64 0.0986328125MB    x   136,064 ops/sec ±5.58% (79 runs sampled)
  crypto-mdc2-latin1 97.6572265625MB   x       195 ops/sec ±1.20% (82 runs sampled)
  crypto-mdc2-latin1 0.0986328125MB    x   134,994 ops/sec ±7.44% (78 runs sampled)
  crypto-mdc2-binary 97.6572265625MB   x       196 ops/sec ±2.07% (82 runs sampled)
  crypto-mdc2-binary 0.0986328125MB    x   131,085 ops/sec ±8.12% (79 runs sampled)
  crypto-mdc2-hex 97.6572265625MB      x       185 ops/sec ±1.44% (78 runs sampled)
  crypto-mdc2-hex 0.0986328125MB       x   128,548 ops/sec ±6.51% (78 runs sampled)
  crypto-rmd160-ascii 97.6572265625MB  x     2,745 ops/sec ±1.05% (84 runs sampled)
  crypto-rmd160-ascii 0.0986328125MB   x   342,578 ops/sec ±13.44% (64 runs sampled)
  crypto-rmd160-utf8 97.6572265625MB   x     2,738 ops/sec ±1.59% (86 runs sampled)
  crypto-rmd160-utf8 0.0986328125MB    x   357,904 ops/sec ±11.34% (69 runs sampled)
  crypto-rmd160-base64 97.6572265625MB x     2,794 ops/sec ±1.51% (86 runs sampled)
  crypto-rmd160-base64 0.0986328125MB  x   349,517 ops/sec ±12.50% (66 runs sampled)
  crypto-rmd160-latin1 97.6572265625MB x     2,853 ops/sec ±0.87% (87 runs sampled)
  crypto-rmd160-latin1 0.0986328125MB  x   366,863 ops/sec ±12.63% (66 runs sampled)
  crypto-rmd160-binary 97.6572265625MB x     2,874 ops/sec ±1.13% (87 runs sampled)
  crypto-rmd160-binary 0.0986328125MB  x   345,423 ops/sec ±14.18% (63 runs sampled)
  crypto-rmd160-hex 97.6572265625MB    x     2,789 ops/sec ±2.96% (87 runs sampled)
  crypto-rmd160-hex 0.0986328125MB     x   353,724 ops/sec ±11.40% (70 runs sampled)
  crypto-sha1-ascii 97.6572265625MB    x     6,192 ops/sec ±3.77% (86 runs sampled)
  crypto-sha1-ascii 0.0986328125MB     x   402,152 ops/sec ±13.29% (63 runs sampled)
  crypto-sha1-utf8 97.6572265625MB     x     5,715 ops/sec ±4.38% (83 runs sampled)
  crypto-sha1-utf8 0.0986328125MB      x   406,294 ops/sec ±12.06% (64 runs sampled)
  crypto-sha1-base64 97.6572265625MB   x     6,397 ops/sec ±1.79% (89 runs sampled)
  crypto-sha1-base64 0.0986328125MB    x   400,805 ops/sec ±12.44% (61 runs sampled)
  crypto-sha1-latin1 97.6572265625MB   x     6,581 ops/sec ±0.95% (89 runs sampled)
  crypto-sha1-latin1 0.0986328125MB    x   462,122 ops/sec ±12.91% (65 runs sampled)
  crypto-sha1-binary 97.6572265625MB   x     6,608 ops/sec ±0.83% (90 runs sampled)
  crypto-sha1-binary 0.0986328125MB    x   425,688 ops/sec ±12.98% (63 runs sampled)
  crypto-sha1-hex 97.6572265625MB      x     6,133 ops/sec ±1.38% (87 runs sampled)
  crypto-sha1-hex 0.0986328125MB       x   409,450 ops/sec ±11.90% (64 runs sampled)
  crypto-sha224-ascii 97.6572265625MB  x     3,708 ops/sec ±1.24% (87 runs sampled)
  crypto-sha224-ascii 0.0986328125MB   x   368,919 ops/sec ±12.74% (65 runs sampled)
  crypto-sha224-utf8 97.6572265625MB   x     3,547 ops/sec ±0.91% (86 runs sampled)
  crypto-sha224-utf8 0.0986328125MB    x   371,195 ops/sec ±10.86% (68 runs sampled)
  crypto-sha224-base64 97.6572265625MB x     3,616 ops/sec ±1.13% (85 runs sampled)
  crypto-sha224-base64 0.0986328125MB  x   375,152 ops/sec ±12.52% (65 runs sampled)
  crypto-sha224-latin1 97.6572265625MB x     3,449 ops/sec ±3.88% (85 runs sampled)
  crypto-sha224-latin1 0.0986328125MB  x   415,873 ops/sec ±11.96% (65 runs sampled)
  crypto-sha224-binary 97.6572265625MB x     3,595 ops/sec ±1.41% (86 runs sampled)
  crypto-sha224-binary 0.0986328125MB  x   381,191 ops/sec ±12.30% (64 runs sampled)
  crypto-sha224-hex 97.6572265625MB    x     3,527 ops/sec ±1.53% (83 runs sampled)
  crypto-sha224-hex 0.0986328125MB     x   364,247 ops/sec ±11.43% (61 runs sampled)
  crypto-sha256-ascii 97.6572265625MB  x     3,664 ops/sec ±1.13% (88 runs sampled)
  crypto-sha256-ascii 0.0986328125MB   x   357,175 ops/sec ±13.35% (62 runs sampled)
  crypto-sha256-utf8 97.6572265625MB   x     3,649 ops/sec ±2.33% (87 runs sampled)
  crypto-sha256-utf8 0.0986328125MB    x   378,557 ops/sec ±10.78% (67 runs sampled)
  crypto-sha256-base64 97.6572265625MB x     3,381 ops/sec ±3.59% (81 runs sampled)
  crypto-sha256-base64 0.0986328125MB  x   380,514 ops/sec ±11.74% (64 runs sampled)
  crypto-sha256-latin1 97.6572265625MB x     3,612 ops/sec ±3.33% (86 runs sampled)
  crypto-sha256-latin1 0.0986328125MB  x   390,228 ops/sec ±13.14% (63 runs sampled)
  crypto-sha256-binary 97.6572265625MB x     3,489 ops/sec ±4.02% (82 runs sampled)
  crypto-sha256-binary 0.0986328125MB  x   401,851 ops/sec ±12.46% (65 runs sampled)
  crypto-sha256-hex 97.6572265625MB    x     3,705 ops/sec ±1.13% (90 runs sampled)
  crypto-sha256-hex 0.0986328125MB     x   367,077 ops/sec ±10.88% (66 runs sampled)
  crypto-sha384-ascii 97.6572265625MB  x     4,977 ops/sec ±1.02% (90 runs sampled)
  crypto-sha384-ascii 0.0986328125MB   x   400,388 ops/sec ±11.84% (68 runs sampled)
  crypto-sha384-utf8 97.6572265625MB   x     5,068 ops/sec ±0.92% (92 runs sampled)
  crypto-sha384-utf8 0.0986328125MB    x   375,418 ops/sec ±9.31% (71 runs sampled)
  crypto-sha384-base64 97.6572265625MB x     4,763 ops/sec ±3.88% (87 runs sampled)
  crypto-sha384-base64 0.0986328125MB  x   392,791 ops/sec ±10.86% (68 runs sampled)
  crypto-sha384-latin1 97.6572265625MB x     4,858 ops/sec ±1.37% (87 runs sampled)
  crypto-sha384-latin1 0.0986328125MB  x   381,373 ops/sec ±13.55% (61 runs sampled)
  crypto-sha384-binary 97.6572265625MB x     4,703 ops/sec ±3.34% (85 runs sampled)
  crypto-sha384-binary 0.0986328125MB  x   409,602 ops/sec ±12.34% (64 runs sampled)
  crypto-sha384-hex 97.6572265625MB    x     5,009 ops/sec ±1.05% (90 runs sampled)
  crypto-sha384-hex 0.0986328125MB     x   361,137 ops/sec ±10.16% (64 runs sampled)
  crypto-sha512-ascii 97.6572265625MB  x     4,797 ops/sec ±1.58% (88 runs sampled)
  crypto-sha512-ascii 0.0986328125MB   x   359,128 ops/sec ±12.08% (61 runs sampled)
  crypto-sha512-utf8 97.6572265625MB   x     4,516 ops/sec ±3.45% (83 runs sampled)
  crypto-sha512-utf8 0.0986328125MB    x   342,422 ops/sec ±8.67% (70 runs sampled)
  crypto-sha512-base64 97.6572265625MB x     4,892 ops/sec ±2.97% (89 runs sampled)
  crypto-sha512-base64 0.0986328125MB  x   377,239 ops/sec ±10.40% (65 runs sampled)
  crypto-sha512-latin1 97.6572265625MB x     4,123 ops/sec ±6.34% (77 runs sampled)
  crypto-sha512-latin1 0.0986328125MB  x   334,425 ops/sec ±15.30% (55 runs sampled)
  crypto-sha512-binary 97.6572265625MB x     4,424 ops/sec ±5.94% (81 runs sampled)
  crypto-sha512-binary 0.0986328125MB  x   378,151 ops/sec ±12.28% (63 runs sampled)
  crypto-sha512-hex 97.6572265625MB    x     4,893 ops/sec ±1.53% (91 runs sampled)
  crypto-sha512-hex 0.0986328125MB     x   351,804 ops/sec ±9.10% (64 runs sampled)
  xxhash-buffer 97.6572265625MB        x    42,855 ops/sec ±3.12% (82 runs sampled)
  xxhash-buffer 0.0986328125MB         x   498,514 ops/sec ±9.30% (53 runs sampled)
  xxhash-hex 97.6572265625MB           x    45,653 ops/sec ±1.53% (87 runs sampled)
  xxhash-hex 0.0986328125MB            x 1,125,398 ops/sec ±6.86% (69 runs sampled)
  xxhash-base64 97.6572265625MB        x    47,212 ops/sec ±1.20% (89 runs sampled)
  xxhash-base64 0.0986328125MB         x 1,280,717 ops/sec ±6.34% (73 runs sampled)
  xxhash-binary 97.6572265625MB        x    47,401 ops/sec ±1.21% (88 runs sampled)
  xxhash-binary 0.0986328125MB         x 1,372,796 ops/sec ±7.41% (67 runs sampled)

XXHash encoding format binary wins by a few cycles, but hex, and base64 are all comparable runner-ups. I have not run these against the current benchmarks used but could do so if desired.

Stop using checksums to generate ETags from bodies

This is a task that came from #11 : we need to just stop using CRC32, no matter how tempting it is because it's fast, to generate ETags based of the body. This library uses CRC32 for the "weak" setting when the body is less than, or equal to, 1KB currently.

/cc @stuartpb

quack quack: only check for properties that would actually be used?

https://github.com/jshttp/etag/blob/master/index.js#L87

thinking of just passing an object like { mtime, size }. what do you think?

Add a link to a definition of ETags in the Readme file

Presently, there is no context presented for what ETags are, so a simple link to here:
https://en.wikipedia.org/wiki/HTTP_ETag
would solve the problem.

Remember that etags also refers to a program to generate a type of source code map for browsing code in Emacs, so people could be easily confused without the reference.

How to generate an ETag from an object?

@dougwilson helped me in this issue expressjs/express#3947 with understanding how express generates ETag's. He said that the ETag is generated from the response body but how do you actually generate an ETag from a JSON object?

This simple test

'use strict'
const etag = require('etag')
const document = {}
const value = etag(document)
console.log('value = ', value)

Gives the error:

TypeError: argument entity must be string, Buffer, or fs.Stats
    at etag (/Users/tomcaflisch/Sites/ciitizen/ciitizen-profile-service/node_modules/etag/index.js:83:11)
    at Object.<anonymous> (/Users/tomcaflisch/Sites/ciitizen/ciitizen-profile-service/test.js:7:15)
    at Module._compile (internal/modules/cjs/loader.js:701:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:712:10)
    at Module.load (internal/modules/cjs/loader.js:600:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:539:12)
    at Function.Module._load (internal/modules/cjs/loader.js:531:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:754:12)
    at startup (internal/bootstrap/node.js:283:19)
    at bootstrapNodeJSCore (internal/bootstrap/node.js:622:3)

weak etags can collide (for body <= 1KB)

See expressjs/express#2667 - weak hashing is completely inappropriate for etags.

optionally include all the headers

http://en.wikipedia.org/wiki/HTTP_ETag#Strong_and_weak_validation

according to the wiki, strong etags mean that the entity headers are unchanged as well. we should optionally allow users include these headers and apply them into the hash

Why generated etag is wrapped in double qoutes?

why is Etag wrapped by double quotes?

> etag('a string')
'"8-VV0B5sgyZrPp+SvYEZBTcMr2J3A"'

what's the point of wrapping it like this?

Make the hashing algorithm(s) configurable

Pretty sure someone's eventually going to ask for this. Not a priority though lol

questions

why does weak default to true if the entity is a String? strings are pretty much buffers
stat-based etags should always be weak since we never check the actual bytes
i don't think there's a point in ever calculating with crc32. i think i benchmarked it earlier and found that crypto.createHash was faster than all of node's crc32 implementations (except maybe sse4_crc32 but that requires a native addon).

Could you please sync the etag algorithm with nginx.

https://serverfault.com/questions/690341/algorithm-behind-nginx-etag-generation

1828 
1829     etag->value.len = ngx_sprintf(etag->value.data, "\"%xT-%xO\"",
1830                                   r->headers_out.last_modified_time,
1831                                   r->headers_out.content_length_n)
1832                       - etag->value.data;
1833 
1834     r->headers_out.etag = etag;
1835 
1836     return NGX_OK;

Use of md5 is not FIPS compliant. Express server requests using etag fail with FIPS compiled NodeJS

We use Express v4 and our client requires any crypto providers be FIPS compliant. So, we build NodeJS v6.2 with the openssl fips module.

When we make requests to the server, they fail with the following error:

Message: Error: error:060A80A3:digital envelope routines:FIPS_DIGESTINIT:disabled for fips
  at Error (native)
  at new Hash (crypto.js:49:18)
  at Object.Hash (crypto.js:48:12)
  at entitytag (/usr/src/app/server/node_modules/etag/index.js:48:6)
  at etag (/usr/src/app/server/node_modules/etag/index.js:90:7)
  at wetag (/usr/src/app/server/node_modules/express/lib/utils.js:57:10)
  at ServerResponse.send (/usr/src/app/server/node_modules/express/lib/response.js:184:17)
  at ServerResponse.json (/usr/src/app/server/node_modules/express/lib/response.js:250:15)
  at app.use (/usr/src/app/server/node_modules/server/server.js:324:25)
  at Layer.handle_error (/usr/src/app/server/node_modules/express/lib/router/layer.js:71:5)

In our effort to use Node with FIPS compliant crypto, we have noticed several modules using md5, which is officially deemed insecure. Now, I realize etag is only using the hash to get a unique identifier for the response content, but the openssl-fips module has no way to know if a module is encrypting some sensitive content, or if it is just hashing a response. :/

We have proven a fix for this locally by simply changing md5 to sha256 on line 48 of the index.js. We think sha256 would be future proof, as sha1 is going to be insecure pretty soon, most likely. This fix would greatly help us in our efforts to prepare for a security penetration test. :)

We realize that use of sha will degrade performance, so what would you think of adding something like this:

A private var, which allows us to customize the hash algorithm.

var etagHashAlgorithm = process.env.NODE_ETAG_HASH_ALG || 'md5';

Use the customized, or defaulted value

.createHash(etagHashAlgorithm)

I've opened a PR for this: #18

Thank you very much for any support you can provide on this,
Alex

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.