mathiasbynens / regenerate Goto Github PK

View Code? Open in Web Editor NEW

362.0 21.0 34.0 321 KB

Generate JavaScript-compatible regular expressions based on a given set of Unicode symbols or code points.

Home Page: https://mths.be/regenerate

License: MIT License

JavaScript 99.26% HTML 0.74%

unicode regular-expressions regexp code-generation ecmascript javascript

regenerate's Issues

Matching lone surrogates only

See mathiasbynens/regexpu#16 and https://gist.github.com/mathiasbynens/bbe7f870208abcfec860.

var set = regenerate()
  .addRange(0xD800, 0xDBFF) // lone high surrogates
  .addRange(0xDC00, 0xDFFF); // lone low surrogates

var match = '𝌆'.match(RegExp('(' + set.toString() + ')'));
console.log(match == null);
// expected: true
// actual: false, since the surrogate halves are matched

Instead, it would make more sense to match lone surrogates only in such cases.

Optimization / “bug” with ranges

regenerate.fromCodePoints([ 0x20, 0x21 ]);
// → '[\\x20\\x21] (correct)

regenerate.fromCodePoints([ 0x20, 0x21, 0x23 ]);
// → '[\\x20-\\x21\\x23]', which works fine, but just '[\\x20\\x21\\x23]' would be better

Should Regenerate use more single-character escape sequences?

All of these, except for \v: http://mathiasbynens.be/notes/javascript-escapes#single

I originally didn’t do this because in character classes I find escape sequences that show the code point value to be more readable. E.g. compare [\x0D-\xFF] to [\n-\xFF] — IMHO the former is clearer.

Should this change?

Corrupted file on NPM repo: [email protected] file: regenerate.js

Hi, I'm currently having a problem where, upon reinstalling the [email protected] npm package the regenerate.js file is completely missing just ahead of "bcdefghijklmnopqrstuvwxyz~" on line 569. Is the npm package corrupted I wonder? I'd appreciate you confirming the files content please?

Non-minimal regex when adding a second range lower than the first

With the specific following instructions, regenerate produces a non-minimal (but correct) regex:

regenerate().addRange( 99, 100 ).addRange( 65, 99 ).toString()
'[cdA-c]'

By switching the order, we have a correct minimal regex:

regenerate().addRange( 65, 99 ).addRange( 99, 100 ).toString()
'[A-d]'

Playing a bit with the boundaries, it seems it only occur when the end of the second range is exactly the same as the start of the first range.

Bug in `.remove(set)`?

var ID_Start = regenerate(require('unicode-6.3.0/properties/ID_Start/code-points'));
var ID_Continue = regenerate(require('unicode-6.3.0/properties/ID_Continue/code-points'));
var set = ID_Continue.clone().remove(ID_Start);

set’s data structure looks messed up. Here’s the first 20 entries:

[ '30', // odd = start
  '3a', // even = end (exclusive, so at least 1 higher than `start`)
  '5f',
  '60',
  'b7',
  'b8',
  '300',
  '370',
  '375', // start
  '375', // end → WTF
  '387',
  '388',
  '38b', // start
  '38b', // end → WTF
  '483',
  '488',
  '591',
  '5be',
  '5bf',
  '5c0' ]

Invert regular expression?

I wonder if it's possible to negate the resulting regexp in regenerate.

One could apparently use negative look-ahead (like (?!expr)), but it is said to perform bad. One could also invert the result of regexp.test(str) but it might not be ideal either.

How would I go about to "flip" the ranges? Ideally, there would be a built-in method .invert(). Another possible approach would be to add the entire range of Unicode characters, then use .remove() and .removeRange() on that.

Avoid overhead of calling `dataFromCodePoints()` etc. internally

When passing CodePointSets as arguments, e.g. setC = regenerate(setA).difference(setB), the overhead of calling dataFromCodePoints() etc. internally can be avoided. We could optimize these cases, although it’s debatable whether it’s worth it.

For difference specifically, we could do something similar to what @inimino uses, now that we use the same data structure:

function dataDifferenceData(dataA, dataB) {
  var ret = [];
  var i = 0;
  var j = 0;
  var a;
  var b;
  var al = dataA.length;
  var bl = dataB.length;
  var last;
  var state = 0;
  if (!al) {
    return [];
  }
  if (!bl) {
    return dataA;
  }
  a = dataA[0];
  b = dataB[0];
  if (isNaN(a) || isNaN(b)) {
    throw Error('cset_difference: bad input');
  }
  for (;;) {
    if (a < b) {
      if (!(state & 1)) {
        if (a == last) {
          ret.pop();
        }
        else {
          ret.push(a);
        }
        last = a;
      }
      state ^= 2;
      a = ++i < al ? dataA[i] : 0x110000;
    } else {
      if (a == 0x110000 && b == 0x110000) {
        return ret;
      }
      if (state & 2) {
        if (b == last) {
          ret.pop();
        } else {
          ret.push(b);
        }
        last = b;
      }
      state ^= 1;
      b = ++j < bl ? dataB[j] : 0x110000;
    }
  }
}

Add `regenerate.prototype.remove(fn)`

Things like regenerate(codePoints).removeRange(0x010000, 0x10FFFF) are very slow, since it creates an array containing all the code points in that range, and then walks through the code points looking for matches.

Something like this would be a much faster solution in that case:

regenerate(codePoints).remove(function(codePoint) {
  return codePoint > 0xFFFF; // remove astral code points from the set
});

Incorrect optimization in surrogateMappings

The surrogate pair of 0x1F610 is \uD83D\uDE10 and it got lost during some RegExp optimizations.
Consider this test case:

const set = regenerate().addRange(0x1F000, 0x1F1FF).addRange(0x1F300, 0x1F5FF).add(0x1F610);
const testChar = String.fromCodePoint(0x1F610);

The RegExp in Unicode mode is correct:

console.log(set.toRegExp('u'));    // /[\u{1F000}-\u{1F1FF}\u{1F300}-\u{1F5FF}\u{1F610}]/u
testChar.match(set.toRegExp('u')); // match found

The RegExp in "legacy" mode is incorrect (missing surrogate pair for 0x1F610):

console.log(set.toRegExp());    // /[\uD83C\uD83D][\uDC00-\uDDFF\uDF00-\uDFFF]/
testChar.match(set.toRegExp()); // null

I know that patches are welcome, but I don't have spare time right now. Will look into the source code later :-)

Script to convert any Regenerate set to code that generates it

This is not really a bug report, but rather a script that is useful when generating test cases:

regenerate.prototype.toCode = function() {
    var data = this.data;
    // Iterate over the data per `(start, end)` pair.
    var index = 0;
    var start;
    var end;
    var length = data.length;
    var loneCodePoints = [];
    var ranges = [];
    while (index < length) {
        start = data[index];
        end = data[index + 1] - 1; // Note: the `- 1` makes `end` inclusive.
        if (start == end) {
            loneCodePoints.push('0x' + start.toString(16).toUpperCase());
        } else {
            ranges.push(
                'addRange(0x' + start.toString(16).toUpperCase() +
                ', 0x' + end.toString(16).toUpperCase() + ')'
            );
        }
        index += 2;
    }
    return 'regenerate(' + loneCodePoints.join(', ') + ')' +
        (ranges.length ? '.' + ranges.join('.') : '');
};

var set = regenerate(0x1D306, 'a').addRange(0x2CEE, 0x4DFF);
console.log(set.toCode());
// → 'regenerate(0x61, 0x1D306).addRange(0x2CEE, 0x4DFF)'

Let’s allow extending regenerate.prototype for plugins like this.

Allow chaining

E.g.

regenerate(regenerate.range(0x0000, 0x000A))
    .add(0x1D306)
    .add([0x00A9, 0x0192])
    .addRange(0x0100, 0x0200)
    .remove(0x0002)
    .removeRange(0x0100, 0x0190)
    .difference([0x0004, 0x0007])
    .intersection([0x192, 0x196])
    .toString();
    // [\\u0192\\u0192\\u0196]

(Silly example, but you get the idea.)

Avoid `Array#splice` where possible

As an optimization, we should replace calls to array.splice(a, b, c) where b=1 with just array[a] = c (and similar for other argument counts).

Add `difference` and others

// get an array of all BMP code points except U+0067, U+23FC, and U+F3FC
var codePoints = regenerate.difference(regenerate.range(0x0, 0xFFFF), [0x67, 0x23FC, 0xF3FC]);

Maybe also these:

union(array1, array2) / add(array1, array2)
intersect(array1, array2)
remove(array1, array2)

Docs: add examples

https://github.com/mathiasbynens/regenerate#regeneratedifferencearray1-array2

Constants for the ranges “BMP code point”, “lead surrogate”, “tail surrogate”

Are there constants for these ranges? If not, I’d add them.

BMP code point: [\0-\uD7FF\uE000-\uFFFF]
lead surrogate: [\uD800-\uDBFF]
tail surrogate: [\uDC00-\uDFFF]

Improve performance / decrease memory usage

At the moment, Regenerate internally stores an array of all code points in the current set in memory. This is nice and easy, but it gets pretty slow for large sets of code points. It might be better to make this array smaller by making use of ranges. For example, instead of:

[0, 1, 2, 3, 4, 5, 6, 10, 12, 13, 14, 15, 18]

…we could just store something like:

[[0, 6], 10, [12, 15], 18]

This would require a complete rewrite of Regenerate, though.

Add option to output ES6 `u`-compatible patterns

This enables the use of \u{…} which would simplify the output.

Write a fuzzer to try and find more bugs

We can use Regenerate v0.5.4 (for which the output is 100% trustworthy), patch its codePointToSymbol to the latest version, and then compare its results with v0.6.x results. Any differences in output are ~~most likely~~ definitely bugs.

Unexpected matches in the resulting RegExp

I was trying to use this for twemoji but I've found a very weird behavior I'm not sure it's me doing it wrong or there's a bug in here (haven't checked your source code yet).

So, the resulting RegExp matches # and every char between 0 and 9 and I've no idea what's going on and why is that, so I've prepared this test code:

var list = ["🇨🇳", "🇺🇸", "🇷🇺", "🇰🇷", "🇯🇵", "🇮🇹", "🇬🇧", "🇫🇷", "🇪🇸", "🇩🇪", "9⃣", "8⃣", "7⃣", "6⃣", "5⃣", "4⃣", "3⃣", "2⃣", "1⃣", "0⃣", "#⃣", "🚳", "🚱", "🚰", "🚯", "🚮", "🚦", "🚣", "🚡", "🚠", "🚟", "🚞", "🚝", "🚜", "🚛", "🚘", "🚖", "🚔", "🚐", "🚎", "🚍", "🚋", "🚊", "🚈", "🚆", "🚂", "🚁", "😶", "😴", "😯", "😮", "😬", "😧", "😦", "😟", "😛", "😙", "😗", "😕", "😑", "😐", "😎", "😈", "😇", "😀", "🕧", "🕦", "🕥", "🕤", "🕣", "🕢", "🕡", "🕠", "🕟", "🕞", "🕝", "🕜", "🔭", "🔬", "🔕", "🔉", "🔈", "🔇", "🔆", "🔅", "🔄", "🔂", "🔁", "🔀", "📵", "📯", "📭", "📬", "💷", "💶", "💭", "👭", "👬", "👥", "🐪", "🐖", "🐕", "🐓", "🐐", "🐏", "🐋", "🐊", "🐉", "🐈", "🐇", "🐆", "🐅", "🐄", "🐃", "🐂", "🐁", "🐀", "🏤", "🏉", "🏇", "🍼", "🍐", "🍋", "🌳", "🌲", "🌞", "🌝", "🌜", "🌚", "🌘", "🃏", "🅰", "🅱", "🅾", "🆎", "🆑", "🆒", "🆓", "🆔", "🆕", "🆖", "🆗", "🆘", "🆙", "🆚", "👷", "🛅", "🛄", "🛃", "🛂", "🛁", "🚿", "🚸", "🚷", "🚵", "🈁", "🈂", "🈲", "🈳", "🈴", "🈵", "🈶", "🈷", "🈸", "🈹", "🈺", "🉐", "🉑", "🌀", "🌁", "🌂", "🌃", "🌄", "🌅", "🌆", "🌇", "🌈", "🌉", "🌊", "🌋", "🌌", "🌏", "🌑", "🌓", "🌔", "🌕", "🌙", "🌛", "🌟", "🌠", "🌰", "🌱", "🌴", "🌵", "🌷", "🌸", "🌹", "🌺", "🌻", "🌼", "🌽", "🌾", "🌿", "🍀", "🍁", "🍂", "🍃", "🍄", "🍅", "🍆", "🍇", "🍈", "🍉", "🍊", "🍌", "🍍", "🍎", "🍏", "🍑", "🍒", "🍓", "🍔", "🍕", "🍖", "🍗", "🍘", "🍙", "🍚", "🍛", "🍜", "🍝", "🍞", "🍟", "🍠", "🍡", "🍢", "🍣", "🍤", "🍥", "🍦", "🍧", "🍨", "🍩", "🍪", "🍫", "🍬", "🍭", "🍮", "🍯", "🍰", "🍱", "🍲", "🍳", "🍴", "🍵", "🍶", "🍷", "🍸", "🍹", "🍺", "🍻", "🎀", "🎁", "🎂", "🎃", "🎄", "🎅", "🎆", "🎇", "🎈", "🎉", "🎊", "🎋", "🎌", "🎍", "🎎", "🎏", "🎐", "🎑", "🎒", "🎓", "🎠", "🎡", "🎢", "🎣", "🎤", "🎥", "🎦", "🎧", "🎨", "🎩", "🎪", "🎫", "🎬", "🎭", "🎮", "🎯", "🎰", "🎱", "🎲", "🎳", "🎴", "🎵", "🎶", "🎷", "🎸", "🎹", "🎺", "🎻", "🎼", "🎽", "🎾", "🎿", "🏀", "🏁", "🏂", "🏃", "🏄", "🏆", "🏈", "🏊", "🏠", "🏡", "🏢", "🏣", "🏥", "🏦", "🏧", "🏨", "🏩", "🏪", "🏫", "🏬", "🏭", "🏮", "🏯", "🏰", "🐌", "🐍", "🐎", "🐑", "🐒", "🐔", "🐗", "🐘", "🐙", "🐚", "🐛", "🐜", "🐝", "🐞", "🐟", "🐠", "🐡", "🐢", "🐣", "🐤", "🐥", "🐦", "🐧", "🐨", "🐩", "🐫", "🐬", "🐭", "🐮", "🐯", "🐰", "🐱", "🐲", "🐳", "🐴", "🐵", "🐶", "🐷", "🐸", "🐹", "🐺", "🐻", "🐼", "🐽", "🐾", "👀", "👂", "👃", "👄", "👅", "👆", "👇", "👈", "👉", "👊", "👋", "👌", "👍", "👎", "👏", "👐", "👑", "👒", "👓", "👔", "👕", "👖", "👗", "👘", "👙", "👚", "👛", "👜", "👝", "👞", "👟", "👠", "👡", "👢", "👣", "👤", "👦", "👧", "👨", "👩", "👪", "👫", "👮", "👯", "👰", "👱", "👲", "👳", "👴", "👵", "👶", "🚴", "👸", "👹", "👺", "👻", "👼", "👽", "👾", "👿", "💀", "💁", "💂", "💃", "💄", "💅", "💆", "💇", "💈", "💉", "💊", "💋", "💌", "💍", "💎", "💏", "💐", "💑", "💒", "💓", "💔", "💕", "💖", "💗", "💘", "💙", "💚", "💛", "💜", "💝", "💞", "💟", "💠", "💡", "💢", "💣", "💤", "💥", "💦", "💧", "💨", "💩", "💪", "💫", "💬", "💮", "💯", "💰", "💱", "💲", "💳", "💴", "💵", "💸", "💹", "💺", "💻", "💼", "💽", "💾", "💿", "📀", "📁", "📂", "📃", "📄", "📅", "📆", "📇", "📈", "📉", "📊", "📋", "📌", "📍", "📎", "📏", "📐", "📑", "📒", "📓", "📔", "📕", "📖", "📗", "📘", "📙", "📚", "📛", "📜", "📝", "📞", "📟", "📠", "📡", "📢", "📣", "📤", "📥", "📦", "📧", "📨", "📩", "📪", "📫", "📮", "📰", "📱", "📲", "📳", "📴", "📶", "📷", "📹", "📺", "📻", "📼", "🔃", "🔊", "🔋", "🔌", "🔍", "🔎", "🔏", "🔐", "🔑", "🔒", "🔓", "🔔", "🔖", "🔗", "🔘", "🔙", "🔚", "🔛", "🔜", "🔝", "🔞", "🔟", "🔠", "🔡", "🔢", "🔣", "🔤", "🔥", "🔦", "🔧", "🔨", "🔩", "🔪", "🔫", "🔮", "🔯", "🔰", "🔱", "🔲", "🔳", "🔴", "🔵", "🔶", "🔷", "🔸", "🔹", "🔺", "🔻", "🔼", "🔽", "🕐", "🕑", "🕒", "🕓", "🕔", "🕕", "🕖", "🕗", "🕘", "🕙", "🕚", "🕛", "🗻", "🗼", "🗽", "🗾", "🗿", "😁", "😂", "😃", "😄", "😅", "😆", "😉", "😊", "😋", "😌", "😍", "😏", "😒", "😓", "😔", "😖", "😘", "😚", "😜", "😝", "😞", "😠", "😡", "😢", "😣", "😤", "😥", "😨", "😩", "😪", "😫", "😭", "😰", "😱", "😲", "😳", "😵", "😷", "😸", "😹", "😺", "😻", "😼", "😽", "😾", "😿", "🙀", "🙅", "🙆", "🙇", "🙈", "🙉", "🙊", "🙋", "🙌", "🙍", "🙎", "🙏", "🚀", "🚃", "🚄", "🚅", "🚇", "🚉", "🚌", "🚏", "🚑", "🚒", "🚓", "🚕", "🚗", "🚙", "🚚", "🚢", "🚤", "🚥", "🚧", "🚨", "🚩", "🚪", "🚫", "🚬", "🚭", "🚲", "🚶", "🚹", "🚺", "🚻", "🚼", "🚽", "🚾", "🛀", "🇦", "🇧", "🇨", "🇩", "🇪", "🇫", "🇬", "🇭", "🇮", "🇯", "🇰", "🇱", "🇲", "🇳", "🇴", "🇵", "🇶", "🇷", "🇸", "🇹", "🇺", "🇻", "🇼", "🇽", "🇾", "🇿", "🌍", "🌎", "🌐", "🌒", "🌖", "🌗", "", "〰", "➰", "➗", "➖", "➕", "❕", "❔", "❓", "❎", "❌", "✨", "✋", "✊", "✅", "⛎", "⏳", "⏰", "⏬", "⏫", "⏪", "⏩", "™", "➿", "©", "®"];
var regenerate = require('regenerate');
var regenerated = regenerate.apply(null, list);
console.log(list.filter(function (chr) {
  return (0 <= chr && chr <= 9) || chr == '#';
}).length ?
  'There should be #0-9 in the RegExp' :
  'There should be NO #0-9 in the RegExp'
);
console.log(regenerated.toRegExp());

This should not result in the following RegExp:

/[#0-9\xA9\xAE\u2122\u23E9-\u23EC\u23F0\u23F3\u26CE\u2705\u270A\u270B\u2728\u274C\u274E\u2753-\u2755\u2795-\u2797\u27B0\u27BF\u3030\uE50A]|\uD83C[\uDCCF\uDD70\uDD71\uDD7E\uDD8E\uDD91-\uDD9A\uDDE6-\uDDFF\uDE01\uDE02\uDE32-\uDE3A\uDE50\uDE51\uDF00-\uDF20\uDF30-\uDF35\uDF37-\uDF7C\uDF80-\uDF93\uDFA0-\uDFC4\uDFC6-\uDFCA\uDFE0-\uDFF0]|\uD83D[\uDC00-\uDC3E\uDC40\uDC42-\uDCF7\uDCF9-\uDCFC\uDD00-\uDD3D\uDD50-\uDD67\uDDFB-\uDE40\uDE45-\uDE4F\uDE80-\uDEC5]/

'cause #0-9 is absolutely undesired as a match.

Thanks for any sort of outcome.

Don’t move unmatched high surrogates to the end of the regex if it’s not needed

Currently:

// Create a regular expression that matches any BMP code point:
regenerate.fromCodePointRange(0x0000, 0xFFFF);
// → '[\\0-\\uD7FF\\uDC00-\\uFFFF]|[\\uD800-\\uDBFF]'

Any unmatched high surrogates need to be moved to the end of the regex only if the list of code points includes at least 1 astral symbol. This is not the case in the above example, so (I think) the result could just be:

// Create a regular expression that matches any BMP code point:
regenerate.fromCodePointRange(0x0000, 0xFFFF);
// → '[\\0-\\uFFFF]'

I might be missing something here, so I’m hoping someone like @slevithan can double-check this.

JavaScript identifier generation script broken since rewrite

https://gist.github.com/mathiasbynens/6334847 We should probably add this to the test suite.

The bug is somewhere in the private surrogateSet function.

Enable continuous integration for browsers using Testling

https://ci.testling.com/

This would require using https://github.com/twada/qunit-tap, or rewriting the tests to use a different testing framework that produces TAP output by default.

Optimize `regenerate.prototype.removeRange`

Ref. #12.

Instead of creating a range in such a case, just convert the arguments to code points (if they aren’t already) and loop over the code points in the set, removing those that are not within the bounds of the range.

`regenerate.fromCodePointRanges` or similar

Make it easier to create regular expressions that consist of a set of ranges:

E.g. when presented with a list like this (taken from http://www.w3.org/TR/turtle/#sec-grammar-grammar):

[A-Z] | [a-z] | [#x00C0-#x00D6] | [#x00D8-#x00F6] | [#x00F8-#x02FF] | [#x0370-#x037D] | [#x037F-#x1FFF] | [#x200C-#x200D] | [#x2070-#x218F] | [#x2C00-#x2FEF] | [#x3001-#xD7FF] | [#xF900-#xFDCF] | [#xFDF0-#xFFFD] | [#x10000-#xEFFFF]

Maybe something like this? (Using different code points than in the above example to keep it simple.)

regenerate.foo([
  '200C',
  'F900-FDCF',
  'FDF0-FFFD',
  '010000-0EFFFF'
]);

Any ideas on how to name such a method (foo in the example), or ideas on what would be a better API are more than welcome :) Perhaps we could overload fromCodePoints at the cost of performance?

isArray is not using the native isArray method, causing incompatibilities with prototype.js

isArray should use the browser's native implementation of isArray if available.
This causes errors when running regenerate together with prototype.js:

symbol.charCodeAt is not a function
	TypeError: symbol.charCodeAt is not a function
	    at symbolToCodePoint (http://localhost:1090/IPdeploy/js/systemjs-babel-browser.js:24493:23)
	    at regenerate.add (http://localhost:1090/IPdeploy/js/systemjs-babel-browser.js:24928:64)
	    at regenerate (http://localhost:1090/IPdeploy/js/systemjs-babel-browser.js:24902:28)
	    at eval (http://localhost:1090/IPdeploy/js/systemjs-babel-browser.js:25033:8)

whatwg/javascript#12
http://esdiscuss.org/topic/isillformedinecma
https://bugs.ecmascript.org/show_bug.cgi?id=1553: '\08' in strict mode throws a SyntaxError in IE/JSC/SpiderMonkey

Fix coverage report

For some reason the HTML coverage report isn’t generated at all. This seems to be a very weird Istanbul issue: gotwarlost/istanbul#55 (comment).

Test more environments

I want to add this to the README:

## Support

Regenerate has been tested in at least Chrome 27-29, Firefox 3-22, Safari 4-6, Opera 10-12, IE 6-10, Node.js v0.10.0, Narwhal 0.3.2, RingoJS 0.8-0.9, and Rhino 1.7RC4.

Let’s make it happen! Environments to test:

The test suite is pretty damn slow, but hey. Testing Rhino can take up to 90 seconds on my machine. Narwhal: 150 seconds.

mathiasbynens / regenerate Goto Github PK

regenerate's Issues

Recommend Projects

Recommend Topics

Recommend Org