/home/mrjunejune/zenbu: third_party/bun/node_modules/js-tokens/README.md annotate

annotate third_party/bun/node_modules/js-tokens/README.md @ 71:75de5903355c

Giagantic changes that update Dowa library to be more align with stb style array and hashmap. Updated Seobeo to be caching on server side instead of file level caching. Deleted bunch of things I don't really use.

author	June Park <parkjune1995@gmail.com>
date	Sun, 28 Dec 2025 20:34:22 -0800
parents	de54585a40f1
children

rev	line source
12 de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	1 Overview [![Build Status](https://travis-ci.org/lydell/js-tokens.svg?branch=master)](https://travis-ci.org/lydell/js-tokens)
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	2 ========
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	3
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	4 A regex that tokenizes JavaScript.
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	5
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	6 ```js
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	7 var jsTokens = require("js-tokens").default
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	8
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	9 var jsString = "var foo=opts.foo;\n..."
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	10
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	11 jsString.match(jsTokens)
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	12 // ["var", " ", "foo", "=", "opts", ".", "foo", ";", "\n", ...]
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	13 ```
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	14
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	15
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	16 Installation
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	17 ============
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	18
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	19 `npm install js-tokens`
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	20
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	21 ```js
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	22 import jsTokens from "js-tokens"
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	23 // or:
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	24 var jsTokens = require("js-tokens").default
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	25 ```
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	26
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	27
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	28 Usage
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	29 =====
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	30
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	31 ### `jsTokens` ###
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	32
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	33 A regex with the `g` flag that matches JavaScript tokens.
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	34
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	35 The regex _always_ matches, even invalid JavaScript and the empty string.
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	36
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	37 The next match is always directly after the previous.
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	38
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	39 ### `var token = matchToToken(match)` ###
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	40
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	41 ```js
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	42 import {matchToToken} from "js-tokens"
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	43 // or:
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	44 var matchToToken = require("js-tokens").matchToToken
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	45 ```
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	46
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	47 Takes a `match` returned by `jsTokens.exec(string)`, and returns a `{type:
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	48 String, value: String}` object. The following types are available:
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	49
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	50 - string
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	51 - comment
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	52 - regex
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	53 - number
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	54 - name
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	55 - punctuator
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	56 - whitespace
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	57 - invalid
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	58
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	59 Multi-line comments and strings also have a `closed` property indicating if the
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	60 token was closed or not (see below).
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	61
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	62 Comments and strings both come in several flavors. To distinguish them, check if
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	63 the token starts with `//`, `/*`, `'`, `"` or `` ` ``.
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	64
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	65 Names are ECMAScript IdentifierNames, that is, including both identifiers and
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	66 keywords. You may use [is-keyword-js] to tell them apart.
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	67
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	68 Whitespace includes both line terminators and other whitespace.
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	69
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	70 [is-keyword-js]: https://github.com/crissdev/is-keyword-js
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	71
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	72
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	73 ECMAScript support
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	74 ==================
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	75
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	76 The intention is to always support the latest ECMAScript version whose feature
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	77 set has been finalized.
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	78
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	79 If adding support for a newer version requires changes, a new version with a
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	80 major verion bump will be released.
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	81
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	82 Currently, ECMAScript 2018 is supported.
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	83
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	84
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	85 Invalid code handling
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	86 =====================
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	87
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	88 Unterminated strings are still matched as strings. JavaScript strings cannot
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	89 contain (unescaped) newlines, so unterminated strings simply end at the end of
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	90 the line. Unterminated template strings can contain unescaped newlines, though,
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	91 so they go on to the end of input.
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	92
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	93 Unterminated multi-line comments are also still matched as comments. They
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	94 simply go on to the end of the input.
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	95
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	96 Unterminated regex literals are likely matched as division and whatever is
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	97 inside the regex.
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	98
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	99 Invalid ASCII characters have their own capturing group.
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	100
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	101 Invalid non-ASCII characters are treated as names, to simplify the matching of
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	102 names (except unicode spaces which are treated as whitespace). Note: See also
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	103 the [ES2018](#es2018) section.
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	104
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	105 Regex literals may contain invalid regex syntax. They are still matched as
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	106 regex literals. They may also contain repeated regex flags, to keep the regex
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	107 simple.
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	108
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	109 Strings may contain invalid escape sequences.
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	110
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	111
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	112 Limitations
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	113 ===========
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	114
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	115 Tokenizing JavaScript using regexes—in fact, _one single regex_—won’t be
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	116 perfect. But that’s not the point either.
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	117
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	118 You may compare jsTokens with [esprima] by using `esprima-compare.js`.
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	119 See `npm run esprima-compare`!
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	120
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	121 [esprima]: http://esprima.org/
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	122
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	123 ### Template string interpolation ###
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	124
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	125 Template strings are matched as single tokens, from the starting `` ` `` to the
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	126 ending `` ` ``, including interpolations (whose tokens are not matched
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	127 individually).
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	128
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	129 Matching template string interpolations requires recursive balancing of `{` and
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	130 `}`—something that JavaScript regexes cannot do. Only one level of nesting is
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	131 supported.
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	132
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	133 ### Division and regex literals collision ###
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	134
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	135 Consider this example:
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	136
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	137 ```js
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	138 var g = 9.82
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	139 var number = bar / 2/g
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	140
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	141 var regex = / 2/g
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	142 ```
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	143
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	144 A human can easily understand that in the `number` line we’re dealing with
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	145 division, and in the `regex` line we’re dealing with a regex literal. How come?
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	146 Because humans can look at the whole code to put the `/` characters in context.
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	147 A JavaScript regex cannot. It only sees forwards. (Well, ES2018 regexes can also
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	148 look backwards. See the [ES2018](#es2018) section).
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	149
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	150 When the `jsTokens` regex scans throught the above, it will see the following
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	151 at the end of both the `number` and `regex` rows:
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	152
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	153 ```js
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	154 / 2/g
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	155 ```
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	156
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	157 It is then impossible to know if that is a regex literal, or part of an
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	158 expression dealing with division.
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	159
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	160 Here is a similar case:
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	161
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	162 ```js
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	163 foo /= 2/g
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	164 foo(/= 2/g)
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	165 ```
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	166
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	167 The first line divides the `foo` variable with `2/g`. The second line calls the
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	168 `foo` function with the regex literal `/= 2/g`. Again, since `jsTokens` only
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	169 sees forwards, it cannot tell the two cases apart.
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	170
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	171 There are some cases where we _can_ tell division and regex literals apart,
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	172 though.
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	173
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	174 First off, we have the simple cases where there’s only one slash in the line:
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	175
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	176 ```js
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	177 var foo = 2/g
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	178 foo /= 2
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	179 ```
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	180
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	181 Regex literals cannot contain newlines, so the above cases are correctly
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	182 identified as division. Things are only problematic when there are more than
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	183 one non-comment slash in a single line.
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	184
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	185 Secondly, not every character is a valid regex flag.
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	186
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	187 ```js
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	188 var number = bar / 2/e
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	189 ```
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	190
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	191 The above example is also correctly identified as division, because `e` is not a
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	192 valid regex flag. I initially wanted to future-proof by allowing `[a-zA-Z]*`
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	193 (any letter) as flags, but it is not worth it since it increases the amount of
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	194 ambigous cases. So only the standard `g`, `m`, `i`, `y` and `u` flags are
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	195 allowed. This means that the above example will be identified as division as
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	196 long as you don’t rename the `e` variable to some permutation of `gmiyus` 1 to 6
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	197 characters long.
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	198
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	199 Lastly, we can look _forward_ for information.
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	200
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	201 - If the token following what looks like a regex literal is not valid after a
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	202 regex literal, but is valid in a division expression, then the regex literal
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	203 is treated as division instead. For example, a flagless regex cannot be
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	204 followed by a string, number or name, but all of those three can be the
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	205 denominator of a division.
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	206 - Generally, if what looks like a regex literal is followed by an operator, the
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	207 regex literal is treated as division instead. This is because regexes are
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	208 seldomly used with operators (such as `+`, `*`, `&&` and `==`), but division
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	209 could likely be part of such an expression.
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	210
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	211 Please consult the regex source and the test cases for precise information on
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	212 when regex or division is matched (should you need to know). In short, you
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	213 could sum it up as:
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	214
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	215 If the end of a statement looks like a regex literal (even if it isn’t), it
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	216 will be treated as one. Otherwise it should work as expected (if you write sane
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	217 code).
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	218
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	219 ### ES2018 ###
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	220
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	221 ES2018 added some nice regex improvements to the language.
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	222
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	223 - [Unicode property escapes] should allow telling names and invalid non-ASCII
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	224 characters apart without blowing up the regex size.
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	225 - [Lookbehind assertions] should allow matching telling division and regex
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	226 literals apart in more cases.
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	227 - [Named capture groups] might simplify some things.
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	228
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	229 These things would be nice to do, but are not critical. They probably have to
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	230 wait until the oldest maintained Node.js LTS release supports those features.
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	231
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	232 [Unicode property escapes]: http://2ality.com/2017/07/regexp-unicode-property-escapes.html
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	233 [Lookbehind assertions]: http://2ality.com/2017/05/regexp-lookbehind-assertions.html
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	234 [Named capture groups]: http://2ality.com/2017/05/regexp-named-capture-groups.html
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	235
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	236
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	237 License
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	238 =======
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	239
de54585a40f1 Adding bun and node modules. June Park <parkjune1995@gmail.com> parents: diff changeset	240 [MIT](LICENSE).

Mercurial

annotate third_party/bun/node_modules/js-tokens/README.md @ 71:75de5903355c