Mercurial

annotate third_party/bun/node_modules/js-tokens/README.md @ 169:295ac2e5ec00

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
[MrJuneJune] Created separate target for generating html from md.
author MrJuneJune <me@mrjunejune.com>
date Mon, 19 Jan 2026 17:33:18 -0800
parents de54585a40f1
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
12
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
1 Overview [![Build Status](https://travis-ci.org/lydell/js-tokens.svg?branch=master)](https://travis-ci.org/lydell/js-tokens)
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
2 ========
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
3
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
4 A regex that tokenizes JavaScript.
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
5
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
6 ```js
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
7 var jsTokens = require("js-tokens").default
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
8
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
9 var jsString = "var foo=opts.foo;\n..."
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
10
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
11 jsString.match(jsTokens)
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
12 // ["var", " ", "foo", "=", "opts", ".", "foo", ";", "\n", ...]
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
13 ```
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
14
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
15
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
16 Installation
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
17 ============
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
18
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
19 `npm install js-tokens`
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
20
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
21 ```js
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
22 import jsTokens from "js-tokens"
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
23 // or:
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
24 var jsTokens = require("js-tokens").default
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
25 ```
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
26
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
27
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
28 Usage
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
29 =====
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
30
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
31 ### `jsTokens` ###
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
32
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
33 A regex with the `g` flag that matches JavaScript tokens.
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
34
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
35 The regex _always_ matches, even invalid JavaScript and the empty string.
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
36
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
37 The next match is always directly after the previous.
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
38
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
39 ### `var token = matchToToken(match)` ###
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
40
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
41 ```js
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
42 import {matchToToken} from "js-tokens"
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
43 // or:
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
44 var matchToToken = require("js-tokens").matchToToken
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
45 ```
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
46
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
47 Takes a `match` returned by `jsTokens.exec(string)`, and returns a `{type:
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
48 String, value: String}` object. The following types are available:
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
49
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
50 - string
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
51 - comment
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
52 - regex
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
53 - number
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
54 - name
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
55 - punctuator
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
56 - whitespace
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
57 - invalid
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
58
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
59 Multi-line comments and strings also have a `closed` property indicating if the
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
60 token was closed or not (see below).
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
61
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
62 Comments and strings both come in several flavors. To distinguish them, check if
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
63 the token starts with `//`, `/*`, `'`, `"` or `` ` ``.
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
64
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
65 Names are ECMAScript IdentifierNames, that is, including both identifiers and
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
66 keywords. You may use [is-keyword-js] to tell them apart.
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
67
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
68 Whitespace includes both line terminators and other whitespace.
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
69
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
70 [is-keyword-js]: https://github.com/crissdev/is-keyword-js
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
71
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
72
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
73 ECMAScript support
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
74 ==================
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
75
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
76 The intention is to always support the latest ECMAScript version whose feature
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
77 set has been finalized.
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
78
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
79 If adding support for a newer version requires changes, a new version with a
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
80 major verion bump will be released.
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
81
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
82 Currently, ECMAScript 2018 is supported.
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
83
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
84
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
85 Invalid code handling
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
86 =====================
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
87
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
88 Unterminated strings are still matched as strings. JavaScript strings cannot
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
89 contain (unescaped) newlines, so unterminated strings simply end at the end of
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
90 the line. Unterminated template strings can contain unescaped newlines, though,
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
91 so they go on to the end of input.
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
92
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
93 Unterminated multi-line comments are also still matched as comments. They
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
94 simply go on to the end of the input.
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
95
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
96 Unterminated regex literals are likely matched as division and whatever is
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
97 inside the regex.
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
98
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
99 Invalid ASCII characters have their own capturing group.
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
100
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
101 Invalid non-ASCII characters are treated as names, to simplify the matching of
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
102 names (except unicode spaces which are treated as whitespace). Note: See also
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
103 the [ES2018](#es2018) section.
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
104
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
105 Regex literals may contain invalid regex syntax. They are still matched as
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
106 regex literals. They may also contain repeated regex flags, to keep the regex
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
107 simple.
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
108
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
109 Strings may contain invalid escape sequences.
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
110
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
111
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
112 Limitations
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
113 ===========
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
114
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
115 Tokenizing JavaScript using regexes—in fact, _one single regex_—won’t be
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
116 perfect. But that’s not the point either.
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
117
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
118 You may compare jsTokens with [esprima] by using `esprima-compare.js`.
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
119 See `npm run esprima-compare`!
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
120
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
121 [esprima]: http://esprima.org/
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
122
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
123 ### Template string interpolation ###
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
124
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
125 Template strings are matched as single tokens, from the starting `` ` `` to the
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
126 ending `` ` ``, including interpolations (whose tokens are not matched
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
127 individually).
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
128
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
129 Matching template string interpolations requires recursive balancing of `{` and
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
130 `}`—something that JavaScript regexes cannot do. Only one level of nesting is
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
131 supported.
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
132
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
133 ### Division and regex literals collision ###
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
134
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
135 Consider this example:
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
136
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
137 ```js
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
138 var g = 9.82
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
139 var number = bar / 2/g
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
140
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
141 var regex = / 2/g
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
142 ```
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
143
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
144 A human can easily understand that in the `number` line we’re dealing with
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
145 division, and in the `regex` line we’re dealing with a regex literal. How come?
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
146 Because humans can look at the whole code to put the `/` characters in context.
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
147 A JavaScript regex cannot. It only sees forwards. (Well, ES2018 regexes can also
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
148 look backwards. See the [ES2018](#es2018) section).
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
149
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
150 When the `jsTokens` regex scans throught the above, it will see the following
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
151 at the end of both the `number` and `regex` rows:
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
152
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
153 ```js
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
154 / 2/g
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
155 ```
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
156
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
157 It is then impossible to know if that is a regex literal, or part of an
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
158 expression dealing with division.
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
159
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
160 Here is a similar case:
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
161
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
162 ```js
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
163 foo /= 2/g
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
164 foo(/= 2/g)
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
165 ```
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
166
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
167 The first line divides the `foo` variable with `2/g`. The second line calls the
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
168 `foo` function with the regex literal `/= 2/g`. Again, since `jsTokens` only
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
169 sees forwards, it cannot tell the two cases apart.
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
170
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
171 There are some cases where we _can_ tell division and regex literals apart,
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
172 though.
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
173
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
174 First off, we have the simple cases where there’s only one slash in the line:
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
175
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
176 ```js
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
177 var foo = 2/g
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
178 foo /= 2
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
179 ```
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
180
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
181 Regex literals cannot contain newlines, so the above cases are correctly
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
182 identified as division. Things are only problematic when there are more than
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
183 one non-comment slash in a single line.
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
184
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
185 Secondly, not every character is a valid regex flag.
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
186
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
187 ```js
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
188 var number = bar / 2/e
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
189 ```
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
190
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
191 The above example is also correctly identified as division, because `e` is not a
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
192 valid regex flag. I initially wanted to future-proof by allowing `[a-zA-Z]*`
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
193 (any letter) as flags, but it is not worth it since it increases the amount of
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
194 ambigous cases. So only the standard `g`, `m`, `i`, `y` and `u` flags are
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
195 allowed. This means that the above example will be identified as division as
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
196 long as you don’t rename the `e` variable to some permutation of `gmiyus` 1 to 6
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
197 characters long.
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
198
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
199 Lastly, we can look _forward_ for information.
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
200
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
201 - If the token following what looks like a regex literal is not valid after a
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
202 regex literal, but is valid in a division expression, then the regex literal
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
203 is treated as division instead. For example, a flagless regex cannot be
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
204 followed by a string, number or name, but all of those three can be the
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
205 denominator of a division.
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
206 - Generally, if what looks like a regex literal is followed by an operator, the
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
207 regex literal is treated as division instead. This is because regexes are
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
208 seldomly used with operators (such as `+`, `*`, `&&` and `==`), but division
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
209 could likely be part of such an expression.
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
210
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
211 Please consult the regex source and the test cases for precise information on
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
212 when regex or division is matched (should you need to know). In short, you
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
213 could sum it up as:
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
214
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
215 If the end of a statement looks like a regex literal (even if it isn’t), it
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
216 will be treated as one. Otherwise it should work as expected (if you write sane
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
217 code).
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
218
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
219 ### ES2018 ###
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
220
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
221 ES2018 added some nice regex improvements to the language.
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
222
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
223 - [Unicode property escapes] should allow telling names and invalid non-ASCII
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
224 characters apart without blowing up the regex size.
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
225 - [Lookbehind assertions] should allow matching telling division and regex
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
226 literals apart in more cases.
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
227 - [Named capture groups] might simplify some things.
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
228
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
229 These things would be nice to do, but are not critical. They probably have to
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
230 wait until the oldest maintained Node.js LTS release supports those features.
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
231
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
232 [Unicode property escapes]: http://2ality.com/2017/07/regexp-unicode-property-escapes.html
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
233 [Lookbehind assertions]: http://2ality.com/2017/05/regexp-lookbehind-assertions.html
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
234 [Named capture groups]: http://2ality.com/2017/05/regexp-named-capture-groups.html
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
235
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
236
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
237 License
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
238 =======
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
239
de54585a40f1 Adding bun and node modules.
June Park <parkjune1995@gmail.com>
parents:
diff changeset
240 [MIT](LICENSE).