js-tokens

Tiny JavaScript tokenizer.

MIT 25 个版本

JavaScript js ECMAScript es token tokens tokenize tokenizer regex regexp

主页

仓库

提交 Issue

安装

npm install js-tokens

yarn add js-tokens

pnpm add js-tokens

bun add js-tokens

README

js-tokens

The tiny, regex powered, lenient, almost spec-compliant JavaScript tokenizer that never fails.

import jsTokens from "js-tokens";

const jsString = 'JSON.stringify({k:3.14**2}, null /*replacer*/, "\\t")';

Array.from(jsTokens(jsString), (token) => token.value).join("|");
// JSON|.|stringify|(|{|k|:|3.14|**|2|}|,| |null| |/*replacer*/|,| |"\t"|)

➡️ Full readme

版本列表

10.0.0 2025-12-08

9.0.1 2024-11-22

9.0.0 2024-02-08

8.0.3 2024-02-03

8.0.2 2023-09-09

8.0.1 2023-01-16

8.0.0 2022-05-15

7.0.0 2021-04-25

6.0.0 2020-04-13

5.0.0 2019-06-15

4.0.0 2018-01-28

3.0.2 2017-06-28

3.0.1 2017-01-30

3.0.0 2017-01-11

2.0.0 2016-06-19

1.0.3 2016-03-27

1.0.2 2015-10-18

1.0.1 2015-06-20

1.0.0 2015-02-26

0.4.1 2015-02-21

0.4.0 2015-02-21

0.3.1 2015-01-06

0.3.0 2014-12-19

0.2.0 2014-06-19

0.1.0 2014-03-08