token-types

Common token types for decoding and encoding numeric and string values

MIT 46 个版本

token integer unsigned numeric float IEEE 754 strtok3

主页

仓库

提交 Issue

安装

npm install token-types

yarn add token-types

pnpm add token-types

bun add token-types

README

token-types

A primitive token library used to read and write from a Uint8Array. Although it is possible to use this module directly, it is primarily designed to be used with strtok3 tokenizer.

Compatibility

Module: version 5 migrated from CommonJS to pure ECMAScript Module (ESM). JavaScript is compliant with ECMAScript 2019 (ES10).

Installation

npm install --save token-types

Usually in combination with strtok3:

npm install --save strtok3

Using TypeScript you should also install @tokenizer/token as a development dependency:

npm install --save-dev @tokenizer/token

Example

import * as strtok3 from 'strtok3';
import * as token from 'token-types';

(async () => {

  const tokenizer = await strtok3.fromFile("somefile.bin");
  try {
    const myNumber = await tokenizer.readToken(token.Float32_BE);
    console.log(`My number: ${myNumber}`);
  } finally {
    tokenizer.close(); // Close the file
  }
})();

Tokens

Numeric tokens

token-types supports a wide variety of numeric tokens out of the box:

Token	Number	Bits	Endianness
`UINT8`	Unsigned integer	8	n/a
`UINT16_BE`	Unsigned integer	16	big endian
`UINT16_LE`	Unsigned integer	16	little endian
`UINT24_BE`	Unsigned integer	24	big endian
`UINT24_LE`	Unsigned integer	24	little endian
`UINT32_BE`	Unsigned integer	32	big endian
`UINT32_LE`	Unsigned integer	32	little endian
`UINT64_BE`	Unsigned integer	64	big endian
`UINT64_LE`*	Unsigned integer	64	little endian
`INT8`	Signed integer	8	n/a
`INT16_BE`	Signed integer	16	big endian
`INT16_LE`	Signed integer	16	little endian
`INT24_BE`	Signed integer	24	big endian
`INT24_LE`	Signed integer	24	little endian
`INT32_BE`	Signed integer	32	big endian
`INT32_LE`	Signed integer	32	little endian
`INT64_BE`	Signed integer	64	big endian
`INT64_LE`*	Signed integer	64	little endian
`Float16_BE`	IEEE 754 float	16	big endian
`Float16_LE`	IEEE 754 float	16	little endian
`Float32_BE`	IEEE 754 float	32	big endian
`Float32_LE`	IEEE 754 float	32	little endian
`Float64_BE`	IEEE 754 float	64	big endian
`Float64_LE`	IEEE 754 float	64	little endian
`Float80_BE`*	IEEE 754 float	80	big endian
`Float80_LE`*	IEEE 754 float	80	little endian

(*) The tokens exceed the JavaScript IEEE 754 64-bit Floating Point precision, decoding and encoding is best effort based.

String tokens

StringType decoding is using @borewit/text-codec.

Custom tokens

Custom tokens can be added, suitable for reading binary files or network messages:

  ExtendedHeader = {
    len: 10,

    get: (buf, off) => {
      return {
        // Extended header size
        size: Token.UINT32_BE.get(buf, off),
        // Extended Flags
        extendedFlags: Token.UINT16_BE.get(buf, off + 4),
        // Size of padding
        sizeOfPadding: Token.UINT32_BE.get(buf, off + 6),
        // CRC data present
        crcDataPresent: common.strtokBITSET.get(buf, off + 4, 31)
      };
    }
  };

Licence

This project is licensed under the MIT License. Feel free to use, modify, and distribute as needed.

版本列表

6.1.2 2026-01-01

6.1.1 2025-08-13

6.1.0 2025-08-12

6.0.4 2025-07-25

6.0.3 2025-06-29

6.0.2 2025-06-29

6.0.1 2025-06-28

6.0.0 2024-07-04

5.0.1 2022-08-02

5.0.0-alpha.3 2022-08-01

5.0.0-alpha.2 2021-12-17

5.0.0-alpha.1 2021-12-16

5.0.0-alpha.0 2021-12-14

5.0.0 2022-08-02

4.2.1 2022-08-02

4.2.0 2022-02-23

4.1.1 2021-07-31

4.1.0 2021-07-17

4.0.1 2021-07-16

4.0.0 2021-07-16

3.1.0 2021-07-16

3.0.0 2021-07-10

2.1.1 2021-01-11

2.1.0 2021-01-10

2.0.0 2019-12-25

1.3.2 2019-12-24

1.3.0 2019-12-23

1.2.2 2019-12-15

1.2.1 2019-12-15

1.2.0 2019-12-14

1.1.0 2019-10-31

1.0.4 2019-10-16

1.0.3 2019-09-07

1.0.2 2019-08-21

1.0.1 2018-10-27

1.0.0 2018-09-12

0.10.0 2018-08-21

0.9.4 2018-05-08

0.9.3 2018-04-28

0.9.2 2017-10-04

0.9.1 2017-08-19

0.9.0 2017-08-12

0.1.2 2017-07-16

0.1.1 2017-07-15

0.0.2 2017-06-19

0.0.1 2017-06-18