lydanne / utf8mb3 Goto Github PK

View Code? Open in Web Editor NEW

Let mysql's utf8 encoding store four-byte characters such as emoji(让 mysql 的 utf8 编码存储表情符号这类的四字节字符).

License: MIT License

JavaScript 47.04% Shell 1.10% HTML 1.37% TypeScript 1.93% CSS 6.03% Rust 42.53%

utf8mb3's Introduction

utf8mb3

Let mysql's utf8 encoding store four-byte characters such as emoji(让 mysql 的 utf8 编码存储表情符号这类的四字节字符).

Feature

Four-byte characters to three-byte characters(四字节字符转三字节字符)
Repeated coding remains unchanged(重复编码内容不变)
support wasm module(支持 wasm 使用)

JS Use

Detailed description 支持 nodejs 和 browser 环境

const {
  encode,
  decode,
  include_encode_utf8mb3,
  include_utf8mb4,
} = require("utf8mb3");

const str = "汉😊😊🛝🛝🛝汉";
// const str = "😊🛝🛝";

console.log(str, str.length);

const utf8mb3 = encode(str);

console.log("encodeUtf8mb4", utf8mb3, utf8mb3.length); // encodeUtf8mb4 汉ꂶꂶ鷶鷶鷶汉 12
console.log("decodeUtf8mb4", decode(utf8mb3)); // decodeUtf8mb4 汉😊😊🛝🛝🛝汉
console.log("include_encode_utf8mb3", include_encode_utf8mb3(utf8mb3)); // include_encode_utf8mb3 true
console.log("include_utf8mb4", include_utf8mb4(str)); // include_utf8mb4 true
console.log("include_utf8mb4", include_utf8mb4("测试")); // include_utf8mb4 false

Rust Use

Detailed description

let str = "😊";
let result = utf8mb3::encode(str);
println!("encode: {}", result); // encode: 
println!("decode: {}", utf8mb3::decode(&result)); // decode: 😊

WASI Use

...

Performance

MacOS: 13 , MacBook Pro m1 2021

Source	10000000 😊 coding time	Test Script
nodejs native	684.573ms	commemorate/utf8mb3-js/test.js
rust native	61.85475ms	packages/utf8mb3/src/lib.rs
wasm run nodejs	96.927ms	packages/utf8mb3-wasm/examples/utf8mb3-js-nodejs-play
wasm run edge browser	100.6ms	packages/utf8mb3-wasm/examples/utf8mb3-js-browser-play
wasi run wasmedge AOT	35ms
wasi run warmer	60ms
wasi run wasmtime	70ms