A syntax highlighter based on TextMate grammars. ESM rewrite of shiki, with more features and capabilities.
Home Page: https://shikiji.netlify.app/
License: MIT License
Important
Shikiji is merged back to Shiki! This repository is now archived. Please continue at shikijs/shiki.
An ESM-focused rewrite of shiki, a beautiful syntax highlighter based on TextMate grammars, with more features and capabilities.
the current approach for dual themes is quite clever indeed, but it can be problematic in a few ways.
.shiki and span selectors. not a deal-breaker.!important, which is pretty much deprecated in this day and age of cascade layers. As a user, it is important (ha!) to me that i have control over the cascade, as i might need to change the colors for particular cases (such as text highlighting, print stylesheets, or forced colors mode).for reference, here is the css currently needed from the user's side for switching themes:
@media (prefers-color-scheme: dark) {
.shiki {
background-color: var(--shiki-dark-bg) !important;
color: var(--shiki-dark) !important;
& span {
color: var(--shiki-dark) !important;
}
}
}(edit: make sure to see update at the end)
i was just thinking about this, and realized that the space toggle technique would be a good fit for this use case, as it would allow users to switch values with a single css declaration.
@media (prefers-color-scheme: dark) {
:root {
--is-shiki-dark: var(--yes);
}
}(where --yes: initial; is just a convenience constant)
the inline style implementation would look like this, with full support for complex textmate grammar.
<pre style="--is-shiki-dark: /* no */;">
...
<span style="color: var(--is-shiki-dark, #ECEFF4) #6F42C1;">this is already very good, but we can take this idea further:
for full control over the cascade, a custom property could be used instead of color.
- <span style="color: var(--is-shiki-dark, #ECEFF4) #6F42C1;">
+ <span style="--shiki-color: var(--is-shiki-dark, #ECEFF4) #6F42C1;">and shikiji could ship with a small stylesheet, which can be imported into user's desired cascade layer:
:root {
--no: ;
--yes: initial;
--is-shiki-dark: var(--no);
}
.shiki {
background-color: var(--shiki-bg);
color: var(--shiki-color);
& span {
color: var(--shiki-color);
}
}(this --shiki-color is an implementation detail that could be obscured/minified into something like --_)s-c. the user only needs to care about the public api, which is --is-shiki-dark)
for supporting multiple themes, cyclic dependency space toggles can be used instead of regular space toggles.
<pre>
...
<span style="
--_s-c-light: hotpink;
--_s-c-dark: red;
--_s-c-hc-light: black; /* hc means "high contrast", _ means implementation detail */
--_s-c-hc-dark: white;
">and shikiji would ship with this small stylesheet that can be imported inside or outside a cascade layer.
:root {
--shiki-theme: var(--shiki--light);
--shiki--light: var(--shiki-theme,);
--shiki--dark: var(--shiki-theme,);
--shiki--hc-light: var(--shiki-theme,);
--shiki--hc-dark: var(--shiki-theme,);
}
.shiki,
.shiki span {
color:
var(--shiki--light, var(--_s-c-light))
var(--shiki--dark, var(--_s-c-dark))
var(--shiki--hc-light, var(--_s-c-hc-light))
var(--shiki--hc-dark, var(--_s-c-hc-dark));
}finally, the user only needs to set --shiki-theme in their own stylesheet.
:root {
--shiki-theme: var(--shiki--light);
@media (prefers-contrast: more) {
--shiki-theme: var(--shiki--hc-light);
}
@media (prefers-color-scheme: dark) {
--shiki-theme: var(--shiki--dark);
}
@media (prefers-color-scheme: dark) and (prefers-contrast: more) {
--shiki-theme: var(--shiki--hc-dark);
}
}here's a small demo showing this technique in action: stackblitz.com/edit/js-txy3pa
let me know what you think or if i can clarify anything.
As of 3 minutes ago, there is support for multiple themes, complete with the ability to modify prefixes. This makes the scope of this change much smaller :D At this point, I would be ok with just replacing inline color style with --shiki-light, so that !important is not necessary in the custom css. Everything else could be left upto the user.
css-variables theming, but as i understand this makes it difficult to achieve full support for complex syntax highlighting, so not ideal.
No response
请问怎么才能把这个项目跑起来呢？想把viteprss 文档跑起来，一直报错，试着删除 vitepress 补丁，安装最新的 vitepress，还是不行
How can I get this project started? I want to run the viteprss document, but keep getting errors. I'm trying to remove the vitepress patch and install the latest vitepress, but it still doesn't work?
https://stackblitz.com/~/github.com/fxzer/shikiji
System:
OS: macOS 14.2.1
CPU: (10) arm64 Apple M1 Pro
Shell: 5.8.1 - /bin/zsh
Binaries:
@antfu/ni v0.21.8
node v18.18.2
pnpm v8.10.0
npm -g v10.2.4pnpm
I want to setup a fine grained bundle for cloudflare worker but want to support all the languages from the web bundle with only one theme. I did something like this:
import { getHighlighterCore, bundledLanguages } from 'shikiji/bundle-web.mjs';
import githubDarkDimmed from 'shikiji/themes/github-dark-dimmed.mjs';
const coreHighlighter = await getHighlighterCore({
themes: [githubDarkDimmed],
langs: Object.values(bundledLanguages),
});
export default {
async fetch(request: Request): Promise<Response> {
const body = await request.text();
const html = coreHighlighter.codeToHtml(body, { theme: githubDarkDimmed, lang: 'html' });
return new Response(html);
},
};Doing something like above works but it also bundles all themes. See dist/index.js file after running pnpm wrangler deploy --dry-run --outdir dist to confirm.
As described above.
System:
OS: Linux 6.1 Arch Linux
CPU: (12) x64 AMD Ryzen 5 4600H with Radeon Graphics
Memory: 10.60 GB / 14.99 GB
Container: Yes
Shell: 5.9 - /bin/zsh
Binaries:
Node: 20.10.0 - ~/.local/share/pnpm/node
Yarn: 1.22.19 - /usr/bin/yarn
npm: 10.2.3 - ~/.local/share/pnpm/npm
pnpm: 8.13.1 - /usr/bin/pnpm
Browsers:
Brave Browser: 120.1.61.109pnpm
Hi there,
I'm unsure how to load custom themes via JSON. (For example, https://github.com/wesbos/cobalt2)
For example, I am using Shikiji as follows
import { bundledLanguages, bundledThemes, getHighlighter, Highlighter } from 'shikiji';
let highlighter: Highlighter
export type Theme = keyof typeof bundledThemes
export type Language = keyof typeof bundledLanguages
interface IHighlight {
codeString: string;
theme: Theme
language: Language
}
export async function highlight({
codeString,
theme = "dracula-soft",
language = "tsx"
}: IHighlight) {
highlighter = await getHighlighter({
themes: [theme],
langs: [language],
})
const [themedTokens, hast] = await Promise.all([
highlighter.codeToThemedTokens(codeString, {
lang: language,
theme: theme
}),
highlighter.codeToHast(codeString, {
lang: language,
theme: theme,
}),
])
return {
themedTokens,
hast,
}
}
I've tried to import themes and then use toShikiTheme to no avail (when I try to pass in a custom theme name, it says it's not supported)
Is it possible with this lib to load custom themes? If so, it would be helpful to see an example of it.
Provide an example or indication of if loading themes (not in BUNDLED_THEMES) is possible.
No response
No response
Hey ❤️
I'm using nuxt/content and since its using this package now via nuxt/mdc I hope I found the right place to address this.
As shown in the readme in the project: https://github.com/antfu/shikiji#class-based-dark-mode
<style>html.dark .shiki span {color: var(--shiki-dark) !important;background: var(--shiki-dark-bg) !important;font-style: var(--shiki-dark-font-style) !important;font-weight: var(--shiki-dark-font-weight) !important;text-decoration: var(--shiki-dark-text-decoration) !important;}</style>
The !important tag in background: var(--shiki-dark-bg) !important; break the highlight feature in nuxt/mdc as it makes it impossible to highlight certain lines in code blocks if dark mode is enabled.
In this case (I hope) a quick example is enough instead of a full reproduction:
Example.md:
```ts {1}
let foo = 'bar'
```
The {1} adds a highlight class to line 1 which enables specific styles for the lines specified.
This won't work anymore as it's been overwritten with !important as mentioned above.
If I manipulate my rendered html by removing the mentioned !important tag, it seems to work as expected.
cheers!
/
/pnpm
Firstly, thanks for your work on shikiji! It solves a very real pain point with shiki, and I'm excited to see the project continue.
I'm developing a project with Next.js 13 (app dir) and I'm running into the following error
error ./node_modules/shikiji/dist/index.mjs
export 'getHighlighterCore' (reexported as 'getHighlighterCore') was not found in './core.mjs' (module has no exports)Import trace for requested module:
./node_modules/shikiji/dist/index.mjs
./src/md-utils/index.js
./src/app/post/[postId]/edit/page.jsx
The mere act of importing shikiji causes the error
import { getHighlighter } from 'shikiji'
Note that I installed shikiji with npm install shikiji and it's listed in my package.json file as "shikiji": "^0.6.3"
(Will produce if necessary)
System:
OS: macOS 13.5.1
CPU: (10) arm64 Apple M1 Max
Memory: 1.23 GB / 64.00 GB
Shell: 5.9 - /bin/zsh
Binaries:
Node: 19.4.0 - ~/.nvm/versions/node/v19.4.0/bin/node
npm: 9.8.1 - ~/.nvm/versions/node/v19.4.0/bin/npm
Browsers:
Chrome: 116.0.5845.110
Safari: 16.6npm
Is there a way to make it show the full popup on hover?
https://shikiji.netlify.app/packages/twoslash
N/An/a
```js name="test" copy
console.log('test')
```
What if I want to get the name parameter?
I hope that the parameters in the meta can be exposed
No response
No response
Change #26 added the lang tag to pre nodes to indicate the programming language in use. However, the HTML spec says the lang tag should only be used for natural languages and not computer languages. Chrome's Lighthouse tool also flags the elements as having invalid (i.e. not BCP 47) tags.
There's additional discussion here about what the right approach is to include this data, which generally boil down to using class or data tags.
You may want to favor matching the CommonMark and Prism pattern of adding class="language-foo" to the code tag as a starting point.
-npm
At present, when rehype-shikiji and rehype-raw are used together, some features that rely on node.data will not work properly.
Because rehype-raw will cause node.data loss, refer to: syntax-tree/hast-util-raw#13
I will try to submit PR if possible.
I think we can refer to the approach in Astro: withastro/astro#5335
Simply put, if no node.data, try to retrieve it from the node.properties.metastring, then the user can preserve it before the rehype-raw.
import { unified } from 'unified'
import rehypeRaw from 'rehype-raw'
import remarkParse from 'remark-parse'
import remarkRehype from 'remark-rehype'
import rehypeStringify from 'rehype-stringify'
import rehypeShikiji from 'rehype-shikiji'
import { inspect } from 'unist-util-inspect'
import { visit } from 'unist-util-visit'
const text = `
<img src="https://example.com/image.png" alt="Example image" />
\`\`\`js {1}
import React from 'react'
\`\`\`
`
const rehypeMetaString = () => tree => {
visit(tree, 'element', node => {
if (node.tagName === 'code' && node.data?.meta) {
node.properties ??= {}
node.properties.metastring = node.data.meta
}
})
}
const file = await unified()
.use(remarkParse)
.use(remarkRehype, { allowDangerousHtml: true })
.use(() => tree => console.log('before rehypeRaw', inspect(tree), '\n'))
.use(rehypeMetaString) // <===
.use(rehypeRaw)
.use(() => tree => console.log('after rehypeRaw', inspect(tree), '\n'))
.use(rehypeShikiji, { highlightLines: true, theme: 'nord' })
.use(rehypeStringify)
.process(text)
console.log(file.toString())The several similar plugins I have seen so far have all solved this problem through this method.
No response
No response
MarkdownIt in the code
https://shikiji.netlify.app/packages/markdown-it
System:
OS: macOS 14.2.1
CPU: (8) arm64 Apple M1
Memory: 456.89 MB / 16.00 GB
Shell: 5.9 - /bin/zsh
Binaries:
Node: 20.10.0 - ~/Library/Caches/fnm_multishells/26215_1704811081596/bin/node
npm: 10.2.3 - ~/Library/Caches/fnm_multishells/26215_1704811081596/bin/npm
pnpm: 8.14.0 - ~/Library/pnpm/pnpm
Browsers:
Safari: 17.2.1Microsoft Edge: 120.0.2210.121 (正式版本) (arm64)
n/a
First off, thank you for this project! It simplifies things significantly. (
I'm not entirely sure if this is a bug or an accident it worked, but:
When switching from the original project to Shikiji:
Invoking codeToHtml with only { lang: "php"} worked fine originally, but after switching the following is produced in console, wherein the code is also not formatted:
Error: [shikiji] Invalid options, either
themeorthemesmust be provided
Adding theme to the codeToHtml invocation fixes things { lang: "php", theme: 'github-dark'} .
https://github.com/dpi/shikiji-theme-bug
System:
OS: macOS 13.4.1
CPU: (12) arm64 Apple M2 Pro
Memory: 61.70 MB / 32.00 GB
Shell: 3.6.1 - /opt/homebrew/bin/fish
Binaries:
Node: 18.15.0 - /usr/local/bin/node
npm: 9.5.0 - /usr/local/bin/npm
Browsers:
Safari: 16.5.1npm
After following the new enhancements from https://shikiji.netlify.app/packages/markdown-it#fine-grained-bundle, a couple things I'm noticing after trying it out via #42
In order to get the imports to properly resolve (at least in my pnpm project) I need to explicitly run pnpm add -D shikiji. I'm guessing this should be added to the docs for the markdown-it plugin, in the Fine-grained Bundle docs section.
fromHighlighter import pathI had to use an import path like this instead of what's in the docs
import { fromHighlighter } from 'markdown-it-shikiji' // works
import { fromHighlighter } from 'markdown-it-shikiji/core' // TypeScript complainsfromHighligherIf I utilize the example provided in the docs, I get the following type error:
Argument of type 'HighlighterCore' is not assignable to parameter of type 'HighlighterGeneric<any, any>'.
As seen here:
based on this usage:
import { fromHighlighter } from 'markdown-it-shikiji'
import { getHighlighterCore } from 'shikiji/core'
import { getWasmInlined } from 'shikiji/wasm'
import MarkdownIt from 'markdown-it'
const highlighter = await getHighlighterCore({
themes: [
import('shikiji/themes/github-light.mjs'),
import('shikiji/themes/github-dark.mjs'),
],
langs: [
import('shikiji/langs/typescript.mjs'),
],
loadWasm: getWasmInlined,
})
md.value = MarkdownIt({
html: false,
xhtmlOut: true,
linkify: true,
breaks: true,
typographer: true,
quotes: '“”‘’',
})
.use(fromHighlighter(highlighter, { // Type Error Here
themes: {
light: 'github-light',
dark: 'github-dark',
},
}))above
//pnpm
i want to get the background-color of the code theme without getting it from the dom
providing a var that have the velue of the background-color
.
Astro code block component
---
import type {
BuiltinLanguage,
BuiltinTheme,
LanguageRegistration,
SpecialLanguage,
ThemeRegistration,
ThemeRegistrationRaw,
} from 'shikiji';
import { getHighlighter } from "shikiji";
export interface Props {
code: string;
lang?: BuiltinLanguage | SpecialLanguage | LanguageRegistration;
theme?: BuiltinTheme | ThemeRegistration | ThemeRegistrationRaw ;
}
const {
code,
lang = 'plaintext',
theme = 'dark-plus',
} = Astro.props;
const shiki = await getHighlighter({
themes: [theme],
langs: [lang],
});
const codeOutput = shiki.codeToHtml(code, {
lang: lang as string,
theme: theme,
});
---
<div class="code app dark">
<div class="header">
<span class="dot"></span>
<span class="dot"></span>
<span class="dot"></span>
</div>
<Fragment set:html={codeOutput} />
</div>
<script>
const CodeEl = document.querySelector("pre.shiki");
const CodeApp = document.querySelector(".code.app");
CodeApp.style.backgroundColor = CodeEl.style.backgroundColor;
</script>我发现shiki.codeToHtml返回的字符串是已经处理pre与code的嵌套关系，这样就使得我们无法对pre内的内容自定义，比如添加语言类型的显示以及添加一个复制代码块的按钮。
我记得shiki的codeToHtml方法只返回code标签，pre标签应该是托尼自己在外包裹了一层，希望能有对外提供一下相应的插槽，好让我们能够添加额外的元素
No response
No response
Feature template mentions VueUse:
I guess I'm pretty ignorant about how these things work, as I can't find this in the source of the repo.
N/A
N/An/a
import MarkdownIt from 'markdown-it'
import Shikiji from 'markdown-it-shikiji'
const md = MarkdownIt()
md.use(await Shikiji({
themes: {
light: 'vitesse-light',
dark: 'vitesse-dark',
},
transformers: [
// transformerNotationDiff(),
transformerTwoSlash({
renderer: rendererRich, // <--
}),
],
}))highlightLines is overwritten with others block
System:
OS: macOS 13.6
CPU: (12) x64 Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
Memory: 11.10 GB / 32.00 GB
Shell: 5.9 - /bin/zsh
Binaries:
Node: 18.18.0 - ~/.nvm/versions/node/v18.18.0/bin/node
Yarn: 1.22.19 - ~/.nvm/versions/node/v18.18.0/bin/yarn
npm: 9.8.1 - ~/.nvm/versions/node/v18.18.0/bin/npm
pnpm: 8.10.0 - ~/.nvm/versions/node/v18.18.0/bin/pnpm
bun: 1.0.11 - ~/.bun/bin/bun
Watchman: 2023.09.25.00 - /usr/local/bin/watchman
Browsers:
Chrome: 120.0.6099.71
Safari: 16.6npm
I noticed that 'hcLight' and 'hcDark' are not included in the list of types for ThemeRegistrationResolved.
Wondering if we could add that?
No response
No response
checked on two phones:
Samsung Galaxy a23: chrome browser
Huawei p20 lite : brave browswernpm
Case: I am using the answer from OpenAI, last time I tried shikiji, it did not recognize some programming languages, resulting in an error.
Here is an example:
const code = await codeToHtml('const a = 1', { lang: 'jsw', theme: 'vitesse-dark' })Unhandled Promise Rejection: Error: [shikiji] Language `jsw` is not built-in.
In highlight.js there is a function getLanguage that allows us to check if a programming language exists, if it doesn't, we can assign plaintext
const language = hljs.getLanguage(lang) ? lang : 'plaintext'I suggest creating a getLanguage function or better yet, handling this situation inside shikiji to avoid a bug that breaks the UI
No response
No response
Currently, I'm trying to use Shikiji to add animated code blocks to a landing page. However, I can't find anything regarding to light/dark theme support for the codeToThemedTokens method. It seems to me that this is not yet a supported feature in Shikiji.
it's useful for rendering code blocks in React.js or other similar libraries, otherwise you can only get this feature with hast/markdown-it.
The codeToTokensWithThemes method does support multiple themes, but it isn't convenient to use. It returns an array of [color: string, theme: string, tokens: ThemedToken[][]] (according to typescript declarations), which need to be flattened into one before rendering them.
I hope there is a more simplified method for this, for instance, we can flatten the tokens under the hood:
interface MultipleThemedToken {
color: {
[theme: string]: string
}
}const tokens: MultipleThemedToken[][] = highlighter.codeToTokensWithThemes(code, { ... })No response
No response
Please first take a look at rehype-pretty-code word highlighting section
I want the same behavior with Shikiji but it seems transformerMetaWordHighlight only works with words without symbols
Thanks for your work ❤️
https://stackblitz.com/edit/vite-rfm8dh
System:
OS: Windows 11 10.0.22000
CPU: (16) x64 Intel(R) Core(TM) i7-10700 CPU @ 2.90GHz
Memory: 4.66 GB / 15.83 GB
Binaries:
Node: 18.18.2 - C:\Program Files\nodejs\node.EXE
Yarn: 1.22.21 - C:\Program Files\nodejs\yarn.CMD
npm: 9.8.1 - C:\Program Files\nodejs\npm.CMD
pnpm: 8.14.1 - C:\Program Files\nodejs\pnpm.CMD
Browsers:
Edge: Spartan (44.22000.120.0), Chromium (120.0.2210.144)
Internet Explorer: 11.0.22000.120pnpm
shiki replace colors for css-variables theme
do the same
or it will be better to write CSS variables directly instead of color
Note
Found another user wanted the same behaviour in shiki shikijs/shiki#33 (comment)
No response
When using with SvelteKit, MDSveX and the shikiji package, it all works well and pre-processes the files.
But, when i go to pre-process a sh codeblock, it throws an error.
i am trying to dynamically load the lanugages, which works, but not for sh. i also tried to specify sh in langs, but it still threwt he error.
this is my mdsvex.config.js
import rehypeSlug from 'rehype-slug';
import rehypeAutolinkHeadings from 'rehype-autolink-headings';
import rehypeExternalLinks from 'rehype-external-links';
import remarkFootnotes from 'remark-footnotes';
import { defineMDSveXConfig as defineConfig, escapeSvelte } from 'mdsvex';
import { getHighlighter } from 'shikiji'
// add any frequently used languages here
const langs = ['javascript', 'typescript', 'jsx', 'tsx', 'svelte', 'html', 'css', 'json', 'python']
const shiki = await getHighlighter({
langs,
themes: ['vitesse-light', 'vitesse-dark']
})
async function mdsvexShikijiHighlighter(code, lang) {
try {
if (!langs.includes(lang)) {
await shiki.loadLanguage(lang)
langs.push(lang)
}
const html = shiki.codeToHtml(code, {
lang,
themes: {
light: 'vitesse-light',
dark: 'vitesse-dark'
}
})
return `{@html \`${escapeSvelte(html)}\`}`
} catch(e) {
return code
}
}
const config = defineConfig({
extensions: ['.svelte.md', '.md', '.svx'],
smartypants: { dashes: 'oldschool' },
highlight: { highlighter: mdsvexShikijiHighlighter },
remarkPlugins: [ remarkFootnotes ],
rehypePlugins: [
[ rehypeExternalLinks, { target: '_blank', rel: ['nofollow', 'noreferrer'] }],
rehypeSlug,
[ rehypeAutolinkHeadings, { behavior: 'prepend', } ]
],
});
export default config;full error:
Error: Error while preprocessing D:/coding/svelte/pursiny-sveltekit/posts/dillinger-kitchen-sink 2023-10-25.md - No grammar provided for <source.shell>\n' +
' at a (file:///D:/coding/svelte/pursiny-sveltekit/node_modules/.pnpm/[email protected]/node_modules/shikiji/dist/core.mjs:413:15520)\n' +
' at t.ScopeDependencyProcessor.processQueue (file:///D:/coding/svelte/pursiny-sveltekit/node_modules/.pnpm/[email protected]/node_modules/shikiji/dist/core.mjs:413:17212)\n' +
' at Registry._loadGrammar (file:///D:/coding/svelte/pursiny-sveltekit/node_modules/.pnpm/[email protected]/node_modules/shikiji/dist/core.mjs:413:30884)\n' +
' at async Registry.loadLanguage (file:///D:/coding/svelte/pursiny-sveltekit/node_modules/.pnpm/[email protected]/node_modules/shikiji/dist/core.mjs:522:15)\n' +
' at async Registry.loadLanguages (file:///D:/coding/svelte/pursiny-sveltekit/node_modules/.pnpm/[email protected]/node_modules/shikiji/dist/core.mjs:546:7)\n' +
' at async Object.loadLanguage (file:///D:/coding/svelte/pursiny-sveltekit/node_modules/.pnpm/[email protected]/node_modules/shikiji/dist/core.mjs:647:5)\n' +
' at async mdsvexShikijiHighlighter (file:///D:/coding/svelte/pursiny-sveltekit/mdsvex.config.js:17:3)\n' +
' at async D:\\coding\\svelte\\pursiny-sveltekit\\node_modules\\.pnpm\\[email protected][email protected]\\node_modules\\mdsvex\\dist\\main.cjs.js:25745:19\n' +
' at async Promise.all (index 0)\n' +
' at async D:\\coding\\svelte\\pursiny-sveltekit\\node_modules\\.pnpm\\[email protected][email protected]\\node_modules\\mdsvex\\dist\\main.cjs.js:25742:4
the repository is private. This should be reproducible anywhere, as it appears to have a missing grammar. if it's not, let me know, and i'll provide a repro.
System:
OS: Windows 10 10.0.19045
CPU: (4) x64 Intel(R) Core(TM) i5-7300HQ CPU @ 2.50GHz
Memory: 11.99 GB / 23.88 GB
Binaries:
Node: 19.6.0 - C:\Program Files\nodejs\node.EXE
Yarn: 1.22.10 - ~\AppData\Roaming\npm\yarn.CMD
npm: 8.13.1 - C:\Program Files\nodejs\npm.CMD
pnpm: 8.9.2 - ~\AppData\Local\pnpm\pnpm.CMD
Browsers:
Edge: Spartan (44.19041.3570.0), Chromium (118.0.2088.61)
Internet Explorer: 11.0.19041.3570
Vivaldi 6.2.3105.58 (Stable channel) (64-bit) (wasn't recognized, primary browser)pnpm
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/120.0.0.0 Safari/537.36pnpm
Hi there,
I noticed that the ThemeRegistration type isn't compatible with the shape of the schema when one exports a current theme via VS Code.
$schema: 'vscode://schemas/color-theme',
ThemeRegistration seems to expect attributes name, bg, fg, and settings. All which don't appear in the vscode://schemas/color-theme schema. Wondering if there is a different type I should use that corresponds better with the vs code schema?
Everything works as expected when I pass in the theme JSON, but I would like to declare the correct type instead of just using any in typescript.
Thank you.
When converting TypeScript code to HTML, Shikiji creates a span that contains a leading space followed by the type name, for example <span> LogLevelStrings</span>. This behavior differs from Shiki which uses one span for the space and another one only for the type name.
Ultimately, this bug causes some themes, for example monokai, to be applied incorrectly since the underline under the type name starts early, being present also under the space.
Packages versions.
shiki: 0.14.7
shikiji: 0.9.12
Highlighting code with Shiki.
// shiki.cjs
const shiki = require("shiki");
shiki
.getHighlighter({
theme: "monokai",
})
.then((highlighter) => {
console.log(
highlighter.codeToHtml(`type LogLevelStrings = keyof typeof LogLevel;`, {
lang: "ts",
})
);
});Highlighting code with Shikiji.
// shikiji.mjs
import { getHighlighter } from "shikiji";
const highlighter = await getHighlighter({
themes: ["monokai"],
langs: ["typescript"],
});
const code = highlighter.codeToHtml(
"type LogLevelStrings = keyof typeof LogLevel;",
{
lang: "typescript",
theme: "monokai",
}
);
console.log(code);HTML output from Shiki and Shikiji.
<!-- Shiki -->
<pre
class="shiki monokai"
style="background-color: #272822"
tabindex="0"
><code><span class="line"><span style="color: #66D9EF; font-style: italic">type</span><span style="color: #F8F8F2"> </span><span style="color: #A6E22E; text-decoration: underline">LogLevelStrings</span><span style="color: #F8F8F2"> </span><span style="color: #F92672">=</span><span style="color: #F8F8F2"> </span><span style="color: #F92672">keyof</span><span style="color: #F8F8F2"> </span><span style="color: #F92672">typeof</span><span style="color: #F8F8F2"> LogLevel;</span></span></code></pre>
<!-- Shikiji -->
<pre
class="shiki monokai"
style="background-color: #272822; color: #f8f8f2"
tabindex="0"
><code><span class="line"><span style="color:#66D9EF;font-style:italic">type</span><span style="color:#A6E22E;text-decoration:underline"> LogLevelStrings</span><span style="color:#F92672"> =</span><span style="color:#F92672"> keyof</span><span style="color:#F92672"> typeof</span><span style="color:#F8F8F2"> LogLevel;</span></span></code></pre>HTML output differences visualized.
Bug reproduced on Shikiji Playground by selecting TypeScript and Monokai; type names after
enum,typeandkey:are affected.
VSCode rendering the same playground code with Monokai.
System:
OS: Linux 6.6 Fedora Linux 38 (Workstation Edition)
CPU: (8) x64 Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz
Memory: 10.18 GB / 15.51 GB
Container: Yes
Shell: 5.9 - /usr/bin/zsh
Binaries:
Node: 20.10.0 - /run/user/1000/fnm_multishells/28509_1703779162622/bin/node
npm: 10.2.3 - /run/user/1000/fnm_multishells/28509_1703779162622/bin/npm
pnpm: 8.12.1 - ~/.bun/bin/pnpm
bun: 1.0.20 - ~/.bun/bin/bun
Browsers:
Chrome: 120.0.6099.129pnpm
Besides line highlighting, we can also support highlighting a specific word/token, for example, a variable.
It is useful for writing code tutorials. As I am migrating from rehype-pretty-code, I need to find a solution for word highlighting.
I'm willing to submit a PR
Inspired by rehype-pretty-code, we can specify words by wrapping them in slashes:
```ts /a/ /b/
const a = "Hello"
```No response
No response
unjs/unwasm will be become soon the default wasm loaded for Nitro/Nuxt and already supported with shikiji!
Usage when unwasm enabled:
import { getHighlighterCore, loadWasm } from 'shikiji/core'
await loadWasm(() => import("shikiji/onig.wasm") as any)
const shiki = await getHighlighterCore()I am also wondering if we could remove the requirement of loadWasm entirely. Especially when import maps support lands (unjs/unwasm#10), we could directly provide those dependencies. For this we might support shikiji/esm that does not conflict with current subpath exports or breaking current users but a path to the future. Wdyt @antfu?
TODO list:
shikiji/esmMaybe instead of using a textmate dialect of some sorts, https://tree-sitter.github.io/tree-sitter/ could be used instead.
https://tree-sitter.github.io/tree-sitter/
No response
No response
when I use the code in the blow, shikiji gives a clue in the second picture.
hope that shikiji surpport the plian text language
No response
No response
Find a way to split script blocks in Vue SFC, pass them to TwoSlash, and merge them back.
Ideally it could be general and made work for more languages.
No response
No response
For now, shikiji is not intended to support inline code blocks, as transformers, integrations, and many others are not supposed to process inline code.
I want to highlight an inline code block, for example: console.log("Hello World").
Allow us to specify language via {js} console.log("Hello World"), and enable inline code processing for each integration.
No response
No response
In the current implementation of the project, real-time markdown processing in the browser suffers from performance issues due to the absence of result caching for code highlighting. Each keystroke triggers a re-computation and re-loads the grammar, which is not efficient, particularly when dealing with unchanged or static content. This repetitive processing leads to suboptimal performance, especially in applications where real-time content updates are frequent.
Some sort of caching mechanism that retains the results of previous highlight transformations result in memory. This is especially useful when building ai apps, as the previous output (streamed content) will not need to be transformed again. This should significantly reduce unnecessary computations and improve the responsiveness of the application.
Example Code
export const rehypeShikijiFromHighlighter: Plugin<
[
HighlighterGeneric<any, any>,
RehypeShikijiCoreOptions & {
cache?: Map<string, Root>;
}
],
Root
> = function (highlighter, options) {
const {
highlightLines = true,
addLanguageClass = false,
parseMetaString,
...rest
} = options;
const prefix = "language-";
const cache = options.cache;
return function (tree) {
visit(tree, "element", (node, index, parent) => {
// other code
const code = toString(head);
// check and reuse cache
if (cache && cache?.has(code)) {
parent.children.splice(index, 1, ...cache?.get(code).children);
return;
}
try {
const fragment = highlighter.codeToHast(code, codeOptions);
// write to cache
cache?.set(code, fragment);
parent.children.splice(index, 1, ...fragment.children);
} catch (error) {
if (options.onError) options.onError(error);
else throw error;
}
});
};
};Maybe also offload this to another thread using webworker? not sure this is a good idea
No response
You cannot highlight lines with the {1,3-4} syntax if you also have markdown-it-attrs added as a plugin, since it uses the same curly bracket notation elsewhere. This is more of a proposal than a bug report - is it worth adding a note to the docs about how to resolve this? I figured out that I can change the delimiter of markdown-it-attrs from { and } to other characters to fix this, and I think other people might encounter this as well. On the topic of changing delimiters, that could be something we consider for the markdown-it-shikiji line number syntax as well.
I found the following when trying a migration from eleventy-plugin-shiki-twoslash to markdown-it-shikiji:
```ts::my-annotation
<!-- … -->
```ts. Shikiji will error on the unknown language ts::my-annotation```js my-attribute
<!-- … -->
```<pre class="shiki <theme>" my-attribute```js class="my-class"
<!-- … -->
```<pre class="shiki <theme> my-class">…</pre><pre class="shiki shiki-themes <theme> [<theme> […]]">…</pre>spans for lines. Shiki-Twoslash uses divs. When migrating, can expect to have to add display:block to lines.tabindex="0" to its pres. Shiki-Twoslash doesn't. I'd be interested to hear the rationale, not something I've seen done.code .line { border-left: 1px solid red })I've only used Shiki in the context of Shiki-Twoslash (via the Eleventy plugin, and via the Shiki-Twoslash playground), and haven't dug into the implementation there. Maybe this is all expected? If so I'd be happy to PR an update to the compatibility docs page.
Lmk a good way to provide one
% pnpm envinfo --system --binaries --browsers
System:
OS: macOS 14.2.1
CPU: (8) arm64 Apple M1
Memory: 301.44 MB / 16.00 GB
Shell: 5.9 - /bin/zsh
Binaries:
Node: 18.17.1 - ~/.asdf/installs/nodejs/18.17.1/bin/node
Yarn: 1.22.21 - /opt/homebrew/bin/yarn
npm: 9.6.7 - ~/.asdf/installs/nodejs/18.17.1/bin/npm
pnpm: 8.14.1 - ~/Library/pnpm/pnpm
Watchman: 2023.12.04.00 - /opt/homebrew/bin/watchman
Browsers:
Brave Browser: 120.1.61.116
Chrome: 120.0.6099.234
Edge: 120.0.2210.144
Safari: 17.2.1pnpm
Loading any language after loading md throws an error:
import { getHighlighter } from "shikiji";
const highlighter = await getHighlighter();
await highlighter.loadLanguage("md");
await highlighter.loadLanguage("js");Error: No grammar provided for <source.js>
at a (file:///Users/sites/rhc-beta/node_modules/shikiji/dist/core.mjs:434:15520)
at t.ScopeDependencyProcessor.processQueue (file:///Users/sites/rhc-beta/node_modules/shikiji/dist/core.mjs:434:17212)
at Registry._loadGrammar (file:///Users/sites/rhc-beta/node_modules/shikiji/dist/core.mjs:434:30884)
at async Registry.loadLanguage (file:///Users/sites/rhc-beta/node_modules/shikiji/dist/core.mjs:559:15)
at async Registry.loadLanguages (file:///Users/sites/rhc-beta/node_modules/shikiji/dist/core.mjs:583:7)
at async Object.loadLanguage (file:///Users/sites/rhc-beta/node_modules/shikiji/dist/core.mjs:693:5)
at async file:///Users/sites/rhc-beta/index.mjs:5:1
Workaround is to load md last.
Version: 0.7.5
Shown above
System:
OS: macOS 13.4
CPU: (10) arm64 Apple M1 Pro
Memory: 50.33 MB / 32.00 GB
Shell: 5.9 - /bin/zsh
Binaries:
Node: 18.17.0 - ~/.nvm/versions/node/v18.17.0/bin/node
Yarn: 1.22.17 - /opt/homebrew/bin/yarn
npm: 9.6.7 - ~/.nvm/versions/node/v18.17.0/bin/npm
pnpm: 8.11.0 - ~/.nvm/versions/node/v18.17.0/bin/pnpm
Browsers:
Brave Browser: 116.1.57.64
Chrome: 119.0.6045.199
Chrome Canary: 121.0.6160.0
Edge: 119.0.2151.97
Firefox Nightly: 102.0a1
Safari: 16.5npm
"Compatibility" is misspelled "Compatibly" in the docs site sidebar nav.
https://shikiji.netlify.app/guide/compat
n/an/a
There may be other types missing somewhere else, you can check.
Add types
No response
No response
Visit https://shikiji.netlify.app/, select Glimmer TS
n/an/a
When using vitepress-plugin-twoslash, running vitepress build will throw the following error:
✓ building client + server bundles...
build error:
TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension ".css" for /home/projects/vite-astijb/node_modules/shikiji-twoslash/style-rich.css
While this error can be resolved by adding the config below in .vitepress/config.ts, I wonder if there is anyway to make vitepress-plugin-twoslash work without extra vite config. If no such way, maybe we could add some documentation to mention this vite config.
vite: {
ssr: {
// Uncomment the following line to fix the build
noExternal: ['shikiji-twoslash', 'vitepress-plugin-twoslash'],
},
}
https://stackblitz.com/edit/vite-astijb
N/Anpm
This project does not work with React-Markdown as it uses async call.
<ReactMarkdown
remarkPlugins={[
remarkGfm,
[remarkMath, { singleDollarTextMath: true }],
]}
rehypePlugins={[
rehypeKatex,
[
rehypeShikiji,
{
themes: {
light: "vitesse-light",
dark: "vitesse-dark",
},
},
],
]}
components={{
code(props) {
const { children, className, node, ...rest } = props;
const match = /language-(\w+)/.exec(className || "");
const lang = match && match[1];
return match ? (
<CodeBlock lang={lang || "text"} codeChildren={children} />
) : (
<code {...rest} className={className}>
{children}
</code>
);
},
}}
>
{message.content}
</ReactMarkdown>
Error: runSync finished async. Use run instead
Using it with Nextjs Client Side and React-Markdown.
System:
OS: macOS 14.0
CPU: (8) arm64 Apple M1 Pro
Memory: 145.05 MB / 16.00 GB
Shell: 5.9 - /bin/zsh
Binaries:
Node: 18.18.0 - /usr/local/bin/node
Yarn: 1.22.19 - /opt/homebrew/bin/yarn
npm: 8.19.2 - /opt/homebrew/bin/npm
pnpm: 7.2.1 - ~/Library/pnpm/pnpm
Browsers:
Chrome: 117.0.5938.149
Safari: 17.0npm
When developing AI applications with LLMs, outputs often include markdown-based content. Currently, we use the rehype toolchain to format these outputs. However, we encounter challenges with unpredictable language outputs, as well as structured output for better llm understanding output such as those demonstrated in the tldraw-llm-starter example 
. rehype-shikiji package currently does not have an option to handle those situation properly.
Adding a top-level option in rehype-shikiji to allow these languages to be displayed as plain text instead of undergoing faulty tokenization.
Maybe add ability/callback to handle error when language is not supported instead of throwing error
No response
I try to use shikiji in Next.js app directory in React Server Component and it gives me the error:
Attempted import error: 'getHighlighterCore' is not exported from 'shikiji/core' (imported as 'getHighlighterCore').
I tried multiple variants and it always fails on getHighlighterCore.
Import getHighlighterCore in Next.js React Server Component.
System:
OS: Windows 10 10.0.25967
CPU: (16) x64 AMD Ryzen 7 6800HS with Radeon Graphics
Memory: 17.64 GB / 31.26 GB
Binaries:
Node: 20.8.1 - ~\AppData\Local\Volta\tools\image\node\20.8.1\node.EXE
Yarn: 1.22.19 - ~\AppData\Local\Volta\tools\image\yarn\1.22.19\bin\yarn.CMD
npm: 9.6.5 - ~\AppData\Local\Volta\tools\image\npm\9.6.5\bin\npm.CMD
pnpm: 8.7.4 - ~\AppData\Local\Volta\tools\image\pnpm\8.7.4\bin\pnpm.CMD
Browsers:
Edge: Chromium (118.0.2088.46)
Internet Explorer: 11.0.25967.1000pnpm
hello, first let me say thank you for this interesting library. i am trying to migrate over from rehype-pretty-code but i just ran into an issue and i'm not sure what i'm doing wrong.
i am using nextjs 14, the @next-mdx package, and tailwind. in my mdx files i am writing some code blocks for example documentation. primarily javascript snippets. so the code blocks look like:
something
something else (notice the empty new line above)the highlighting works perfectly but somewhere along the tree transformation process these extra new lines are being inserted.
any ideas what i'm doing wrong? thanks!
see pictures in post
windows 11, chrome, node v20.9npm
Thanks for this library ❤️🔥
We're looking to replace shiki and shiki-twoslash with shikiji and shikiji-twoslash in svelte.dev, kit.svelte.dev and learn.svelte.dev sites. There, we use CSS variables heavily for syntax highlighting and theme switching.
--sk-code-ts-bg: hsl(var(--sk-back-2-hsl));
--sk-code-base: hsl(45, 7%, 75%);
--sk-code-comment: hsl(0, 0%, 55%);
--sk-code-keyword: hsl(204, 88%, 65%);
--sk-code-function: hsl(19, 67%, 75%);
--sk-code-string: hsl(41, 37%, 68%);
--sk-code-number: hsl(120, 100%, 25%);
--sk-code-template-string: hsl(2, 80%, 47%);
--sk-code-tags: var(--sk-code-function);
--sk-code-important: var(--sk-code-string);
--sk-code-diff-base: hsla(0, 0%, 100%, 0.5);
--sk-code-diff-inserted: hsl(120, 46%, 48%);
--sk-code-diff-removed: hsl(2, 64%, 70%);And the --shiki-color- variables are set accordingly:
body,
body.light,
body.dark {
--shiki-color-text: var(--sk-code-base);
--shiki-color-background: hsl(var(--sk-back-3-hsl));
--shiki-token-constant: var(--sk-code-base);
--shiki-token-string: var(--sk-code-string);
--shiki-token-comment: var(--sk-code-comment);
--shiki-token-keyword: var(--sk-code-keyword);
--shiki-token-parameter: var(--sk-code-base);
--shiki-token-function: var(--sk-code-function);
--shiki-token-string-expression: var(--sk-code-string);
--shiki-token-punctuation: var(--sk-code-base);
--shiki-token-link: var(--sk-code-keyword);
}It would be nice to have some way to have CSS variables work, as in the codebase, these same variables(--sk-code-) are used in the svelte REPL as well. A big refactor which turns it to dual theme approach will be quite hard to do at this scale.
Cant move to shikiji without this
(I did commented on an issue with same content, but that didn't get any response, hence opening a new one)
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.