

Starting with release 1.43, VS Code also allows extensions to provide tokenization through a Semantic Token Provider. Tokens are used for syntax highlighting, but also to classify the source code into areas of comments, strings, regex. The TextMate tokenization engine runs in the same process as the renderer and tokens are updated as the user types. VS Code extensions can contribute grammars through the grammar contribution point. TextMate grammars are a structured collection of regular expressions and are written as a plist (XML) or JSON files. VS Code's tokenization engine is powered by TextMate grammars. The tokenization of text is about breaking the text into segments and to classify each segment with a token type. To see both semantic and syntax token, use a built-in theme (for example, Dark+) on a TypeScript file. Theming: Using themes or user settings to map the tokens to specific colors and stylesīefore diving into the details, a good start is to play with the scope inspector tool and explore what tokens are present in a source file and what theme rules they match to.Tokenization: Breaking text into a list of tokens.

There are two components to syntax highlighting: It is responsible for colorizing keywords like if or for in JavaScript differently than strings and comments and variable names. Syntax highlighting determines the color and style of source code displayed in the Visual Studio Code editor.
