TextEncoder and TextDecoder in Deno | by Mayank Choubey | in Tech Tonic

Purpose

TextEncoder and TextDecoder are some of the core functionalities present in Deno. Deno also exposes them to users as part of the web API. Text encoding and decoding is a part of the WHATWG standard (https://encoding.spec.whatwg.org/#interface-textencoder).

These two classes are used extensively within Deno. They are useful in converting data from UTF-8 to bytes (encoding) and vice versa (decoding). The encoder and decoder are very useful whenever there is a need to deal with raw data (like buffers, streams, etc.) either to:

• convert raw data (or Uint8Array) to a string
• convert a string into raw data (or Uint8Array)

For example-

new Deno.Buffer(new TextEncoder().encode("123"));

TextEncoder and TextDecoder are very useful. Once you start using them, you would wonder how you worked without them.

Classes and Functions

Both TextEncoder and TextDecoder are available in the global space, so there is no need to import anything.

TextEncoder is a class, so an object needs to be created before using it.

new TextEncoder();

There are two useful functions in TextEncoder:

• encode: Convert a string into Uint8Array
• encodeInto: Convert a string into Uint8Array and place it in a given destination

new TextEncoder().encode("abcd") new TextEncoder().encodeInto("abcd", )

TextDecoder is also a class, so needs an object before use:

new TextDecoder();

There is one useful function in TextDecoder:

• decode: Convert a Uint8Array into a string

new TextDecoder().decode();

Inputs

TextEncoder doesn’t take any arguments at the time of creation.

TextDecoder has a couple of optional arguments at the time of creation:

• type: The encoding type of the data. Default is UTF-8, which is also the most common encoding type.
• options.fatal: Throw an error if decoding fails
• options.ignoreBOM: Ignore byte order mark (not common to use)

In most cases, UTF-8 would be enough.

TextEncoder’s encode functions take the following inputs:

• encode: The input is always a string
• encodeInto: The input is a string and a destination of type Uint8Array

encode(input?: string) encodeInto(source: string, destination: Uint8Array)

TextDecoder’s decode function takes a single Uint8Array as the mandatory input.

decode(input?: BufferSource, options?: TextDecodeOptions);

The only option present in the options is a stream kv which must always be false. If stream: true is passed, it’d result in an error. In other words, streams are not supported with TextDecoder.

Outputs

The output of TextEncoder’s encode function is a Uint8Array.

encode(input?: string): Uint8Array

The output of encodeInto function is an object that contains the read and written bytes (both are numbers). The output object would be useful in determining if encodeInto resulted in success.

encodeInto(source: string, destination: Uint8Array): TextEncoderEncodeIntoResult TextEncoderEncodeIntoResult {     read?: number;     written?: number; }

The output of TextDecoder’s decode function is always a string:

decode(input?: BufferSource, options?: TextDecodeOptions): string

Examples

Now that we’ve seen the useful functions, interfaces, inputs, and outputs, it’s time to go over some examples and see the usefulness of the encoding and decoding functions.

First, let’s start with simple encoding functions:

new TextEncoder().encode(); //Uint8Array(0) [] new TextEncoder().encode("A"); //Uint8Array(1) [ 65 ] new TextEncoder().encode("Hello Deno!"); //Uint8Array(11) [    72, 101, 108, 108,   111,  32,  68, 101,   110, 111,  33 ]

The output of encode function can be used in a variety of places like copying it to a buffer, stream, net, file, etc. Here is an example of creating a buffer out of the encoded output.

const buf=new Deno.Buffer(new TextEncoder().encode("Hello Deno!")); buf.length; //11

Next, let’s see an example of the encodeInto function. This function needs an array that could hold the output.

const bytes = new Uint8Array(11); new TextEncoder().encodeInto("Hello Deno!", bytes); bytes; //Uint8Array(11) [    72, 101, 108, 108,   111,  32,  68, 101,   110, 111,  33 ]

Now, let’s move on to TextDecoder and the decode function. As mentioned before, the decode function always takes Uint8Array as input and produces a string as the output.

new TextDecoder().decode(new TextEncoder().encode("Hello Deno!")); //Hello Deno!

Finally, let’s see how Deno uses these encoders and decoders in its own code. The first is writing text to a file. The low-level file writing ops only deals with bytes or Uint8Array. TextEncoder comes to the rescue whenever text needs to be written to a file.

function writeTextFileSync(     path,     data,     options = {},   ) {     const encoder = new TextEncoder();     return writeFileSync(path, encoder.encode(data), options); }

Similarly, let’s see the example of Deno’s file read operation. In this case, TextDecoder would be used to convert bytes into a string.

function readTextFileSync(path) {     const file = openSync(path);     const contents = readAllSync(file);     file.close();     const decoder = new TextDecoder();     return decoder.decode(contents); }