David Sherret

gagen - Writing complex GitHub Action workflow files

2026-04-02T20:30:00Z

GitHub action files can be a nightmare to maintain.

Conditions often need to be repeated across many steps.
Referencing values/ids by a string is fragile (ex. matrix values).
Maintaining pinned dependencies is difficult.
YAML is hard to work with.

What's an easier way to maintain these?

Initial Solution

In the Deno repo, our YAML file was complicated and the CI was slow. In 2023, we decided to generate the YAML with TypeScript.

Essentially it looked similar to the following:

const ci = {
  name: "ci",
  jobs: {
    build: {
      name: "...",
      steps: [{
        // ...etc...
      }],
    },
  },
};

const finalText = yaml.stringify(ci);
Deno.writeTextFileSync(
  new URL("./ci.generated.yml", import.meta.url),
  finalText,
);

This was a good first step because now applying a condition to multiple steps only required piping the step objects through functions:

function skipIfDraftPr(steps: Record<string, unknown>[]): unknown[] {
  const condition = "github.event.pull_request.draft == true";
  return [
    ...steps.map((step) => {
      step.if = "if" in step ? `${condition} && (${step.if})` : condition;
      return step;
    }),
  ];
}

Although the above was a good first step, a few years had passed and our CI was again too slow. This was mostly due to us having way more tests now. So, we decided to split up our single job with a matrix into build, and many test jobs to parallelize that work. We'd tried to do this in the past, but the upload and download artifact steps were slow enough that it made it not worth it. It's 2026 now and it's fast.

An issue though is that doing this would be too complicated to maintain. The solution I came up with was gagen.

`gagen`

gagen allows you to define steps and then describe the relationships between steps along with the conditions that a step should occur.

import { conditions, step, workflow } from "gagen";

const checkout = step({
  uses: "actions/checkout@v6",
});

const test = step.dependsOn(checkout)({
  name: "Test",
  run: "cargo test",
});

const installDeno = step({
  uses: "denoland/setup-deno@v2",
});

const lint = step
  .dependsOn(checkout)
  // this condition gets propagated to installDeno, but not checkout
  .if(conditions.isBranch("main").not())(
    {
      name: "Clippy",
      run: "cargo clippy",
    },
    step.dependsOn(installDeno)({
      name: "Deno Lint",
      run: "deno lint",
    }),
  );

// only specify the leaf steps — the other steps
// are pulled in automatically
workflow({
  name: "ci",
  on: ["push", "pull_request"],
  jobs: [{
    id: "build",
    runsOn: "ubuntu-latest",
    steps: [lint, test],
  }],
}).writeOrLint({
  filePath: new URL("./ci.generated.yml", import.meta.url),
  header: "# GENERATED BY ./ci.ts -- DO NOT DIRECTLY EDIT",
});

This outputs the following workflow file:

# GENERATED BY ./ci.ts -- DO NOT DIRECTLY EDIT

name: ci
on:
  - push
  - pull_request
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd
      - name: Test
        run: cargo test
      - name: Clippy
        if: github.ref != 'refs/heads/main'
        run: cargo clippy
      - uses: denoland/setup-deno@667a34cdef165d8d2b2e98dde39547c9daac7282
        if: github.ref != 'refs/heads/main'
      - name: Deno Lint
        if: github.ref != 'refs/heads/main'
        run: deno lint

# gagen:pin actions/checkout@v6 = de0fac2e4500dabe0009e67214ff5f5447ce83dd
# gagen:pin denoland/setup-deno@v2 = 667a34cdef165d8d2b2e98dde39547c9daac7282

Notice:

Dependencies like actions/checkout@v6 get locked to the hash.
- On subsequent runs, gagen uses the output file as the lockfile.
The condition to not run on main is specified only once. It's then automatically propagated backward to the necessary steps.
The denoland/setup-deno step runs at the latest time that it can. This means if the cargo clippy step fails, no time is wasted running denoland/setup-deno unnecessarily (so faster feedback).

Under the hood, how gagen works is it creates a graph between steps and then when creating each workflow it evaluates the graph and conditions. This means you can reuse step objects between workflows and jobs too.

Typed values

We've resolved most of the above, but now we're still left with the problem that referencing values/ids by a string is fragile.

- 1. Conditions often need to be repeated across many steps.
  2. Referencing values/ids by a string is fragile (ex. matrix values).
- 3. Maintaining pinned dependencies is difficult.
- 4. YAML is hard to work with.

gagen provides some helpers for doing that. For example, matrices are typed:

import { defineMatrix, workflow } from "gagen";

const matrix = defineMatrix({
  include: [
    { runner: "ubuntu-latest" },
    { runner: "macos-latest" },
  ],
});

matrix.runner; // ExpressionValue("matrix.runner") — autocompletes
matrix.foo; // TypeScript error — not a matrix key

workflow({
  // ...
  jobs: [
    {
      id: "build",
      runsOn: matrix.runner,
      strategy: { matrix },
      steps: [test],
    },
  ],
}).writeOrLint({
  filePath: new URL("./ci.generated.yml", import.meta.url),
});

This allows for getting auto-complete on the matrix values when writing something like matrix.os.equals("linux"), which can then be used in a step.

Also, there's a helper for artifacts:

import { artifact, step, workflow } from "jsr:@david/gagen@<version>";

const buildArtifact = artifact("build-output");

workflow({
  name: "CI",
  on: ["push", "pull_request"],
  jobs: [
    {
      id: "build",
      runsOn: "ubuntu-latest",
      steps: [
        step({ name: "Build", run: "make build" }),
        buildArtifact.upload({ path: "dist/" }),
      ],
    },
    // `needs: [build]` is inferred automatically from the artifact link
    {
      id: "deploy",
      runsOn: "ubuntu-latest",
      steps: [
        buildArtifact.download({ dirPath: "output/" }),
        step({
          name: "Deploy",
          run: "make deploy",
        }),
      ],
    },
  ],
}).writeOrLint({
  filePath: new URL("./ci.generated.yml", import.meta.url),
});

How to keep `ci.generated.yml` up-to-date?

An obvious problem with this solution is that we need to ensure the YAML file is up to date with the code generation file.

To achieve this, the writeOrLint function will ensure the output is up to date when the script being executed is passed a --lint CLI flag, so we can add that as a CI step:

// note: this requires ci.ts to have a shebang in it that
// runs the typescript code using your preferred runtime
const lintStep = step({
  name: "Lint CI generation",
  run: "./.github/workflows/ci.ts --lint",
});

Impact?

By taking advantage of all this, in February I was able to increase the complexity of the generated output and simplify the maintained code generation script.

Now it has:

A build job for each platform uploading the executable artifacts.
Many test jobs downloading the executable artifacts and running tests in parallel.

Note: The blue dips on main are release workflow runs, which do less work. Also, sorry the chart is not great, but I created this a couple months ago and now the raw data seems gone.

The main slowness now is compiling Deno on certain platforms (like Mac x86).

Code
Output

Sure, this could have been done in regular YAML, but I believe the code is way more maintainable. Yes, it's still complicated, but maintainable.

For more on what gagen can do, read the docs on GitHub: https://github.com/dsherret/gagen

First-class JSONC manipulation in JavaScript

2025-10-12T23:20:00Z

Previously I wrote about first-class JSONC manipulation in Rust. This weekend I wrapped this project to make it available in JavaScript.

Current approach (not great)

From my understanding, the current way most people modify JSONC files in JavaScript is via the jsonc-parser npm package.

To use this, you create a list of edits, then you apply the edits. For example:

import { assertEquals } from "@std/assert";
import { applyEdits, modify } from "jsonc-parser";

const jsonText = `{
  "value": 1
}`;
const edits = modify(
  jsonText,
  ["value"], // JSON path to modify
  2, // new value
  {
    formattingOptions: {
      insertSpaces: true,
      tabSize: 2,
    },
  },
);
const finalText = applyEdits(jsonText, edits);
assertEquals(
  finalText,
  `{
  "value": 2
}`,
);

This works, but quickly becomes complex for simple scenarios. Say we have a configuration file that could possibly look like this...

{
  "plugins": [
    "https://plugins.dprint.dev/json-0.17.0.wasm"
  ]
}

...and we want to programmatically append a new url to the plugins array. We need to handle the plugins property not existing, it existing with a non-array value, it being empty, or it having other elements.

Ask an AI to help you with this with npm:jsonc-parser and you'll see the code is quite complex.

Goal

Similar to the last blog post, the API I idealized was one where the code looks similar to this list where each step reads like a high-level description of intent:

Parse the text.
Get and ensure the root value is an object.
Get and ensure that object has a plugins array value property.
Append the url to the plugins array.
Get the final text.

Solution

I published jsonc-morph this weekend that achieves this API. You can install it via deno add jsr:@david/jsonc-morph or npm install jsonc-morph

Here's an example:

import { parse } from "@david/jsonc-morph";
import { assertEquals } from "@std/assert";

const jsonText = `{
  "plugins": [
    "https://plugins.dprint.dev/json-0.17.0.wasm" // json plugin
  ]
}`;
// 1. Parse the text.
const root = parse(jsonText);
// 2. Get and ensure the root value is an object.
const rootObj = root.asObjectOrForce();
// 3. Get and ensure that object has a plugins array value property.
const plugins = rootObj.getIfArrayOrForce("plugins");
// 4. Append the url to the plugins array.
plugins.append("https://plugins.dprint.dev/typescript-0.95.11.wasm");

// 5. Get the final text.
assertEquals(
  root.toString(),
  `{
  "plugins": [
    "https://plugins.dprint.dev/json-0.17.0.wasm", // json plugin
    "https://plugins.dprint.dev/typescript-0.95.11.wasm"
  ]
}`,
);

The complexity is abstracted away, and low level concerns are automatically handled.

Comments in the file are maintained and not shifted around when making changes.
Proper indentation and newlines are handled for us.
If the data currently uses trailing commas, that will be respected.
- Trailing commas can be forced by calling root.setTrailingCommas(true);

You might have noticed this API is similar to my project ts-morph, which is for modifying TypeScript/JavaScript files.

Implementation

So how does this work under the hood?

This implementation uses a concrete syntax tree (CST) which is like an abstract syntax tree (AST), but also stores the whitespace, tokens, and comments in the tree. This allows for easily manipulating the tree in place taking into account everything found in the file, then printing it out when done.

I already did all the hard work in Rust though, so for this JS library I used Claude to generate wrapper code with wasm-bindgen then built it with wasmbuild to get a Wasm module that makes it available to JS.

Maintainable string and bytes pre-allocation in Rust

2024-12-30T00:00:00Z

A common performance optimization in software development is to pre-allocate strings/bytes before appending to them. In Rust, failing to do this may cause the implementation of std::string::String or std::vec::Vec to frequently reallocate bytes internally to deal with its growing size, which is slow. Additionally, the amount of bytes we end up with at the end may be way more than we actually need.

Why we pre-allocate

Take the following code:

fn main() {
  let mut text = String::new();
  let spaces = " ".repeat(100); // a 100 byte string
  println!("Len: {}, Capacity: {}", text.len(), text.capacity());
  for _ in 0..9 {
    text.push_str(&spaces);
    println!("Len: {}, Capacity: {}", text.len(), text.capacity());
  }
}

When we run this, we can see the string being reallocated often as the length increases and in the end we're left with an over-allocated string of 1600 bytes instead of 900:

Len: 0, Capacity: 0
Len: 100, Capacity: 100
Len: 200, Capacity: 200
Len: 300, Capacity: 400
Len: 400, Capacity: 400
Len: 500, Capacity: 800
Len: 600, Capacity: 800
Len: 700, Capacity: 800
Len: 800, Capacity: 800
Len: 900, Capacity: 1600

However, when the capacity is correctly pre-allocated:

fn main() {
  let mut text = String::with_capacity(900);
  let spaces = " ".repeat(100); // a 100 byte string
  println!("Len: {}, Capacity: {}", text.len(), text.capacity());
  for _ in 0..9 {
    text.push_str(&spaces);
    println!("Len: {}, Capacity: {}", text.len(), text.capacity());
  }
}

We only allocate text once and end up with a string equal to its capacity:

Len: 0, Capacity: 900
Len: 100, Capacity: 900
Len: 200, Capacity: 900
Len: 300, Capacity: 900
Len: 400, Capacity: 900
Len: 500, Capacity: 900
Len: 600, Capacity: 900
Len: 700, Capacity: 900
Len: 800, Capacity: 900
Len: 900, Capacity: 900

Problem

Seen this kind of code before?

let capacity = items
  .iter()
  .filter_map(|i| i.maybe_name.as_ref())
  .enumerate()
  .map(|(i, name)| if i > 0 { 2 } else { 0 } + name.len())
  .sum::<usize>();
let mut text = String::new();
text.try_reserve_exact(capacity)?;

for (i, name) in items
  .iter()
  .filter_map(|i| i.maybe_name.as_ref())
  .enumerate()
{
  if i > 0 {
    text.push_str(", ");
  }
  text.push_str(name);
}
debug_assert_eq!(text.len(), capacity);

The above code:

Calculates the byte capacity of the string ahead of time.
Allocates bytes with that capacity, returning an error when it doesn't have enough memory to allocate.
Builds up the final string without causing additional allocations of text.
Finally, it does a debug assertion to ensure the final text length matches the capacity we calculated ahead of time.

Although this code is performant, it has a couple of problems:

It's complicated.
No single source of truth.
- The capacity calculation code could get out of sync with the code that builds the string... the debug assertion helps, but not if all scenarios aren't tested in debug.

Solution

A solution to this problem is to make the code to calculate the capacity the same as the code to build up the string. I've rolled this up into a crate that makes it easy: capacity_builder

// same functionality as the above code, but simpler
use capacity_builder::StringBuilder;

let text = StringBuilder::<String>::build(|builder| {
  for (i, name) in items
    .iter()
    .filter_map(|i| i.maybe_name.as_ref())
    .enumerate()
  {
    if i > 0 {
      builder.append(", ");
    }
    builder.append(name);
  }
})?;

This runs the closure twice: once to compute the capacity and a second time to build the string. The final string will have a length equal to its capacity and will never reallocate itself while it's being built.

Some features:

Prevents allocations in the closure by only accepting values by reference (possible thanks to Rust's amazing borrow checker)
Numbers can be appended (or anything that implements capacity_builder::StringAppendable)
Can be made to work with any string type and not just std::string::String
Building up bytes is possible via capacity_builder::BytesBuilder

I've been integrating this into Deno's codebase and it's enabled us to start pre-allocating strings/vectors in complex cases that previously weren't maintainable.

If you have any suggestions or run into any issues, please open an issue on the project's GitHub: https://github.com/dsherret/capacity_builder

First-class JSONC manipulation in Rust

2024-10-20T00:15:00Z

In Deno and dprint (two Rust projects I maintain), there are certain cases where a JSON with comments (JSONC) configuration file needs to be programmatically updated.

For example, running the following in Deno...

> deno add jsr:@david/dax
Add jsr:@david/dax@0.42.0

...adds the @david/dax JSR package as a dependency to the configuration file.

Current approach (not good)

Our current approach involves parsing a JSONC file with jsonc-parser to an AST, then using that to build up a collection of "text changes" and finally applying the text changes to the original text.

For example, say I have the following dprint.jsonc file and we want to add a new url to the plugins array:

{
  "plugins": [
    "https://plugins.dprint.dev/json-0.19.1.wasm"
  ]
}

To do that, we'd examine this code, then construct a collection of text changes like the following and have some other code manipulate the original string to apply theses changes.

[{
  "range": [66, 66],
  "newText": ",\n    \"https://plugins.dprint.dev/toml-0.6.3.wasm\""
}]

Example non-Rust pseudocode

/// Adds a plugin url to the dprint config file's plugins array.
/// ```jsonc
/// {
///   "plugins": [
///     "https://plugins.dprint.dev/toml-0.6.3.wasm",
///     "<new url goes here>"
///   ]
/// }
/// ```
function addPluginToJson(jsonText, url) {
  const changes = [];
  // parse to an ast
  const ast = parseJson(jsonText);

  // if the root is not an object, just replace it with one
  if (ast.value?.kind !== "object") {
    return `{
  "plugins": [
    "${url}"
  ]
}
`;
  }

  // find the plugins property
  const pluginsProp = ast.value.properties
    .find(p => p.name === "plugins");
  if (pluginsProp?.value?.kind !== "array") {
    // doesn't exist, so add it to the root object
    const lastProperty = ast.value.properties.at(-1);
    const insertIndex = lastProperty?.end ?? ast.value.start + 1;
    const maybeComma = lastProperty == null ? "," : "";
    changes.push({
      range: [insertIndex, insertIndex],
      text: `${maybeComma}\n  "plugins": [\n    "${url}"  ]`,
    });
  } else {
    // add the url to the existing plugins array
    const lastPlugin = pluginsProp.value.at(-1);
    const insertIndex = lastPlugin?.end ?? pluginsProp.value.start + 1;
    const maybeComma = lastPlugin == null ? "," : "";

    changes.push({
      range: [insertIndex, insertIndex],
      text: `${maybeComma}\n    "${url}"`,
    });
  }

  // apply the text changes to the json text
  return applyTextChanges(jsonText, changes);
}

This is very complex. To do the high level task of adding an array element, we need to do a lot of low level work. A proper implementation of this would need to deal with indentation, understand what newline kind the file uses, handle comments, and understand if the file uses trailing commas.

We could address these concerns in the code, but doing so would significantly increase its complexity and hurt maintainability. It would mean similar complex solutions throughout the codebase making new features, changes, and bug fixes time consuming.

Discarded Solution: Better text change API

Some solutions in the wild look like this:

const editResult = modify(jsonText, ["plugins"], newPluginUrl, {
  isArrayInsertion: true,
});
const newText = applyEdits(jsonText, editResult);

While this solution works for many cases, I don't believe it provides the flexibility I want for more complex JSONC modifications, such as manipulating comments. I also wanted a solution where subsets of the JSONC data can be focused on and manipulated in place.

Goal

The API I idealized was one where the code looks similar to this list where everything is described at a high level:

Parse the text.
Get and ensure the root value is an object.
Get and ensure that object has a plugins array value property.
Append the url to the plugins array.
Get the final text.

Solution

The newly released 0.26 version of jsonc-parser now includes a "cst" feature that can be enabled in your Cargo.toml file:

jsonc-parser = { version = "0.26", features = ["cst"] }

This exposes the jsonc_parser::cst module.

Now, let's rewrite the above example code using this new API:

use jsonc_parser::cst::CstRootNode;
use jsonc_parser::cst::CstInputValue;
use jsonc_parser::errors::ParseError;
use jsonc_parser::json;

/// Add a plugin url to the dprint config file's plugins array.
///
/// ```jsonc
/// {
///   "plugins": [
///     "https://plugins.dprint.dev/toml-0.6.3.wasm",
///     "<new url goes here>"
///   ]
/// }
/// ```
pub fn add_to_plugins_array(
  file_text: &str,
  url: &str,
) -> Result<String, ParseError> {
  let root_node = CstRootNode::parse(file_text, &Default::default())?;
  let root_obj = root_node.object_value_or_set();
  let plugins = root_obj.array_value_or_set("plugins");

  plugins.ensure_multiline();
  plugins.append(json!(url));

  Ok(root_node.to_string())
}

The complexity is abstracted away, and low level concerns are automatically handled.

Comments in the file are maintained and not shifted around when making changes.
Proper indentation and newlines are handled for us.
If the data currently uses trailing commas, that will be respected.
- Trailing commas can be forced by calling root_obj.set_trailing_commas(...)

There's a lot more you can do with this. I'd recommend reading the documentation to see what's possible and please consider contributing if you see any other improvements. Also, please open issues for any bugs or scenarios you think it could be smarter about.

Implementation

For parsing, I didn't want to implement a new parser for the CST, so I just reused the existing AST parser in jsonc-parser, then converted that to a CST. The parser already had an option for collecting tokens & comments, and if you have the AST, tokens, comments, & original text, you can easily construct a CST.

On the internal structure of the CST, I didn't want to include any dependencies to help with this (by default, jsonc-parser has zero dependencies), so I rolled with my own solution. Internally, each node in the tree contains an Rc<RefCell<T>> where T is its data and parent. The parent is referenced via a weak reference so that the memory used gets cleaned up when you're done (this means you must not drop the root node or a panic may occur to prevent bugs when doing certain operations ). I'm unsure if this is the best solution here, but it seems to work fine and generally the root node is kept around to get the final text anyway.

file_test_runner

2024-05-12T23:15:00Z

cargo test in Rust is an excellent tool, but sometimes writing Rust code isn't the best way to maintain certain types of tests.

Case: Deno

Take Deno's codebase. The pattern for writing an integration test with the Deno binary has required writing code like the following in Rust:

// ~/tests/integration/run_tests.rs
itest!(002_hello {
  // these are relative from ~/tests/testdata
  args: "run --quiet --reload run/002_hello.ts",
  output: "run/002_hello.ts.out",
});

This macro expands to a #[test] function, which launches the Deno binary with the provided arguments and asserts its output against the provided file.

Problems:

Requires recompiling the test binary when adding/changing/deleting tests.
Test definition is in a different folder than the files being run or asserted—lots of tests in a single file using lots of data files in another folder.
- Changing folders felt like context switching because of how far away they were.
- It was hard to associate what testdata files were for what test.
  - These files were often not deleted when the test definition was deleted.
- It didn't encourage developers to write a lot of tests.

Ideally a single test or group of related tests should be co-located in the same folder as the testdata files and not be defined in Rust.

Case: dprint's formatters

In dprint's formatter codebases, I've long stored tests in text files. For example, here's an example in dprint-plugin-json (JSON/JSONC formatter) at tests/specs/strings/Strings_All.txt:

~~ lineWidth: 80 ~~
== should support single quote strings ==
'te\'st'

[expect]
"te'st"

== should support double quote strings ==
"test\"test"

[expect]
"test\"test"

As you can see, it's groups of related tests stored in the same file.

Problems:

Had custom filtering.
Had its own custom infrastructure for running these tests.

Benefits:

Didn't require recompiling after updates.
Test file was tailored to the situation being tested.

Goals for writing a new test runner

I wanted a test runner that:

Allows storing the test definition close to the files used in the tests and the expected output.
Doesn't require recompiling Rust code when adding, changing, or deleting tests.
Is non-opinionated to allow structuring the tests according to the needs of the project.
Runs the tests in parallel.
Allows filtering via cargo test <test_name>.

Solution: file_test_runner

The solution I've settled on is file_test_runner.

This does two main steps:

Collects tests in any format on the file system.
Runs each test using custom provided code.

The basic setup is as follows:

Add a [[test]] section to the project's Cargo.toml with the default test harness disabled:
```
[[test]]
name = "specs"
path = "tests/spec_test.rs"
harness = false
```

Create a tests/spec_test.rs file to run the code:

use file_test_runner::collect_and_run_tests;
use file_test_runner::collection::CollectedTest;
use file_test_runner::collection::CollectOptions;
use file_test_runner::RunOptions;
use file_test_runner::TestResult;

fn main() {
  collect_and_run_tests(
    CollectOptions {
      base: "tests/specs".into(),
      strategy: Box::new(..omitted..),
      filter_override: None,
    },
    RunOptions {
      parallel: true,
    },
    // custom function to run the test...
    |test| {
      // do something like this, or do some checks yourself and
      // return a value like TestResult::Passed
      TestResult::from_maybe_panic(AssertUnwindSafe(|| {
        // run the test here
      }))
    }
  )
}

Add test files or directories in any format to the tests/specs/ folder as specified above and update the code above to handle it.

Collecting tests

Tests can be collected from the file system using several strategies (note the strategy property under CollectOptions above).

For example, by a file in a directory:

// goes recursively through each directory under the base
// ("tests/specs") and finds the directories with a
// `__test__.jsonc` file
strategy: Box::new(TestPerDirectoryCollectionStrategy {
  file_name: "__test__.jsonc".into(),
})

Or by all descendant files:

// goes recursively through each directory under the base
// ("test/specs") excluding readme.md files and collects
// a test per file
strategy: Box::new(TestPerFileCollectionStrategy {
  file_pattern: None,
})

If you need more flexibility than that, you can implement your own file_test_runner::collection::strategies::TestCollectionStrategy:

pub trait TestCollectionStrategy<TData = ()> {
  fn collect_tests(
    &self,
    base: &Path
  ) -> Result<CollectedTestCategory<TData>, CollectTestsError>;
}

This is extremely flexible and even allows collecting multiple tests within the same file (the file_test_runner::collection::strategies::FileTestMapperStrategy is helpful for that).

Running tests

After tests are collected, file_test_runner will go through each category of tests running them in parallel on different threads, providing each test to the closure to run a test:

// custom function to run the test...
|test| {
  // do something like this, or do some checks yourself and
  // return a value like TestResult::Passed
  TestResult::from_maybe_panic(AssertUnwindSafe(|| {
    // Properties:
    // * `test.name` - Fully resolved name of the test
    // * `test.path` - Path to the test file this test is associated with
    // * `test.data` - Data associated with the test that may have been
    //                 set by the collection strategy
  }))
}

Benefits

file_test_runner handles collecting tests, orchestrating tests, and reporting test results to the console.
Tests can be added/modified/deleted without recompiling the Rust test binary—it's very fast to re-run a test.
Test reporter output looks very similar to cargo test's default, but also shows how long each test takes to run.
Tests can be filtered using cargo test <test_name>.
Tests can be structured in whatever makes most sense for what's being tested since it's non-opinionated. Stuff like snapshot testing can be implemented within the run test function.
Tests are run in parallel.

You can see the documentation for the implementation we ended up using in Deno here (we're still in the process of migrating all the itests to it).

Future

Right now there's not any support for cargo test -- --nocapture. I'm not entirely sure how to handle it (especially when a test uses multiple threads), but at the moment it doesn't capture any output within a test and a test implementation needs to handle that itself.

Overall I'm quite satisfied with this crate and it's made adding these kind of tests in several of Deno's repos much easier.

dax - Cross-platform shell tools for Node.js

2024-02-09T15:00:00Z

In July 2022, I released dax for Deno providing a cross-platform shell for JavaScript written in JavaScript:

const data = $.path("data.json").readJsonSync();
await $`git add . && git commit -m "Release ${data.version}"`;

This is similar and inspired by zx, but because it uses a cross-platform shell with common built-in cross-platform commands, more code is going to work the same way on different operating systems.

Initially, I wrote dax for Deno because Deno is by far the best JavaScript runtime for single file scripting—all dependencies can be expressed in the script file itself including npm dependencies; there's no node_modules folder (less clutter), and no separate install command necessary.

Once written, dax used APIs that only worked on Deno and creating a Node.js distribution was a decent amount of work.

Nowadays, Node.js has improved in its support for Web APIs and improvements to dnt (a tool I created for building Deno modules for Node) have made maintaining a Node.js distribution much easier.

Due to this, I'm happy to say that dax is now available on npm for users of Node.js:

// example.mjs
import $ from "dax-sh";

await $`echo 'Hello from dax!'`;

$ npm install --save-dev dax-sh
$ node example.mjs
Hello from dax!
$ time node example.mjs
Hello from dax!
node example.mjs 0.08s user 0.01 system 98% cpu 0.090 total

You can check out dax's documentation here for more details:

https://github.com/dsherret/dax

A long aside: build dax into Deno?

Part of what kicked off my desire to create a Node.js distribution for dax was the release of Bun's shell, which credits dax as a source of inspiration.

This led to requests for dax to be baked into Deno's runtime.

In my opinion, this would be a step backwards for dax and not a good long term decision for Deno.

I want to explain why I think this and it would be interesting to hear your feedback. Note these are my personal opinions and not the opinions of the Deno team (which I'm a member, but dax is a personal project I work on in my personal time).

Runtime coupling

Coupling a complex API like dax to the runtime means you can no longer upgrade them independently. Being able to depend on a specific version of dax and a specific version of your runtime is a massive benefit. It means you can freely upgrade your runtime version and the code using dax will mostly likely keep working too—the chance of encountering a new dax bug while upgrading your runtime is very low because they're decoupled.

Additionally, it also means when you upgrade your runtime, you don't need to also upgrade all your dax code at the same time in case there's a breaking change.

It also means you likely don't need to tell people to use a certain version of Deno in order to get the latest dax features ("hey, why doesn't this work? Oh, that dax feature is only in Deno version x.x.x"). Instead, the code specifies the dax version it depends on so when you execute it, it likely works or dax can provide specific error messages for the runtime when not.

Vendor Lock-in

Being able to use the same API in different runtimes is a massive benefit. It lowers vendor lock-in risk and lowers the complexity when working with multiple runtimes because the APIs you're using are the same. It also means when the next great runtime comes around you're not locked in with all this code depending on a specific runtime (or a specific version of a specific runtime 😱).

When dax is published as a library, you can switch runtimes and still depend on the same version of dax.

Scope

Dax is not only a shell, but a collection shell tools. It's a swiss army knife that provides opinionated ways of doing common tasks you need to do in automation scripts. It has APIs for...

progress and selection,
making URL requests,
logging,
dealing with paths,
and in the future, CLI argument parsing and work caching.

All these APIs work together with each other and the shell. They're opinionated for simplicity. Baking opinionated APIs into a runtime wouldn't be a good idea because people have different opinions and opinions change over time. In the case of dax being a library, someone else can come along and improve on its API or make something better in the future, at which point dax can become a relic just like old JS frameworks.

One suggestion is to cut the scope of dax back to a shell only rather than a collection of shell tools, but the shell is still quite large. For example, you can build your own custom $ to suite your needs and inject your own custom shell commands written in JavaScript.

Cutting it back further to not include that and some other features is possible, but the shell itself is still quite intricate and there's lots of tiny design decisions that are better left to a library like dax to get wrong and then be improved upon by a future library or future major version of dax. Also at a certain point scope gets cut back enough that it starts becoming less useful.

Built-in runtime APIs should be permanent

I'm still slowly figuring out an appropriate API for dax. I don't believe anything is going to change drastically, but making a mistake if it were a built-in runtime API would be fatal. Built-in APIs and the decisions made should ideally be permanent. When they're not permanent or get removed, that creates a lot of headaches.

When it's in a library, it's behind a separately versioned API, so the chance of your code not working with the runtime anymore is slim, and making breaking changes in library that's behind a versioned API is much more manageable.

Imagine if a similar API to dax had been integrated into the runtime that made the mistake of spawning the system shell because we hadn't thought to make it cross platform yet? Image what other possibilities for this API we'll discover in the future and be glad we can easily make the changes to improve it because it exists as a library.

Performance?

Part of the argument to integrate this API into the runtime is for performance, but dax starts up in 90ms on my machine in Node.js and 70ms in Deno. It executes commands almost as fast as using Deno's Command API (2ms slower on my machine). Could it be faster? Probably... I haven't done any extensive benchmarking on dax because I develop it in my free time around all the other projects I do.

It's fast enough for my needs. You'd definitely be able to show it being slower than some native code in a hot loop, but generally automation scripts only execute a handful of commands (maybe ~10 commands) and spend most of their time waiting for long complex tasks to finish (for me, stuff like cargo build), so gaining some milliseconds by it being built-in and native doesn't help much in most real world scripts.

Plus being less productive writing automation scripts with a less featureful API will use up far more of your time than the few milliseconds saved with it being built-in, which won't even be meaningfully saved in most real world scenarios.

If we're optimizing for performance only, dax actually doesn't need to be built-in and could go native using Deno's FFI support, but in my opinion creating less portable less auditable code written in a language not as many people understand to have a slightly better performance experience is a bad trade.

Convenience of no dependency?

I wouldn't categorize having no dependency as a convenience because the runtime coupling I talked about in a previous section leads to inconvenience. Maybe it's slightly annoying in Node.js because it requires adding `dax-sh`` to a package.json and installing it, but in Deno you can just write:

#!/usr/bin/env -S deno run -A
import $ from "https://deno.land/x/dax/0.39.0/mod.ts";

await $`echo Hello`;

Is writing that difficult? I don't believe so, and now my script has all the information to know what version of dax to use or I can swap it out for a similar dependency that has the API I like instead.

It's great in Deno because I don't even need to run a separate install script—I just run that script directly and it will use the version I specified. Of course, I could use a bare specifier like "dax" by creating a deno.json with an embedded import map to make import $ from "dax"; work:

{
  "imports": {
    "dax": "https://deno.land/x/dax/0.39.0/mod.ts"
  }
}

`jsr:@deno/shell@1`?

Overall, I get the desire for having dax built-in, but I don't believe it's the right long term decision. Perhaps if there's a desire for a shell only and not a swiss army knife of automation scripts, then the core functionality in dax could be extracted out to a simpler package on the upcoming JSR registry behind its own versioned API.

import $ from "jsr:@deno/shell@1";

await $`echo 'Hello there!'`;

Let me know if there's a desire for a less functional, more lightweight version of dax like that and I'll look into making it happen.

Again, you can now install dax via npm install --save-dev dax-sh and use it in Node.js. Read the documentation here: https://github.com/dsherret/dax

Lost scrobbles and JavaScript Jupyter Notebooks

2024-01-16T21:00:00Z

I've been a Last.fm user since 2008 and a pro user for the past few years to support the service.

For those not familiar, Last.fm allows someone to track the music they've listened to from mostly any music client. These listens are called "scrobbles".

For example, Spotify has integration through a connected app and every time I listen to a song on Spotify it sends that song to Last.fm.

Spotify Wrapped

Every December, Spotify releases Spotify Wrapped, which I'm sure you're familiar with.

This year, I noticed some discrepancy between the results from Spotify and Last.fm.

Spotify - Top artists and songs

Last.fm - Top artists

Last.fm - Top songs

There are some clear discrepancies between these two sources and I use Spotify to listen to music ~99% of the time. Most notably, my top artist for Spotify is Alpha 9, yet Last.fm says it's i_o. Neither appear in the other.

My initial thought was that perhaps Spotify has a different reporting period than Last.fm. This is definitely the case with my top artist on Last.fm (i_o) as I listened to him a lot early December 2022 and I was looking at these numbers around December 1st 2023. That said, it doesn't explain everything. One issue I've had is Last.fm scrobbles just not happening and I then need to reconnect it to get it working again. I was kind of curious just how many scrobbles were being lost.

Luckily it's possible to get all the raw data from Spotify and Last.fm so that I can try to better understand what's happening.

Getting the data

Spotify

Spotify provides this data for download under Profile > Account > Privacy settings.

https://www.spotify.com/ca-en/account/privacy/

Last.fm

Last.fm is a little trickier, but luckily there is a third party website that helps download your entire listening history via Last.fm's API:

https://benjaminbenben.com/lastfm-to-csv/

I found this did the job, but it would occasionally error on some API calls. A quick fix was to apply this patch to the repo to retry on failure in order to make it more reliable.

JavaScript Jupyter Notebooks

Now that I had the data, I needed a way to analyze it. My data analysis background is very poor and usually in this case I would just write a quick script in whatever language is easiest for the task.

In this case, Deno recently released Jupyter Notebook support and I hadn't really tried it out yet (my colleague Bartek did most of the work implementing it) nor had I ever used a Jupyter Notebook. This seemed like a good occasion.

I setup Jupyter for Deno in VS Code, created a notebook.ipynb file, a deno.json file with {} in it (to activate Deno's language server and automatically get a Deno lockfile), then added the Last.fm & Spotify data to the same folder.

Loading and normalizing the data

Last.fm

The Last.fm data is a csv file. I created a notebook cell that loaded and normalized the data like so:

import $, { PathRef } from "https://deno.land/x/dax@0.36.0/mod.ts";
import { parse as parseCsv } from "npm:csv-string@4.1.1";

interface NormalizedRow {
  artist: string;
  album: string;
  track: string;
  date: Date;
}

const lastFmText = $.path("lastfm.csv").readTextSync();
const lastFmData: NormalizedRow[] = parseCsv(lastFmText).map((
  row: string[],
) => ({
  artist: row[0].trim(),
  album: row[1].trim(),
  track: row[2].trim(),
  date: new Date(row[3] + " GMT"),
})).reverse();

console.log("Loaded", lastFmData.length, "rows");

Spotify

Spotify stores the streaming data in multiple JSON files.

Streaming_History_Audio_2012-2014_0.json
Streaming_History_Audio_2014-2015_1.json
...
Streaming_History_Audio_2022-2023_10.json
Streaming_History_Audio_2023_11.json

I loaded it like so:

interface SpotifyRow {
  ts: string;
  ms_played: number;
  master_metadata_track_name: string;
  master_metadata_album_artist_name: string;
  master_metadata_album_album_name: string;
  reason_start: string;
  reason_end: string;
}

function normalizeSpotify(row: SpotifyRow): NormalizedRow {
  return {
    artist: row.master_metadata_album_artist_name.trim(),
    album: row.master_metadata_album_album_name.trim(),
    track: row.master_metadata_track_name.trim(),
    date: new Date(Date.parse(row.ts) - row.ms_played),
  };
}

function loadFromFile(path: PathRef) {
  return path
    .readJsonSync<SpotifyRow[]>()
    .filter((row) =>
      row.master_metadata_album_artist_name != null
      && row.master_metadata_album_album_name != null
      && row.master_metadata_track_name != null
      // not perfect, but probably a close enough approximation
      // to what counts as a scrobble
      && (row.reason_end === "trackdone" || row.ms_played > 120_000)
    )
    .map(normalizeSpotify);
}

const spotifyData = Array.from($.path(".").readDirSync())
  .filter(entry =>
    entry.isFile && entry.name.startsWith("Streaming_History_Audio")
  )
  .map(entry => entry.path)
  .sort((a, b) => a.toString().localeCompare(b.toString()))
  .map(path => loadFromFile(path))
  .flat();

console.log("Loaded", spotifyData.length, "rows");

What's hard to determine here is what Spotify play is worthy of being counted as a Last.fm scrobble. Spotify stores all listens—even if it's only 3 seconds—whereas Last.fm only stores plays that it considers to be actually listening to the song. I don't know how Last.fm does this, so I approximated it with the condition row.reason_end === "trackdone" || row.ms_played > 120_000, which is definitely inaccurate.

Due to me not really knowing this condition, you should be interpret the charts in this post with a low level of confidence as how I modify this condition can have a big impact on the output. That said, I think how this condition is structured is probably good enough to get some idea about what's going on.

At this point I have the Last.fm data in lastFmData and Spotify data in spotifyData.

Outputting difference in total plays in 2023

To start, I wanted to find out on which days did I have more total plays in Spotify vs Last.fm in 2023:

import { display } from "https://deno.land/x/display@v1.1.2/mod.ts";
import * as Plot from "npm:@observablehq/plot@^0.6";
import { DOMParser } from "npm:linkedom@^0.16";

interface DateRange {
  start: Date;
  end: Date;
}

function filterData(data: NormalizedRow[], dateRange: DateRange) {
  return data.filter(row =>
    row.date >= dateRange.start && row.date < dateRange.end
  );
}

async function displayPlaysPerDay(dateRange: DateRange) {
  // There is most definitely a better way of doing this directly with
  // observablehq/plot, but I didn't want to spend too much time going
  // through the documentation figuring it out

  function getDayKey(date: Date) {
    return date.getFullYear() + "-" + date.getMonth() + "-" + date.getDate();
  }

  function getPlaysPerDay(data: NormalizedRow[]) {
    const counts = new Map<string, number>();
    for (const { date } of data) {
      const day = getDayKey(date);
      counts.set(day, (counts.get(day) ?? 0) + 1);
    }
    return counts;
  }

  const spotifyPlaysPerDay = getPlaysPerDay(
    filterData(spotifyData, dateRange),
  );
  const lastFmPlaysPerDay = getPlaysPerDay(
    filterData(lastFmData, dateRange),
  );
  let date = dateRange.start;
  const rows = [];
  while (date < dateRange.end) {
    const day = getDayKey(date);
    const spotifyCount = spotifyPlaysPerDay.get(day) ?? 0;
    const lastFmCount = lastFmPlaysPerDay.get(day) ?? 0;
    rows.push({
      date: new Date(date),
      count: spotifyCount - lastFmCount,
    });
    date.setDate(date.getDate() + 1);
  }

  // create a virtual document
  const document = new DOMParser().parseFromString(
    `<!DOCTYPE html><html lang="en"></html>`,
    "text/html",
  );

  // output the results
  await display(Plot.plot({
    marks: [
      Plot.line(rows, {
        x: "date",
        y: "count",
        z: null,
        stroke: (r) => r.count >= 0 ? "green" : "red",
      }),
    ],
    document,
  }));
}

await displayPlaysPerDay({
  start: new Date("2023-01-01 00:00:00 EST"),
  // I downloaded all the data from Last.fm around December 1st, but
  // it would be several weeks until I received the data from Spotify.
  end: new Date("2023-12-01 00:00:00 EST"),
});

Outputs:

Positive (green) values show more Spotify plays on a day. Negative (red) values show more Last.fm scrobbles.

In an ideal world, the Spotify plays would be a subset of the Last.fm ones or only lag behind by a day or two (as shown occasionally in the chart), but that doesn't seem to be the case here. I further looked into these numbers and found that there are ~830 plays that are exclusive to Spotify and ~410 scrobbles exclusive to Last.fm. It's not clear who is at fault here getting the data into Last.fm and I don't want to speculate in this post.

Overall, this is not terrible, but it also doesn't seem perfect.

Past years play count difference - 2017-2022

Looking at past years, here's 2017-2022 inclusive:

await displayPlaysPerDay({
  start: new Date("2017-01-01 00:00:00 EST"),
  end: new Date("2023-01-01 00:00:00 EST"),
});

The large red spike was me backfilling Last.fm manually with their API because scrobbling got disconnected for a month or so.

Past years play count difference - 2012-2016

Finally, here's 2012-2016 inclusive:

await displayPlaysPerDay({
  start: new Date("2012-01-01 00:00:00 EST"),
  end: new Date("2017-01-01 00:00:00 EST"),
});

My transition to Spotify started in 2012 and it seems like the Last.fm / Spotify integration for my account was very reliable at the start.

Top 10 artists plays exclusive to Spotify in 2023

I also looked at the plays exclusive to Spotify in 2023 and saw this huge standout, which explains why my top artist on Spotify (Alpha 9) was barely present in the Last.fm:

Thoughts on Deno Jupyter Notebook experience

Overall it was quite enjoyable to use Deno in a Jupyter Notebook.

I like how dependencies are expressed in the notebook itself. Deno's single file scripting support translates well to notebooks.
- Of course, I could have put the dependencies in the deno.json file if I wanted. That's useful for scenarios like sharing the same dependencies between notebooks.
TypeScript support in the editor makes it easy to understand APIs directly in VS Code.
With TypeScript, type checking errors alert me in real time about certain mistakes.

There were a few annoyances that should be improved over time though.

I didn't like that I had to write this code:
```
const document = new DOMParser().parseFromString(
  `<!DOCTYPE html><html lang="en"></html>`,
  "text/html",
);
```
I'm not sure what the solution is to getting rid of that, but I feel like it's something that could be abstracted away. That said, it's not too big of a deal.
TypeScript types flowing between notebook cells don't work at the moment.
- I opened https://github.com/denoland/deno/issues/21709
- It was pointed out that https://github.com/denoland/vscode_deno/issues/932 is the tracking issue.
- Looks like this depends on the lsp-types crate in Rust supporting notebook cells
  - https://github.com/gluon-lang/lsp-types/pull/268
In VS Code, the "Save As" button for SVGs should show the last saved location rather than the current folder in the "Save As" dialog, in my opinion.

Other than that, I really enjoyed the experience and I'm looking forward to JavaScript/TypeScript becoming more prevalent in Jupyter Notebooks.

Disabling the required modifier informing System.Text.Json

2023-04-15T21:00:00Z

Today, I looked into C# 11's new features, which include the required modifier. According to the docs, this is what it does:

The required modifier indicates that the field or property it's applied to must be initialized by an object initializer. Any expression that initializes a new instance of the type must initialize all required members. The required modifier is available beginning with C# 11. The required modifier enables developers to create types where properties or fields must be properly initialized, yet still allow initialization using object initializers.

This is great because I can now define a type like the following:

public record MyRecord
{
  public required string MyValue { get; set; }
}

And when I go to initialize it I will get a compile time error when not specifying this property in an object initializer, similar to how TypeScript works by default:

var myValue = new MyRecord
{
  // Error - CS9035 - Required member 'MyRecord.MyValue' must be
  // set in the object initializer or attribute constructor.
};

`required` modifier and `System.Text.Json`

Here lies the problem. After upgrading some code to use required, I started getting runtime exceptions. The reason is that in .NET 7, three ways were added to mark a property or field as required for JSON deserialization:

There are three ways to mark a property or field as required for JSON deserialization:

By adding the required modifier, which is new in C# 11.

By annotating it with JsonRequiredAttribute, which is new in .NET 7.

By modifying the JsonPropertyInfo.IsRequired property of the contract model, which is new in .NET 7.

Source

In my opinion, and without knowing all the details, the required modifier should not have been on that list. It all boils down to this:

Ensuring that a property appears in an object initializer and ensuring that a property is required for JSON serialization are separate matters.

Exception 1 - Nullable types without a JSON property

Take this example:

using System.Text.Json;

var result = JsonSerializer.Deserialize<MyRecord>("{}");
Console.WriteLine(result?.MyValue);

public record MyRecord
{
  public string? MyValue { get; set; }
}

Here I have a nullable string property. In this example, it will deserialize to null because it doesn't appear as a property in the empty JSON object. This code works fine.

Now let's upgrade to C# 11 and take advantage of the required keyword to ensure that this property is always assigned to in an object initializer:

public record MyRecord
{
  public required string? MyValue { get; set; }
}

We've just created a runtime exception in the above code.

System.Text.Json.JsonException: 'JSON deserialization for type 'MyRecord' was missing required properties, including the following: MyValue'

In my opinion, the required modifier should have no effect on this and deserialization should not throw an exception similar to before. This would be a far less error-prone default for users of the API. How often do developers care about nullable properties not appearing in the JSON? Nullable properties are often excluded to reduce the serialized data's size.

Instead, if I wanted this behaviour, I should instead have been able to opt into it via the JsonRequiredAttribute:

// my desired API for the above behaviour
public record MyRecord
{
  [JsonRequired]
  public required string? MyValue { get; set; }
}

.NET Runtime issue that was closed as by design: #76527

Exception 2 - Ignoring a property with a `required` modifier in deserialization

Say we have some data that we use in our application on the server and we also want to send it to the client, but without a property. This type is only ever serialized and never deserialized.

Instead of defining a new type, we could be a bit lazy and just mark the property as ignored via [JsonIgnore].

public record MyRecord
{
  // should be sent to the client
  public string MyClientProperty { get; set; } = null!;
  // should only be accessible on the server and not sent to the client
  [JsonIgnore]
  public string MyServerProperty { get; set; } = null!;
}

This works fine. Let's upgrade to using the required modifier in C# 11 to ensure the server always assigns to this property in object initializers:

public record MyRecord
{
  // should be sent to the client
  public required string MyClientProperty { get; set; }
  // should only be accessible on the server
  [JsonIgnore]
  public required string MyServerProperty { get; set; }
}

We've unfortunately just introduced a runtime exception in our code:

System.InvalidOperationException: 'JsonPropertyInfo 'MyServerProperty' defined in type 'MyRecord' is marked required but does not specify a setter.'

This fails because System.Text.Json does serialization AND deserialization validation on the type. The required modifier means this will fail deserialization (which we won't ever do in this case). If anything, it seems there should be a way to mark a type as serializable only, similar to what serde does.

.NET Runtime issues that were closed as by design:

Solution

The above two runtime exceptions were just what I ran into within a few minutes of trying out the required modifier so there might be more.

To figure out how to get my desired behaviour, within System.Text.Json we can see it does the following to determine if a property is required or not (Source):

propertyInfo.IsRequired =
  memberInfo.GetCustomAttribute<JsonRequiredAttribute>(inherit: false) != null
    // shouldCheckForRequiredKeyword is based on the context of where the property appears
    || (shouldCheckForRequiredKeyword && memberInfo.HasRequiredMemberAttribute());

Luckily, they have provided a way to override this and a hint is given on the Required properties page I linked to earlier. Essentially, we need to create a custom TypeInfoResolver that builds upon the functionality of the DefaultJsonTypeInfoResolver to only set a property as required when it has the JsonRequiredAttribute.

using System.Text.Json;
using System.Text.Json.Serialization;
using System.Text.Json.Serialization.Metadata;

var result = JsonSerializer.Deserialize<MyRecord>(
  "{}",
  new JsonSerializerOptions
  {
    TypeInfoResolver = new DefaultJsonTypeInfoResolver
    {
      Modifiers =
      {
        static typeInfo =>
        {
          foreach (var info in typeInfo.Properties)
          {
            if (info.IsRequired)
            {
              info.IsRequired = info.AttributeProvider?.IsDefined(
                typeof(JsonRequiredAttribute),
                inherit: false
              ) ?? false;
            }
          }
        }
      }
    },
  }
);

Console.WriteLine(result?.MyValue);

public record MyRecord
{
    // [JsonRequired] // uncomment the attribute to see this take effect
    public required string? MyValue { get; set; }
}

In ASP.NET Core, you can set this globally when configuring your JSON options:

builder.Services.AddControllers().AddJsonOptions(options =>
{
    options.JsonSerializerOptions.TypeInfoResolver =
      new DefaultJsonTypeInfoResolver
      {
          // same code as above goes here
      };
});

Now, you can upgrade to liberally using the required modifier without worrying about introducing probably needless System.Text.Json runtime exceptions.

Hope it helps!

Updatable text in a console in Rust

2023-01-30T14:00:00Z

Showing progress bars/messages and getting user input is a common task that many CLI applications have to do. This post will outline a Rust crate called console_static_text, which logs text that can be updated at the bottom of a console window.

API

The API behind this is low level and basic. Essentially there is only one function, which is to render some text that can be sent to the console that overwrites the previous state and updates the state for the new text. The rest of the functionality is helper methods built on top of that.

Here's an example that logs the numbers 0 to 199 to the console, then clears the text:

# cargo.toml dependency
console_static_text = { version = "0.7.0", features = ["sized"] }

use console_static_text::ConsoleStaticText;
use std::time::Duration;

fn main() {
  // returns `None` when not a tty
  let mut static_text = ConsoleStaticText::new_sized().unwrap();

  for i in 0..200 {
    static_text.eprint(&i.to_string());
    std::thread::sleep(Duration::from_millis(30));
  }

  static_text.eprint_clear();
}

Outputs as:

Note that internally eprint and eprint_clear are helper methods around the render(new_text: &str) method. For example, this is the implementation of static_text.eprint(...):

pub fn eprint(&mut self, new_text: &str) {
  // `render` returns `None` when there's nothing to update
  if let Some(text) = self.render(new_text) {
    std::io::stderr().write_all(text.as_bytes()).unwrap();
  }
}

Example - Automatic word wrapping

console_static_text will automatically word wrap text and only update the console if there are changes.

The following example shows word wrapping and how the crate handles the console window resizing:

use console_static_text::ConsoleStaticText;
use std::time::Duration;

fn main() {
  let mut static_text = ConsoleStaticText::new_sized().unwrap();

  let text = format!(
    "{}\nPress ctrl+c to exit...",
    "some words repeated ".repeat(40).trim(),
  );
  let mut last_size = None;

  loop {
    let mut delay_ms = 60;
    let current_size = static_text.console_size();

    if last_size.is_some() && last_size.unwrap() != current_size {
      // debounce while the user is resizing
      delay_ms = 200;
    } else {
      // this will not update the console when the size hasn't
      // changed since the output should be the same
      static_text.eprint_with_size(&text, current_size);
    }

    std::thread::sleep(Duration::from_millis(delay_ms));
    last_size = Some(current_size);
  }
}

I'm not aware of a good cross platform way to handle console resize events, so this example renders every little while and only if necessary (handled internally in the static text object).

Resizing - Not perfect

If the console has resized since the last render, console_static_text does its best to redraw over the previous text, but often some text from a previous render will be left over. From my knowledge, I don't believe it's practical to make this perfect... for example, some terminals might add or remove line breaks when resizing or move the current cursor position to a hard to predict spot.

Here's an example showing how lines can be added or removed on Windows in a console. Both commands executed are the same, but have different outputs based on the size of the console when the command was executed:

I think handling all these scenarios would be a lot of effort and most users don't expect things to be perfect when resizing the console and the render area isn't full screen anyway.

Example - Logging above while outputting

The key to logging above while the static text is outputting is to:

Clear the existing static text.
Log your new text.
Redraw the static text.

For example:

use console_static_text::ConsoleStaticText;
use std::io::Write;
use std::time::Duration;

fn main() {
  let mut static_text = ConsoleStaticText::new_sized().unwrap();

  for i in 0..200 {
    let i_str = i.to_string();
    if i % 10 == 0 {
      // only get the console size once and use
      // the same console size state on all calls
      let size = static_text.console_size();
      let mut new_text = String::new();

      // first clear the existing static text
      if let Some(text) = static_text.render_clear_with_size(size) {
        new_text.push_str(&text);
      }

      // log the new text
      new_text.push_str(&format!("Hello from {}\n", i));

      // then redraw the static text
      if let Some(text) = static_text.render_with_size(&i_str, size) {
        new_text.push_str(&text);
      }

      // now output everything to stderr in one go
      std::io::stderr().write_all(new_text.as_bytes()).unwrap();
    } else {
      static_text.eprint(&i_str);
    }

    std::thread::sleep(Duration::from_millis(30));
  }

  static_text.eprint_clear();
}

Outputs as:

Example - User input

Here's an example that asks the user to make a selection.

It uses Crossterm to get the console size, turn on/off raw mode (necessary for getting arrow key presses), hide/show the cursor, and get key presses. We need to use Crossterm or something like it for this because console_static_text is only concerned with displaying text.

use std::io::stderr;

use console_static_text::ConsoleSize;
use console_static_text::ConsoleStaticText;
use console_static_text::TextItem;
use crossterm::event;
use crossterm::event::Event;
use crossterm::event::KeyCode;
use crossterm::event::KeyEvent;
use crossterm::execute;

struct DrawState {
  active_index: usize,
  message: String,
  items: Vec<String>,
}

pub fn main() {
  assert!(crossterm::tty::IsTty::is_tty(&std::io::stderr()));
  let mut static_text = ConsoleStaticText::new(|| {
    // since we're already using crossterm, get the size from
    // it and don't bother with console_static_text's "sized"
    // feature in order to reduce our dependencies
    let (cols, rows) = crossterm::terminal::size().unwrap();
    ConsoleSize {
      rows: Some(rows),
      cols: Some(cols),
    }
  });
  let mut state = DrawState {
    active_index: 0,
    message: "Which option would you like to select?".to_string(),
    items: vec![
      "Option 1".to_string(),
      "Option 2".to_string(),
      "Option 3 with long text. ".repeat(10),
      "Option 4".to_string(),
    ],
  };

  // enable raw mode to get special key presses
  crossterm::terminal::enable_raw_mode().unwrap();
  // hide the cursor
  execute!(stderr(), crossterm::cursor::Hide).unwrap();

  // render, then act on up and down arrow key presses
  loop {
    let items = render(&state);
    static_text.eprint_items(items.iter());

    if let Event::Key(event) = event::read().unwrap() {
      // in a real implementation you will want to handle ctrl+c here
      // (make sure to handle always turning off raw mode)
      match event {
        KeyEvent {
          code: KeyCode::Up, ..
        } => {
          if state.active_index == 0 {
            state.active_index = state.items.len() - 1;
          } else {
            state.active_index -= 1;
          }
        }
        KeyEvent {
          code: KeyCode::Down,
          ..
        } => {
          state.active_index =
            (state.active_index + 1) % state.items.len();
        }
        KeyEvent {
          code: KeyCode::Enter,
          ..
        } => {
          break;
        }
        _ => {
          // ignore
        }
      }
    };
  }

  // disable raw mode, show the cursor, clear the static text, then
  // display what the user selected
  crossterm::terminal::disable_raw_mode().unwrap();
  execute!(stderr(), crossterm::cursor::Show).unwrap();
  static_text.eprint_clear();
  eprintln!("Selected: {}", state.items[state.active_index]);
}

/// Renders the draw state
fn render(state: &DrawState) -> Vec<TextItem> {
  let mut items = Vec::new();

  // display the question message
  items.push(TextItem::new(&state.message));

  // now render each item, showing a `>` beside the active index
  for (i, item_text) in state.items.iter().enumerate() {
    let selection_char = if i == state.active_index { '>' } else { ' ' };
    let text = format!("{} {}", selection_char, item_text);
    items.push(TextItem::HangingText {
      text: std::borrow::Cow::Owned(text),
      indent: 4,
    });
  }

  items
}

Recommended architecture

The stdout and stderr pipes are a global concept in a CLI application. For that reason, I recommend the implementation of your output to the global stdout and stderr pipes should be controlled in one place and have that code use a single instance of a ConsoleStaticText.

For example, either have a logging implementation using a single ConsoleStaticText that's passed around to do all your logging (as is done in dprint) or create a single global instance of ConsoleStaticText that's used in the application (as is done in dax). Then create an abstraction on top of that to handle rendering your application's state, getting user input, and handle logging at the same time so nothing conflicts.

Projects using this

The following projects I work on are using this now: Deno (in 1.29 and above), dprint, and dax (via Wasm)

All the examples in this blog post are found in the console_static_text repo: https://github.com/dsherret/console_static_text

As always, thanks for reading!

dax - Cross-platform shell tools for Deno

2022-07-18T14:00:00Z

Automation scripts in repositories are often written in a shell scripting language such as bash. That's not ideal—on top of these scripting languages being difficult to use, they're not cross-platform. This makes it harder for Windows users to contribute and there are often small differences between Linux and Mac (or shell configurations) that may lead to broken scripts for contributors.

Using a cross-platform programming language, like JavaScript, is more ideal, but often what can be expressed in a shell scripting language succinctly (such as executing a command) is verbose when using the APIs offered out of the box by JavaScript runtimes.

The library zx made this a lot easier by bringing the best of shell scripting languages into JavaScript with the introduction of an easy to use API, but I believe there are some improvements that can be made to this idea.

In this post I'm going to outline a new tool called dax, which is inspired by zx.

// example dax API usage
let branch = await $`git branch --show-current`.text();
await $`dep deploy --branch=${branch}`;

Cross-platform shell

dax has an API very similar to zx, but its shell is cross-platform using the parser from deno_task_shell with a rewrite of the execution logic in JavaScript.

So commands like the following work the same on Linux, Mac, and Windows:

await $`echo Hello`; // Hello
await $`MY_VAR=there && echo Hello $MY_VAR`; // Hello there
await $`LOG_LEVEL=0 some_command`;

Additionally, the shell has a few built-in cross-platform commands.

// outputs "Hello", sleeps for 1 second, then outputs "There"
await $`echo Hello && sleep 1 && echo There`;

Note: It's not possible to use a custom shell with dax as that's heavily discouraged since it more easily leads to code that's not cross-platform. That said, if you really need to call into sh, for example, then you can run it directly (sh -c <command>).

Exporting shell environment

Any changes to the shell environment will not be exported to the executing process by default.

// outputs: C:\dev\my_project\sub_dir
await $`cd sub_dir && echo $PWD && export MY_VAR=5`;
// outputs: undefined
console.log(Deno.env.get("MY_VAR"));
// outputs: C:\dev\my_project
console.log(Deno.cwd());

However, the shell environment may be exported when desired by using the .exportEnv() method:

await $`cd sub_dir`.exportEnv();
await $`echo $PWD`; // C:\dev\my_project\sub_dir
console.log(Deno.cwd()); // C:\dev\my_project\sub_dir

await $`export MY_VAR=5 && cd ../`.exportEnv();
console.log(Deno.env.get("MY_VAR")); // 5
console.log(Deno.cwd()); // C:\dev\my_project

Note also, that you can modify a shell's environment before executing without changing the current process' environment:

// outputs:
// C:\dev\my_project\sub_dir
// 5
await $`echo $PWD && echo $MY_VAR`
  .cwd("./sub_dir")
  .env("MY_VAR", "5");

High level helpers with an available low level API

When executing commands and you want to get the output, there are several helper methods that make this easy:

// get the stdout of a command (makes stdout "quiet")
const result = await $`echo 1`.text();
console.log(result); // 1

// get the result of stdout as json (makes stdout "quiet")
const result = await $`echo '{ "prop": 5 }'`.json();
console.log(result.prop); // 5

// get the result of stdout as bytes (makes stdout "quiet")
const result = await $`echo 'test'`.bytes();
console.log(result); // Uint8Array(5) [ 116, 101, 115, 116, 10 ]

// get the result of stdout as a list of lines (makes stdout "quiet")
const result = await $`echo 1 && echo 2`.lines();
console.log(result); // ["1", "2"]

In the case you need to access to more detail, that is available too along with several other properties not shown here:

const result = await $`deno eval 'console.log(1); console.error(2);'`;
console.log(result.code); // 0
console.log(result.stdoutBytes); // Uint8Array(2) [ 49, 10 ]
console.log(result.stdout); // 1\n
console.log(result.stderr); // 5\n
const output = await $`echo '{ "test": 5 }'`;
console.log(output.stdoutJson);

No custom CLI or globals

Deno allows for expressing dependencies in code and this is perfect for automation scripts. Instead of needing to install a custom CLI or npm package, you can just import the module directly...

// script.js
import $ from "https://deno.land/x/dax@0.7.0/mod.ts";

await $`echo 'Hello there!'`;

...and run it right away...

> deno run -A script.js
Hello there!

This command could then be easily aliased in a deno task with Deno's task runner meaning your contributors only need to execute the task to run the script without needing to follow any other setup instructions.

Being able to express all your dependencies directly in your script files is a huge advantage that Deno offers over Node.js. It makes it easy to import a module you want to use in a single script without having to manage dev dependencies and do an npm install.

No global configuration

Using zx in a library or application code is a little risky because it has global configuration. With dax there is no global configuration in order to prevent modifying the behaviour of other code using it.

Additionally, with dax you can create your own local $ object to use that has the configuration you like. This is done by using the builder APIs and build$ function.

import { build$, CommandBuilder } from "https://deno.land/x/dax@0.7.0/mod.ts";

const commandBuilder = new CommandBuilder()
  .exportEnv()
  .noThrow();

const $ = build$({ commandBuilder });

// since exportEnv() was set, this will now actually change
// the directory of the executing process
await $`cd test && export MY_VALUE=5`;
// will output "5"
await $`echo $MY_VALUE`;
// will output it's in the test dir
await $`echo $PWD`;
// won't throw even though this command fails (because of `.noThrow()`)
await $`deno eval 'Deno.exit(1);'`;

This CommandBuilder API is what $ uses internally, so you can also design your own APIs that execute shell commands using it.

Utilities on `$`

Since this is a shell scripting toolkit, the module offers several utilities built-in and has all of these available on the $ object for quick access.

Here's a few of them:

await $.sleep(100); // ms
await $.sleep("1.5s");

const denoPath = await $.which("deno"); // path to deno executable

// re-export of deno_std's path
const fileName = $.path.basename("./my_sub_dir/mod.ts"); // mod.ts

// re-export of deno_std's fs
for await (const file of $.fs.expandGlob("**/*.ts")) {
  console.log(file);
}

Fetch alternative

There is a fetch API alternative built-in, but with a less error-prone builder API that throws on non-2xx status codes by default.

// download a file as JSON
const data = await $.request("https://plugins.dprint.dev/info.json").json();
// or long form
const response = await $.request("https://plugins.dprint.dev/info.json");
console.log(response.code);
console.log(await response.json());

Often enough myself and others would forget to handle non-success codes in scripts with fetch, leading to cryptic errors. So the $.request(..) function throws by default. That said, this can be disabled, or only disabled for specific status codes, via the .noThrow() method (ex. await $.request("...").noThrow()).

Note: Similarly to commands, if you don't like the defaults for $.request, you can use build$ to create your own $ with a RequestBuilder that changes the defaults (see custom $).

Logging API

In an effort to simplify logging in scripts (reducing need for a color API), there is also a built-in logging API that only logs over stderr:

// logs with no formatting
$.log("Hello!");
// log with the first word as bold green
$.logStep("Fetching data from server...");
// or force multiple words to be green by using two arguments
$.logStep("Setting up", "local directory...");
// similar to $.logStep, but with red
$.logError("Error Some error message.");
// similar to $.logStep, but with yellow
$.logWarn("Warning Some warning message.");
// logs out text in gray
$.logLight("Some unimportant message.");

// log indented within
await $.logIndent(async () => {
  $.log("This will be indented.");
  await $.logIndent(async () => {
    $.log("This will indented even more.");
    // do maybe async stuff here
  });
});

Looks like this:

Conclusion

I just started on this a little over a week ago so it might evolve a bit and have some breaking changes. Overall, I'd appreciate any feedback. I've already begun integrating this into some of our automation scripts used at Deno to try to find some of the pain points and verbose code that could be simplified.

As always, thanks for reading and I hope this is useful!

Repo: https://github.com/dsherret/dax

P.S. this module was originally called ax, but deno.land/x doesn't allow 2 character module names... so I added a d to the front for Deno, but that so happens to be my cat's name and so this module is now named after him.