Documentation is a critical component to the developer experience serving as an invaluable resource for any code project. When a project is small a simple README will suffice but as it begins to grow it’s common for documentation to expand across multiple pages and files. And since documentation is a living document it never reaches a final draft. This introduces the risk that documentation will drift from the code that it represents which could create a frustrating experience or erode the trust developers put in the documentation.

How then can the contract between documentation and code be upheld? One option would be to incorporate a process around code changes to find documentation that needs to be updated, for example:

• Add or create a pull request template with tasks to check for necessary documentation updates
• At release, check features and breaking changes, then check and update relevant documentation

There is also the method of including unit tests alongside API documentation generated from docblock comments like this example from the doctest-js project:

/**
 * Returns the sum of 2 numbers
 *
 * @example sum(1, 2)
 * //=> 3
 */
export const sum = (a, b) => {
  return a + b
}

If the implementation were to change where 3 wasn’t returned from sum(1, 2) for some reason, then this docblock example would break and likely be caught in CI. This works well for unit testing around specific APIs and API documentation but what about longer extensive examples that are scattered across more extensive documentation?

Let’s assume there’s an imaginary drawing library and within its documentation there is the following code example:

import { canvas, circle, rect } from "mega-library";

// calling canvas automatically draws the contents
// to the screen
canvas([
  circle(20, 50, 50),
  rect(30, 40, 200, 200)
]);

To summarize the example, the initial version of this library has:

• canvas, a function that takes an array of objects to draw automatically
• circle, a function that returns a circle, in this case with a 20px radius at coordinates of 50, 50.
• rect, a function that returns a rectangle with a width of 30px, height of 40px at coordinates of 200, 200.

In the next version, after some feedback, some breaking API changes have been made so that the circle and rect functions instead of providing coordinates as individual parameters will instead use an object (for example: our circle from before will be circle(20, { x: 50, y: 50 })). Additionally, instead of automatically drawing the canvas to the screen the canvas must be explicitly drawn via canvas.draw().

Realizing that updates to the code example are needed changes are hastily made resulting in a few mistakes:

import { canvas, circle, rect } from "mega-library";

canvas([
  circle(20, { x: 50, y: 50 }),
  rect(30, 40, 200, 200) // << whoops, forgot to update to { x: 200, y: 200 }
] // << invalid js, forgot the closing parentheses, `]` should be `]);`

// canvas.draw() << forgot to include calling the `draw` method on canvas

Despite best intentions there’s been a few mistakes made to the code example:

1. The rect API didn’t get updated to its new usage.
2. The code won’t even compile since I have a syntax error with the missing closing parentheses on the call to canvas.
3. Even if the previous issues were corrected this example would leave users confused because the final canvas.draw() call is missing. Without calling the draw method the example would still run, there would be no console errors, and nothing would be drawn on the screen leaving developers to debug why the example in documentation didn’t work.

I ran into these similar types of issues while working on my own library for GraphQL Mocking, graphql-mocks. My documentation was written largely in markdown but built into a static react app through facebook’s documentation project, Docusaurus. Here’s where the magic comes in, my solution was to write a Docusaurus plugin that would take a .js file containing a code example that could be converted to markdown but also import the same .js file and run it against assertions in tests.

Continuing the example, the previous code example would be in a file: draw-example.source.js. The Docusaurus plugin would takes the contents of draw-example.source.js and generate a markdown file. Since Docusaurus supports importing and rendering markdown in mdx files (a mix of markdown and react/jsx) I was able to use the generated markdown directly in the documentation:

// docs/drawing.mdx
import DrawExample from 'code-examples/draw-example.source.md';

Here we can see how to draw a circle and reactangle on a canvas:
<DrawExample/>

Where <DrawExample/> would represent the rendered markdown including code formatting and highlighting where necessary.

In testing the code example the same source javascript file would be used, draw-example.source.js, imported and then tesed with necessary assertions:

import { expect } from 'chai';
import { result } from './draw-example.source';

it('docs/draw-example', async () => {
  // testing implementation details from `draw-example`
  // totally made up, but you can see how this works

  // test two objects to be drawn
  expect(result.objects.length).to.equal(2);
    
  const [circle, rect] = result.objects;

  // test first object is a circle and its properties
  expect(circle.type).to.equal('circle');
  expect(circle.radius).to.equal(20);
  expect(circle.x).to.equal(50);
  expect(circle.y).to.equal(50);  

  // test second object is a rect and its properties
  expect(rect.type).to.equal('rect');
  expect(rect.width).to.equal(30);
  expect(rect.height).to.equal(40);
  expect(rect.x).to.equal(200);
  expect(rect.y).to.equal(200);
});

The tricky part in creating the custom plugin was how to support both test and documentation versions of the same source .js file. In the case of the example draw-example.source.js it ends up looking like this:

import { canvas, circle, rect } from "mega-library";

canvas([
  circle(20, { x: 50, y: 50 }),
  rect(30, 40, { x: 200, y: 200 })
]);

codegen(`
  const {output} = require('./helpers');
  module.exports = output("module.exports.result = canvas.draw();", "canvas.draw();");
`);

Note: I’ve fixed the example’s previous mistakes above here. Had I left the file with the mistakes it would have failed to be parsed and/or the test would have failed in CI and I would have been forced to update the code example or test anyways.

You might notice that new codegen bit at the end, that’s from babel-plugin-codegen. It’s a babel plugin that allows the file to first run arbitrary javascript within a codegen template string, with the outer module.exports block being what gets injected into the result of the final file. If this seems a bit meta, it is, I recommend checking out the babel-plugin-codegen repo for more examples.

For the setup to work I had to include this babel plugin in the Docusaurus babel file and within the mocha test runner otherwise this floating codegen reference would either fail to be parsed in tests or awkwardly show up in code examples.

output(`
  "/* string of test environment javascript to be included */",
  "/* string of documentation environment javascript to be included */",
`)

The output helper takes two strings. The first being javascript to be included in the case of a test environment, usually module.export parts of the example to be tested. The second string is javascript to be shown for the code example. The difference between these should be kept to a minimum to avoid introducing a mistake between the two.

In draw-example.source.js above I’m outputting module.exports.result = canvas.draw(); in a test environment and canvas.draw(); in a documentation environment when being shown in Docusaurus. This lets me keep the majority of the example the exact same between the test and documentation cases only conditionally splitting at the end for what needs to be exported for testing or what gets shown to the user in the rendered documentation.

How does the output helper know which environment it’s in. Well, this output is running in the context of a babel plugin, likely within node, so in my case I rely on an environment variable CODE_EXAMPLE_ENV with possible values of test or docs. Signalling the environment could be done in different ways but this has worked for me. The environment variable is set in yarn scripts for creating documentation or running tests so I don’t need to remember to include it. If does happen to be unset an error is thrown reminding that it is required to be set.

While it was initially challenging connecting the different pieces of this code example system, it has really ended up working great. I have a bunch of examples written this way now. I am able to guarantee that the code examples are valid parsable javascript that import real parts of my library and test the integration for the entire example. In generating the markdown example I also run the resulting code example through prettier to ensure consistent formatting.

Running the documentation tests in CI for every pull request as an early warning system means I have been able to catch subtle API changes that have required updates not just to the code example itself but the explanations around the code example.

I think the ergonomics around codegen and the output helper could be improved, maybe even into a newer babel plugin that enforces some conventions around this use case. This version is tightly coupled to Docusaurus and its build system but could be adapted to be more generically robust.

Even with the overhead I believe this is a huge step forward in ensuring that documentation, an extremely important artifact in any project, is kept accurate. This helps build trust with those who use the project, avoids frustrating experiences, and adds an additional layer of testing around the intended use of the project. Let me know your thoughts, especially if there are similar examples of this concept in the wild.