PoC: Generating Tests from purescript doc examples

Hi, I’ve been thinking about how to improve library documentation and found that I – at least – really like to read the examples in the docs first and foremost, as they provide a lot of information in little time.

But writing documentation (and examples in the docs) is annoying - especially because they can easily get outdated, as they are not tested.

My idea is to generate tests from the examples in the doc comments. A PoC-Implementaion can be found here: https://github.com/csicar/purescript-doctest

Take for example the documentation of Array.fromFoldable, which looks like this:

Array.fromFoldable (Just 1) = [1]

We could then parse this docs-comment and generate a test-file, that looks like this:

main = suite "Array" $ do
    it "Spec : fromFoldable" $ 
        Array.fromFoldable (Just 1) `shouldEqual` [1]

Making it practical

No defined Synax & Semantics

Parsing such an example would be hard, as = can serve the purpose of declaring a variable and asserting equality (like here).
Examples often also assume, that some imports have magically appeared (like Just in the example above).
Sometimes it is useful to show the reader what type an expression has. I’ve used -- :: TypeOfTheThing instead of :: TypeOfTheThing in the past, as it shows, that specifying a type is not required. But this again has no well-defined semantics.

Some examples aren’t and shouldn’t be runnable tests

Sometimes it is useful so have “handwavy” examples with ellipses and abstractions applied to them, which means we need a way to distinguish between “runnable” and “handwavy” examples

Examples must be easy to read and write

Both reader and writer of examples should not need to learn a new language.

PSCI-Logs as a solution?

Luckily we already PSCI with well-defined syntax and semantics, which is understood by most purescript programmers. PSCI also supports showing the value of an expression and showing a type with :t.

The idea is to use the logs from a PSCI-Session as the content of examples in the docs:

> Array.fromFoldable (Just 1)

This makes it clear, which part is input (everything after >) and what part of the output (and should be asserted in a test).
This also works for types, of course:

> import Data.Maybe
> :t Nothing <> Just []
forall t4. Maybe (Array t4)

The syntax used by the PoC is actually to similar to PSCI, that I was able to reuse the PSCI-Command parser, that is build into purs.


This idea is so neat!

Is the link missing?

Yeah, I’ve missed that. Thanks for letting me know. It’s fixed now

@csicar this might interest you: https://github.com/paulyoung/literate-purescript


I like how rustdoc lets you hide some of the setup code from appearing in the documentation. Allow viewing the setup code in the comment block by clicking on the source link in pursuit, but otherwise keep the reference page from becoming too busy.

1 Like

Yeah, good idea.That would probably require support in purs itself
It would also be possible to “hide” some commands after the codefence start:
```purescript run \n > import Test \n > helper = …

I was aware of the project, good idea and it’s really cool that you where able to implement it in purescript itself.
Could be useful for also testing examples in a README.md file, for example.

Is the goal of this project to match the features of doctest for Haskell’s Haddock?

Here’s their solution for setup code, along with example input comments and output docs for reference.

1 Like

I wasn’t trying to match the feature set of any other similar system in particular. But it’s good to get a perspective on how other people solved the problem.

I’m not yet sure where the right balance between verbosity and explicitness is. E.g. do we even want to hide setup code like imports? Often a library user wants to know how the library is indended to be imported (import ... as Xyz exposing (Xyz)) .

I should also add, that currently all examples in a module share a test-file, so you only need to write imports once per module

1 Like

Would you consider renaming this from purescript-purepur to purescript-doctest to match the precedent set by other languages (haskell, rust, python, elixir, elm, c++)?


As it happens I took a stab at this 2 years ago too: https://github.com/hdgarrood/purs-doctest/ your approach looks very similar to mine, which I guess is encouraging!

It looks extremely similar to you library. Even to the point of using the same parts (Cheapskate and Lang.Purescript.Interactive) for parsing https://github.com/hdgarrood/purs-doctest/blob/master/src/Language/PureScript/Docs/Doctest/Parse.hs#L58. That’s actually quite amazing :slight_smile:

I’ll need to look into your code some, as you seems, that you do not need to parse and print normal declarations.

I also did not know that $Something is a valid Namespace. I’ll take advantage of that :wink:

I probably should spend more time looking for other tools, that are trying to solve the same problem.
That could’ve avoided some work…

I saw that some stuff is outdated in purs-doctest, but I think by combining the two we can get a decent tool


Yeah, I’m happy to do that. The name was a placeholder anyway

$Something is not really a valid namespace - the compiler won’t let you define a module with that name. It is, however, valid as a JS identifier. We take advantage of this fact in the repl to ensure that the temporary module we generate for executing code in the repl cannot have the same name as a user-defined module.

1 Like

Oh, I see. Thanks for the clarification