How do I write a QuickCheck test "interpreter" for a Free-based program?

Help
#1

Context

Currently, I’m trying to learn how to write, test, and benchmark a full application using the onion architecture via a Free-based DSL and interpreter. To keep things simple, I’m not breaking each layer’s “language”'s members into their own individual data type that I then compose together via VariantF. I plan to do that later.

Following my current understanding of the onion architecture, I’ve broken the code down into these four layers

Core

  • data types for the game’s concepts (e.g. Guess, RandomInt, etc.) and their creation via a smart constructor (e.g. mkRandomInt :: Bounds -> Int -> Either NotWithinBounds RandomInt)
  • It’s language:
data GameF a
  = ExplainRules a
  | SetupGame (GameInfo -> a)
  | PlayGame GameInfo (GameResult -> a)

Domain

  • a function called gameLoop that keeps requesting the user for a guess until the player wins (correct guess) or loses (ran out of guesses)
  • The language I defined here breaks the “big idea” Core language into something that looks like an implementation:
data RandomNumberOperationF a
  = NotifyUser String a
  | DefineBounds (Bounds -> a)
  | DefineTotalGuesses (RemainingGuesses -> a)
  | GenerateRandomInt Bounds (RandomInt -> a)
  | MakeGuess Bounds (Guess -> a)

API

  • interfaces with the Infrastructure level by instantiating the Core types correctly before passing them off to the Domain level. If the user provides an invalid input, the API will recursively run until it gets a valid input.
  • The language:
data API_F a
  = Log String a
  | GenRandomInt Bounds (Int -> a)
  | GetUserInput PromptMessage (String -> a)

Infrastructure

  • I implemented this level by using Node.ReadLine in Aff.

Problem

The program works fine, but I’m having trouble figuring out how to test this program using QuickCheck.

I believe the test code should work something like this:

  • generate two values: 1) all values needed to interpret API_F (i.e. the random integer and all the user’s inputs) and 2) the expected game result given those inputs
  • evaluate the computation to its final output using runFree rather than interpreting it into another monad via foldFree

Then, I imagine the test would look something like this:

runCore :: Free GameF ~> Free RandomNumberOperationF

runDomain :: Free RandomNumberOperationF ~> Free API_F

newtype TestData =
  TestData { random :: Int, userInputs :: Array String, result :: GameResult }

quickCheck (\(TestData data) -> 
  let gameResult = runAPI data.random data.userInputs (runDomain (runCore game))
  in gameResult ==? data.result
)

runAPI :: Int -> Array String -> Free API_F GameResult -> GameResult
runAPI random userInputs = runFree go where
  go = case _ of 
    Log _ next -> pure next -- we don't care what gets logged here, so ignore it
    GenRandomInt _ reply -> pure (reply random)
    GetUserInput _ reply -> 

        nextInput <- {- not sure how to write this code... -}

        pure (reply nextInput)

I’ve written the TestData generators, but I don’t know how to produce nextInput above.

What I’ve Tried

I wasn’t sure how to resolve this as I’m not sure whether there is some code I should be using that I’m not aware of, or whether my approach is just severely flawed and I need to do something different. As I thought about writing this, I then came up with one idea (use Refs), but that didn’t work:

I thought about changing the type of userInputs in TestData to Ref, creating a new Ref in my TestData generator before calling pure $ TestData { ... userInputs: ref ... } and using liftEffect $ Ref.modify' in go to produce the next value. However, that resulted in a compiler error:

Could not match type
         
    t1 a0
         
  with type
      
    a0
      

while checking that type t1 t2
  is at least as general as type a0
while checking that expression pure next
  has type a0
in value declaration runAPI

where a0 is a rigid type variable
        bound at line 123, column 10 - line 131, column 35
      t1 is an unknown type
      t2 is an unknown type

Questions

So, my questions are:

  • How do I write a test “interpreter” for a Free-based program, so that it is testable using QuickCheck?
  • Is my current approach flawed at a fundamental level, and if so, how/why?
  • What can I do to produce the nextInput value?

I would provide a gist of the code, but I think that’s a bit much to ask of someone.

#2

Looks like the solution is via Test.QuickCheck.Monadic from the original Haskell library, which has not yet been imported into the Purescript version:

There’s also this variant which is experimental but has two advantages:

  1. specifying the correctness of your programs becomes less adhoc
  2. you get testing for race conditions for free
3 Likes
#3

I think I just figured this out. In the Data Types a la Carte paper, it mentions that the state monad isn’t a free monad, so one cannot compose (i.e. use) it in a computation. However, it explained that one could define a language that simulates the state monad.

When I recalled that, I then remembered the Run already has implementations for the Reader/State/Writer monad transformers. So, rather than interpreting my program into an Effect-based interpreter, which AFAIK QuickCheck cannot currently do, I can interpret it into one of those types and then run them.

I hadn’t considered this because I was focusing on writing tests for the Free-based version of my code, not the Run-based version.

Edit: After working more on the Run-based version, I got this to work.

1 Like
#4

I wish I’d been able to participate in this thread earlier, but I don’t know how to do this! I’d love to see the solution you came up with.

#5

See this folder and 04-Infrastructure.purs:
https://github.com/JordanMartinez/purescript-jordans-reference/tree/latestRelease/21-Hello-World/09-Games/test/02-Random-Number

3 Likes