Collaborating on improving the REPL evaluator

I would like to improve the REPL experience in two ways:

  1. Currently it reevaluates all bindings in your session on every expression submitted. That is, it bundles up a new module with everything in it and runs a fresh node process. This makes it problematic if you want to support effectful bindings. Currently you cannot run effectful code and use the result in subsequent expressions. You can use unsafePerformEffect, but that will run your effects again and again. I would like to run a persistent node process for the entire repl session rather than a process to evaluate each expression.
  2. There’s no way to run async code. Once we allow effectful code to run, it would be nice to support running Aff, where the repl will block until a result is received.

For the first case, I’ve written some JavaScript to handle a persistent REPL session, where you can submit bindings or an expression to evaluate. It correctly handles recursive binding groups, which is a little tricky. It would need a basic socket server wrapper, but I don’t see any issues with that. The real work is updating the purs repl code to manage this node process, and likely changing how it invokes the compiler, so that it can grab and submit the individual bindings to the evaluator instead of a module with everything in it. Additionally it would have to know how to extract the type of an evaluated expression and retain that in the type environment.

For the second case, I would like to change the signature of the Eval code to return an Aff-like signature instead of Effect Unit, and also return the evaluated type. Something like

eval :: forall a b.
  Eval a b =>
  -- The value to evaluate
  (Unit -> a) ->
  -- The callback to invoke if an exception occurs
  (Error -> Effect Unit) ->
  -- The callback to invoke with evaluated value `b`
  (b -> Effect Unit) ->
  -- Returns a canceller, which will cleanup whatever is running when invoked
  Effect (Effect Unit)

I’ve prototyped something like this in my evaluator, and doesn’t seem to be an issue.

Would anyone like to collaborate on this? Do other PureScript maintainers or users have any opinions on these changes?

11 Likes

I’m definitely keen for this, although I don’t think I can set much time aside to collaborate on it unfortunately. I’d love to be able to support <- in the repl, for instance.

@kritzcreek worked on something called purpl a while back that I think might have had similar goals, so he may have some insight.

There’s no way to run async code. Once we allow effectful code to run, it would be nice to support running Aff , where the repl will block until a result is received.

Well, that explains the error I was unable to get past this evening. lol

Neat! Once I’m done with my semester in about a month I’ll have a lot more free time and would love to help out!

1 Like

It appears that both Aff code and <- work in the repl in a :paste block. But in this case, all three lines are printed after the delay. Maybe this is really just Effect code with the launchAff_. Is the goal of these enhancements to get this working without the :paste block, and have the print timing work as expected?

> :pa
… launchAff_ $ do
…   log "waiting"
…   toAff $ sleep 1000
…   log "done waiting"

waiting
unit
done waiting

It appears that both Aff code and <- work in the repl in a :paste block. But in this case, all three lines are printed after the delay. Maybe this is really just Effect code with the launchAff_ . Is the goal of these enhancements to get this working without the :paste block, and have the print timing work as expected?

I mean <- as a top-level repl binding, so you can do:

> a <- readFile "file.txt"

And a would be in scope for future repl calls. The repl currently does not let you evaluate effectful code, and maintain a binding to its result.

2 Likes

Is your code published anywhere so I can take a look?

1 Like

Here is a gist of a session evaluator.

2 Likes

Hi @natefaubion!

I am working on some other things for the ecosystem but this is something that I wished for a bit now. I would like to help you improve the REPL.

So I went through the code, and somehow understand a little what is happening.

That is, it bundles up a new module with everything in it and runs a fresh node process.

I believe that these are two functions that you’re referring to, correct?

The real work is updating the purs repl code to manage this node process, and likely changing how it invokes the compiler so that it can grab and submit the individual bindings to the evaluator instead of a module with everything in it

What part of of the code we would need to change? Using your solution the evalExpr function should be passed to the handlExpression as the evaluator?

Can you give me a bit more detail about how you’d expect your solution to fit with the current code?

2 Likes

Right now the node repl backend has noops for everything but eval, however it supports other stateful hooks. We should use the other hooks to manage a basic socket server which wraps the session evaluator. This server should support things like adding imports, evaluating declarations/binding groups, evaluating expressions, and resetting state.

The current workflow is too coarse to support this however. handleExpression for example builds a new module with the entire history baked into it as let bindings and writes that module out. The node backend will then run a new process invoking that whole module. This means every binding in your history is reevaluated on every submitted expression. handleExpression and handleDecls would need to return only the JS source for those expression that have been submitted, so they could be sent to the repl evaluator. A non-trivial part of the work will be teasing this out.

1 Like

I had a look at the version that @kritzcreek was working on (thanks for mentioning it @paulyoung ), and it seems like he was trying to use node’s vm object. It seems like vm have all the functionalities that we would need, is that not a possible solution? (There is also a repl module)

Also, wouldn’t it be better to just rip the REPL into a separate project? What do we gain from having it within the compiler?

1 Like

It was not clear to me what advantages you get by using vm as opposed to stock eval. You still have to be careful about scope and shadowing since you still want PureScript’s let semantics. That is you don’t want to mutate a context object when you add bindings. I did not look too heavily into all that the vm module offers. What I wanted to achieve was possible with normal eval, so I didn’t see a need to involve it. If there’s a clear advantage to vm, then it should probably be used instead.

You want your compiler and repl to be in sync. A separate project just introduces ways for this to drift, so the question is what advantage do you get by having it as a separate project?

1 Like

I don’t think that there are any clear advantages except that it seems like it has everything that allows us to control the context/scope in which a code can be evaluated. I am working on implementing small POC.

You want your compiler and repl to be in sync. A separate project just introduces ways for this to drift, so the question is what advantage do you get by having it as a separate project?

I see it as an app/tool of its own, so I just thought that it would just be easier to manage/document if it was separate.

As of now, we cannot use do notation in the REPL — what part of the code do we need to change so that we can allow top-level binding?

Last question, it seems like quite a lot would not need to change so would this improvement end up requiring pretty much re-writing the core of the REPL module and the handleCommand of the Language.Purescript.Interactive module?

1 Like

I’m interested in this as well, I had a crack a long time ago at a POC of a persistant repl env and have long thought it should be done properly.

I’m resurrecting the repl for the purerl backend at the moment, which is rather involving both making use of the compiler library and also duplicating most of the repl code in there - which is not something that should influence the design here as such, just to say that I’m in the headspace

1 Like