Asking for guides of making new backends

[thautwarm]

Hi, Purescript community,

I just found Purescript extremely practical and productive(syntax level support for row polymorphisms in a higher rank programming language looks perfect to me), thus I want to make several code generation backends(for some dynamic languages like Python/Julia/Ruby/Lua, etc) for you.

The code at https://github.com/purescript/purescript/blob/master/src/Language/PureScript/CodeGen
shows it’s likely to be not a cumbersome task, but before I start I wonder if implementing the codegen interface like CodeGen.JS is sufficient to export the whole ecosystem of Purescript to another programming language? Is there anything special I need to take care of? e.g., any code other than those in CodeGen responsible for code generation, or some special hacks for JavaScript, etc.

Thanks in advance.

[thimoteus]

I don’t know the answer to your questions, but have you taken a look at purescript-native?

[thautwarm]

Hi, thanks for your reply. Incidentally, I was just now checking the code, it’s awesome and I appreciate it a lot.

I’m considering to reuse purescript-native by providing a new CoreImp.AST -> Text, but still have some problems with this pipeline(especially, where to implement the AST -> Text generator, where to register a new back end, whether the IL.Optimizers applicable for a new back end.

[hdgarrood]

The intended API for alternative backends is to use the -g corefn option with purs compile, which causes the compiler to write out an intermediate language called the core functional representation (often abbreviated to “corefn”) alongside the output JavaScript files. This way, alternative backends shouldn’t need to maintain a fork of the whole compiler, which requires a lot more effort. See https://github.com/purescript/documentation/blob/master/ecosystem/Alternate-backends.md and in particular the “Actively Maintained” section for examples.

There are no special hacks for JavaScript in the corefn output as far as I’m aware; it’s intended to be agnostic of the compilation target. However, something you do need to be aware of is the fact that quite a lot of libraries use the FFI. Whenever a library uses the FFI, if you want to use that library with your backend, you’ll need to write code equivalent to the JavaScript FFI code in your target language; quite a lot of libraries use the FFI so this can be quite a big task. Additionally, some of those uses of the FFI do rely on JavaScript specific behaviour, in which case it can be less clear what to do. For example, Aff is based to a large extent on JavaScript’s single-threaded event loop model. You’ll have to use your judgement.

[paulyoung]

This is incomplete but should give an idea of using the corefn JSON: https://github.com/paulyoung/pureswift

If I was to do that project over again I’d probably do it in Haskell instead and try an approach that offers the benefits of forking the compiler but makes backends more pluggable.

[RAbraham]

That would be so great! Can I put a vote for Python?

[thautwarm]

Your reply is really informative which did help me with the implementation. Sorry for my late text reply.

Now after some investigations, I think I have realized and is encountering the FFI problems you mentioned.

Whenever a library uses the FFI, if you want to use that library with your backend, you’ll need to write code equivalent to the JavaScript FFI code in your target language; quite a lot of libraries use the FFI so this can be quite a big task. Additionally, some of those uses of the FFI do rely on JavaScript specific behaviour, in which case it can be less clear what to do. For example, Aff is based to a large extent on JavaScript’s single-threaded event loop model. You’ll have to use your judgement.

I found so many tasks to do, even for the most essential library prelude. I also realized this is actually a larger thing than the bare backend implementation, thus now I got stuck.

I wonder if you or other people have any idea about addressing the FFI problems.

I used to think if I can gather many people to write FFI python files (when possible for a library) for most existing purescript libraries, problems will get solved.

However this approach is quite difficult, and an implicit drawback is, this way just aggravates the design/motivations for people to make new backends in the future, because they may come up with better solutions to unify the FFI use for all backends, and in this case they will suffer from worse backward compatibility issues.

Another idea occurred to me is, just drop the libraries that require JS FFI. This will be painful, because certainly a larget number of libraries will not be supported by purescript-python, and it seems extremely difficult to test if a purescript ibrary should be rejected by specific backends, e.g., purescript-python.

I guess I’m in trouble. Thanks for any suggestions.

[jy14898]

For cases where it’s possible, you could try and run a js -> python transpiler to get an initial implementation

For cases where it isn’t possible, you could automatically generate a crashing stub for each foreign function and have it crash on evaluation

Really I imagine in the future that modules either separate out their JS dependency into a different module, or each module has to provide .js .py … for each foreign function for it to qualify as usable in that language?

[thautwarm]

For cases where it’s possible, you could try and run a js -> python transpiler to get an initial implementation

This usually produces very inefficient code because of the distinct object model and Python’s lack of optimizations. I think it could be hard for people to accept the inefficiency of things like Js2Py.

For cases where it isn’t possible, you could automatically generate a crashing stub for each foreign function and have it crash on evaluation

Yep, this is similar to my point 2 aforementioned.

Really I imagine in the future that modules either separate out their JS dependency into a different module, or each module has to provide .js .py … for each foreign function for it to qualify as usable in that language?

This is what I think to be an approximately perfect approach, however this cannot be done by myself, I mean this needs a consensus among the whole community.

If PureScript offcial members could promote this style, and more people can get involved in providing FFI files for alternative backends, I believe the FFI problem can get solved.

Currently I’m trying to put the cross-backend FFI compatibility into the second priority.

I think the most important thing is showing the official members/core developers that purescript-python is an actively maintained backend, and after that we might be able to discuss about adapting your above proposal of dealing FFI files(.js, .go, .py, …).

[jy14898]

Thinking about it some more, ideally we move to a model where if something is foreign for performance reasons, then you supply both a PS-implemented version and a foreign implemented version, and some way of selecting the implementation at compile time.

If we didn’t want to add new syntax, we could introduce a ‘dynamic’ foreign importing system where

-- compiler solved instances
-- we can in theory optimize the use of optimize away
class ForeignOptimize foreignName where
    optimize :: forall a. SProxy foreignName -> a -> a

-- compiler implements these behind the scenes
instance foreignOptimize_func_fast :: ForeignOptimize "func_fast" where
    optimize _ a = {- compiler points this to the foreign js/py/go func -}

else instance foreignOptimize_no_foreign :: ForeignOptimize s where
    optimize _ a = a

func_slowPS :: ...
func_slowPS x y z = ...

func = optimize (SProxy :: _ "func_fast") func_slowPS

This is just how I might implement it using a class, in reality it could just be a new syntax which is equivalent in some sense to this

(I realise this impl doesn’t deal with the fact that different modules will have different instances of this class, but not really the main point)

[thautwarm]

Great point. If all developers could try to provide a purescript implementation if possible, things could be much easiler for a new backend!

Besides, for optional FFI implemenations, could we have this:

foreign import ffi_impl :: Maybe (...)
ps_impl = ...
func 
  | Just f <- ffi_impl = f
  | otherwise = ps_impl

[jy14898]

That works great too as long as the developer inserts the Nothing stubs into the backend-specific-foreign
(wouldn’t want to do that automatically unless we add a new keyword to foreign import)

[thautwarm]

(wouldn’t want to do that automatically unless we add a new keyword to foreign import)

foreign import ffiGetDefault ::
   forall a. String -> a -> a

ps_impl = ...
func = ffiGetDefault "func" ps_impl

[thautwarm]

It seems that purescript-native @andyarvanitis chose this approach by purescript-native-go-ffi:

I used to think if I can gather many people to write FFI python files (when possible for a library) for most existing purescript libraries, problems will get solved.

However this approach is quite difficult, and an implicit drawback is, this way just aggravates the design/motivations for people to make new backends in the future, because they may come up with better solutions to unify the FFI use for all backends, and in this case they will suffer from worse backward compatibility issues.