Call for Participation: Help ES modules team test migration tools

Ecosystem
#1

If you have a PureScript project with non-trivial FFI, then we need your help!

The ES modules working group is evaluating tools to automatically migrate PureScript FFI from CommonJS modules to ES modules. We need help testing these tools on a larger set of PureScript projects than the working group has direct access to (ie. our own work code bases or open source libraries).

The Process

If you have PureScript code that uses the FFI, then we would like your help testing out tools for automatic migration. The general process is:

  1. Choose a tool and run it on your code base.
  2. If there are any errors, tell us about them here on Discourse!
  3. If there are no errors, take a look at the generated output to make sure it looks as you would expect. Verify that they did indeed work correctly and didnā€™t silently pass through bad output.
  4. Whether or not you ran into any issues, please tell us! We need reports that the tools worked just as much as we need reports that they didnā€™t.
  5. If youā€™re feeling especially motivated, try repeating the process with a different tool.

Some issues can be difficult to discover without compiling the ES modules code and then bundling the result. For example, if you previously exported a function called new as exports.new = function () { ... }, then lebab will transform it to export function new () { ... }, which is invalid code because new is a reserved keyword in JavaScript.

If you would like to be extra thorough, then you can take these additional steps:

  1. Download the es modules version of the compiler
  2. Compile your transformed source code
  3. Bundle the resulting output directory with esbuild, webpack, or parcel

If you were able to transform your code without issues, but then encountered one after compiling and bundling, itā€™s especially important that we can report on this to users!

The Tools

The best migration tool we have evaluated so far is lebab, and itā€™s the one weā€™d like everyone to test if they can test just one. You can use lebab to migrate ES5 code to ES6 code, but for the sake of testing weā€™ll only use the transform they provide for changing a CommonJS module to ES modules.

You can do just that transformation with one of the following commands:

# replace all *.js files in the src directory by rewriting them from
# commonjs modules to es modules
$ npx lebab --replace src --transform commonjs

# you can also provide glob patterns, if you would like
$ npx lebab --replace 'src/js/**/*.jsx' --transform commonjs

The CommonJS ā†’ ES modules transform is considered unsafe, because there are some edge cases the tool is unable to handle. These are issues worth checking for in your updated code:
https://github.com/lebab/lebab#unsafe-transforms

Other Tools

Another option you can try is cjstoesm, which has instructions on GitHub:
https://github.com/wessberg/cjstoesm

So far these are the only two tools weā€™ve tried ā€“ if you try others please let us know!

5 Likes
#2

Just to repeat our experience at Awake (I posted this in the proposal to disallow CJS):

  • We have an ES module in our main src tree that serves as an entry point for code splitting (dynamic import). These tools do not like to be handed files that are already ESM. They both just exited.
  • We are requireing ES modules in our FFI files (eg. JS React components in our non-PS src tree). The mapping from ESM to CJS is imperfect and cannot be accurately resolved without knowing the module you are importing is ESM. These tools basically just map syntax. So something like var wat = require('js/wat/index.js').default either fails or maps incorrectly to the equivalent of import $wat from 'js/wat/index.js'; const wat = $wat.default (ie, a double default dereference).

Iā€™d personally consider both of these issues with how we are mixing ESM/CJS, as they are both in dodgy territory, but if we are doing it then others might be as well.

3 Likes
#3

I get a CORS error from Dhall when I try to pull the package set listed on the project website.

#4

@paf311 Does this link work for you instead? If so, Iā€™ll update it on the site.
https://github.com/working-group-purescript-es/package-sets/releases/download/0.1.0/package-set.dhall

#5

I can try it on https://github.com/mikesol/purescript-wags/blob/main/src/WAGS/Interpret.js and report back!

1 Like
#6

I was able to get it to work by using spago to pull files from this package set, but Iā€™m not sure how to get spago to use the custom executable without clobbering the purs on my PATH already, so I ran purs-mac-x86_64 repl '.spago/*/*/src/**/*.purs' directly, which worked.

However, the REPL seems to be broken, since it doesnā€™t give any output when I evaluate expressions.

#7

Also, may I suggest changing the output file extension to .mjs so that generated code can be debugged using node.

Edit: one last question: I see a lot of ā€œexport varā€ in the FFI modules. Was that automatically converted, or is there another reason why itā€™s not using export const or export function?

#8

However, the REPL seems to be broken, since it doesnā€™t give any output when I evaluate expressions.

Also, may I suggest changing the output file extension to .mjs so that generated code can be debugged using node.

I havenā€™t worked on the ES modules PR myself, so Iā€™ll let others speak to these questions.

I see a lot of ā€œexport varā€ in the FFI modules. Was that automatically converted, or is there another reason why itā€™s not using export const or export function?

lebab will preserve usage of var unless you ask it to transform it via --transform let.

#9

Iā€™d expect if someone was using a tool, they might all get mapped to var since thatā€™s the semantics of CJS exports. I could see export function, but that would also require transforming you code from binding = function(){ .. } to function binding () {} which may change properties of your code in JS land (like the presence of a name property on the function object). I donā€™t think any tool is going to do a const conversion since that would require non-trivial analysis to ensure itā€™s safe.

#10

I think the branch is currently writing a package.json in output which sets modules to true, which means that node will interpret .js files as esm by default, and requires .cjs if you want require.

#11

lebab works well! No issues to report (my file is huge but tame at the end of the day, so it doesnā€™t really hit any corner cases).

3 Likes
#12

Iā€™m on windows atm and it looks like there arenā€™t available binaries for that yet. Are the commands for building the 0.15 branch up to date? I cloned that branch and built but the binary says 0.14 and spago wants a newer version for that package set.

1 Like
#13

I think you will need to overwrite the purescript version of the working group package set to 0.15.0-alpha-01, since this is the version in the current master but the working group package set hasnā€™t been updated yet.

1 Like
#14

Youā€™ll need to update the package set to the working-group-purescript-es's one for v0.15.0. Below is what Iā€™d do for Linux:

spago init --no-comments
cat << EOF > packages.dhall
let upstream =
      https://raw.githubusercontent.com/working-group-purescript-es/package-sets/main/packages.dhall
        sha256:bd4871410dc601d8d634ff1b09bbcdb048c8eb192b1cbbfc0f3271e92329d363

in  upstream
  with metadata.version = "v0.15.0-alpha-01"
EOF
spago install
spago build
#15

Ok thanks. Which branch on that fork? The default?

#16

you can use the master of GitHub - purescript/purescript: A strongly-typed language that compiles to JavaScript since it has already been merged

1 Like
#17

Oh, sorry, I thought you were talking about the package set, not purescript. We did make an alpha release of v0.15.0 (Release v0.15.0-alpha-01 Ā· purescript/purescript Ā· GitHub). You can use that if you donā€™t want to compile master from source though master has a few more merged PRs.

1 Like
#18

I didnā€™t have any issues updating react-basic-hooks, aff-promise, or assert. If these are on the right track Iā€™ll update react-basic, react-basic-classic, react-basic-dom, and react-basic-emotion.

1 Like
#19

The 0.15 migration script was run on parsing, so now Iā€™m trying it out with purs v0.15.0-alpha-02.

spago -x spago-dev.dhall install succeeds.

spago -x spago-dev.dhall build succeeds.

spago -x spago-dev.dhall test fails with this error. What could be going on here? There is no FFI in this package.

node:internal/modules/cjs/loader:1146
      throw err;
      ^

Error [ERR_REQUIRE_ESM]: require() of ES Module /home/jbrock/work/oth/purescript-parsing/output/Test.Main/index.js from /home/jbrock/work/oth/purescript-parsing/[eval] not supported.
Instead change the require of index.js in /home/jbrock/work/oth/purescript-parsing/[eval] to a dynamic import() which is available in all CommonJS modules.
    at [eval]:1:1
    at Script.runInThisContext (node:vm:129:12)
    at Object.runInThisContext (node:vm:305:38)
    at [eval]-wrapper:6:22 {
  code: 'ERR_REQUIRE_ESM'
}

Node.js v17.1.0
[error] Tests failed: exit code: 1

UPDATE

It looks like maybe I should be using spago master branch instead of the latest 20.7 releaseā€¦ Add support for es modules bundling using esbuild by sigma-andex Ā· Pull Request #862 Ā· purescript/spago Ā· GitHub

UPDATE

I tried the Wed Mar 23 spago master branch and I get the same error.

UPDATE

This works.

node --input-type=module --eval "import {main} from './output/Test.Main/index.js'; main();"

UPDATE

#20

You can also set "type": "module" in your package.json, which along with latest spago should be enough to run the tests.

1 Like