Call for Participation: Help ES modules team test migration tools

If you have a PureScript project with non-trivial FFI, then we need your help!

The ES modules working group is evaluating tools to automatically migrate PureScript FFI from CommonJS modules to ES modules. We need help testing these tools on a larger set of PureScript projects than the working group has direct access to (ie. our own work code bases or open source libraries).

The Process

If you have PureScript code that uses the FFI, then we would like your help testing out tools for automatic migration. The general process is:

  1. Choose a tool and run it on your code base.
  2. If there are any errors, tell us about them here on Discourse!
  3. If there are no errors, take a look at the generated output to make sure it looks as you would expect. Verify that they did indeed work correctly and didn’t silently pass through bad output.
  4. Whether or not you ran into any issues, please tell us! We need reports that the tools worked just as much as we need reports that they didn’t.
  5. If you’re feeling especially motivated, try repeating the process with a different tool.

Some issues can be difficult to discover without compiling the ES modules code and then bundling the result. For example, if you previously exported a function called new as exports.new = function () { ... }, then lebab will transform it to export function new () { ... }, which is invalid code because new is a reserved keyword in JavaScript.

If you would like to be extra thorough, then you can take these additional steps:

  1. Download the es modules version of the compiler
  2. Compile your transformed source code
  3. Bundle the resulting output directory with esbuild, webpack, or parcel

If you were able to transform your code without issues, but then encountered one after compiling and bundling, it’s especially important that we can report on this to users!

The Tools

The best migration tool we have evaluated so far is lebab, and it’s the one we’d like everyone to test if they can test just one. You can use lebab to migrate ES5 code to ES6 code, but for the sake of testing we’ll only use the transform they provide for changing a CommonJS module to ES modules.

You can do just that transformation with one of the following commands:

# replace all *.js files in the src directory by rewriting them from
# commonjs modules to es modules
$ npx lebab --replace src --transform commonjs

# you can also provide glob patterns, if you would like
$ npx lebab --replace 'src/js/**/*.jsx' --transform commonjs

The CommonJS → ES modules transform is considered unsafe, because there are some edge cases the tool is unable to handle. These are issues worth checking for in your updated code:
https://github.com/lebab/lebab#unsafe-transforms

Other Tools

Another option you can try is cjstoesm, which has instructions on GitHub:
https://github.com/wessberg/cjstoesm

So far these are the only two tools we’ve tried – if you try others please let us know!

5 Likes

Just to repeat our experience at Awake (I posted this in the proposal to disallow CJS):

  • We have an ES module in our main src tree that serves as an entry point for code splitting (dynamic import). These tools do not like to be handed files that are already ESM. They both just exited.
  • We are requireing ES modules in our FFI files (eg. JS React components in our non-PS src tree). The mapping from ESM to CJS is imperfect and cannot be accurately resolved without knowing the module you are importing is ESM. These tools basically just map syntax. So something like var wat = require('js/wat/index.js').default either fails or maps incorrectly to the equivalent of import $wat from 'js/wat/index.js'; const wat = $wat.default (ie, a double default dereference).

I’d personally consider both of these issues with how we are mixing ESM/CJS, as they are both in dodgy territory, but if we are doing it then others might be as well.

3 Likes

I get a CORS error from Dhall when I try to pull the package set listed on the project website.

@paf311 Does this link work for you instead? If so, I’ll update it on the site.
https://github.com/working-group-purescript-es/package-sets/releases/download/0.1.0/package-set.dhall

I can try it on https://github.com/mikesol/purescript-wags/blob/main/src/WAGS/Interpret.js and report back!

1 Like

I was able to get it to work by using spago to pull files from this package set, but I’m not sure how to get spago to use the custom executable without clobbering the purs on my PATH already, so I ran purs-mac-x86_64 repl '.spago/*/*/src/**/*.purs' directly, which worked.

However, the REPL seems to be broken, since it doesn’t give any output when I evaluate expressions.

Also, may I suggest changing the output file extension to .mjs so that generated code can be debugged using node.

Edit: one last question: I see a lot of “export var” in the FFI modules. Was that automatically converted, or is there another reason why it’s not using export const or export function?

However, the REPL seems to be broken, since it doesn’t give any output when I evaluate expressions.

Also, may I suggest changing the output file extension to .mjs so that generated code can be debugged using node.

I haven’t worked on the ES modules PR myself, so I’ll let others speak to these questions.

I see a lot of “export var” in the FFI modules. Was that automatically converted, or is there another reason why it’s not using export const or export function?

lebab will preserve usage of var unless you ask it to transform it via --transform let.

I’d expect if someone was using a tool, they might all get mapped to var since that’s the semantics of CJS exports. I could see export function, but that would also require transforming you code from binding = function(){ .. } to function binding () {} which may change properties of your code in JS land (like the presence of a name property on the function object). I don’t think any tool is going to do a const conversion since that would require non-trivial analysis to ensure it’s safe.

I think the branch is currently writing a package.json in output which sets modules to true, which means that node will interpret .js files as esm by default, and requires .cjs if you want require.

lebab works well! No issues to report (my file is huge but tame at the end of the day, so it doesn’t really hit any corner cases).

3 Likes

I’m on windows atm and it looks like there aren’t available binaries for that yet. Are the commands for building the 0.15 branch up to date? I cloned that branch and built but the binary says 0.14 and spago wants a newer version for that package set.

1 Like

I think you will need to overwrite the purescript version of the working group package set to 0.15.0-alpha-01, since this is the version in the current master but the working group package set hasn’t been updated yet.

1 Like

You’ll need to update the package set to the working-group-purescript-es's one for v0.15.0. Below is what I’d do for Linux:

spago init --no-comments
cat << EOF > packages.dhall
let upstream =
      https://raw.githubusercontent.com/working-group-purescript-es/package-sets/main/packages.dhall
        sha256:bd4871410dc601d8d634ff1b09bbcdb048c8eb192b1cbbfc0f3271e92329d363

in  upstream
  with metadata.version = "v0.15.0-alpha-01"
EOF
spago install
spago build

Ok thanks. Which branch on that fork? The default?

you can use the master of GitHub - purescript/purescript: A strongly-typed language that compiles to JavaScript since it has already been merged

1 Like

Oh, sorry, I thought you were talking about the package set, not purescript. We did make an alpha release of v0.15.0 (Release v0.15.0-alpha-01 · purescript/purescript · GitHub). You can use that if you don’t want to compile master from source though master has a few more merged PRs.

1 Like

I didn’t have any issues updating react-basic-hooks, aff-promise, or assert. If these are on the right track I’ll update react-basic, react-basic-classic, react-basic-dom, and react-basic-emotion.

1 Like

The 0.15 migration script was run on parsing, so now I’m trying it out with purs v0.15.0-alpha-02.

spago -x spago-dev.dhall install succeeds.

spago -x spago-dev.dhall build succeeds.

spago -x spago-dev.dhall test fails with this error. What could be going on here? There is no FFI in this package.

node:internal/modules/cjs/loader:1146
      throw err;
      ^

Error [ERR_REQUIRE_ESM]: require() of ES Module /home/jbrock/work/oth/purescript-parsing/output/Test.Main/index.js from /home/jbrock/work/oth/purescript-parsing/[eval] not supported.
Instead change the require of index.js in /home/jbrock/work/oth/purescript-parsing/[eval] to a dynamic import() which is available in all CommonJS modules.
    at [eval]:1:1
    at Script.runInThisContext (node:vm:129:12)
    at Object.runInThisContext (node:vm:305:38)
    at [eval]-wrapper:6:22 {
  code: 'ERR_REQUIRE_ESM'
}

Node.js v17.1.0
[error] Tests failed: exit code: 1

UPDATE

It looks like maybe I should be using spago master branch instead of the latest 20.7 release… Add support for es modules bundling using esbuild by sigma-andex · Pull Request #862 · purescript/spago · GitHub

UPDATE

I tried the Wed Mar 23 spago master branch and I get the same error.

UPDATE

This works.

node --input-type=module --eval "import {main} from './output/Test.Main/index.js'; main();"

UPDATE

You can also set "type": "module" in your package.json, which along with latest spago should be enough to run the tests.

1 Like