Reducing incremental rebuild time with large file dependency

Help
#1

I’m noticing slow incremental rebuilds in a project containing a large Tailwind.purs file. If I make a small edit to Example.purs (which includes Tailwind.purs), rebuilds take about 4 seconds. I’m wondering if there are some additional optimization opportunities where the unchanged contents of Tailwind.purs could be remembered between rebuilds.

Setup:

git clone --branch tailwind-purs https://github.com/milesfrain/halogen.git myApp
cd myApp
npm install -g purescript spago
# or if you prefer local installation:
# npm install
spago build
# slow initial build expected

Slow incremental rebuild (with or without edit to small Example.purs):

time spago build
[info] Installation complete.
[info] Build succeeded.
spago build  2.15s user 0.54s system 273% cpu 0.985 total

echo " " >> src/Example.purs

time spago build            
[info] Installation complete.
Compiling Example
Compiling Main
[info] Build succeeded.
spago build  5.11s user 1.04s system 163% cpu 3.760 total
1 Like
#2

This is probably due to parsing externs too greedily. With the old JSON externs format this was necessary to get the compiler version, but with the new CBOR format we can get the version without parsing the whole file. We shouldn’t need to parse externs unless demanded.

#3

In this case, we still need to parse the whole externs for the Tailwind file, right? Since Example has changed, and Example depends on Tailwind?

#4

That’s a good point, yes. If it depends on tailwind, then it’s going to have to parse the externs regardless. @milesfrain What version of the compiler are you using?

1 Like
#5

Using 0.13.8, which is at least 2x faster

1 Like
#6

Is this an accurate summary of the issue?:

  • If a file changes, then each imported file/module must be re-parsed.
  • Since the Example module imports the Tailwind module, then Tailwind.purs must be re-parsed whenever Example.purs changes.

Are there any suggestions of things I can try to improve rebuild times?

Could the compiler cache what it learns about Tailwind.purs after the first parse, so it can skip re-parsing?

1 Like
#7

No, not quite. It’s not the source file which is being re-parsed, it’s the externs file. The compiler is already caching what it learns about Tailwind.purs, and the externs file is that cache; it contains information about the module’s interface, which is necessary for type-checking downstream modules. Parsing externs is unavoidable if a module which imports it has changed. If you’re looking for ways to speed things up, then the first thing to do is build a compiler with profiling enabled and see if anything jumps out.

#8

Are there instructions for building the compiler with profiling enabled?
I read through these guides, but didn’t see anything related to profiling:

I believe the answer might be somewhere in here, but I’m not sure whether I need the --work-dir flag. I recall that was mentioned before, but that’s supposedly unnecessary with stack 2.x (I have 2.3.1 installed).

#9

No, it’s not really any different from any other Stack project. I think just stack build --profile followed by stack exec purs ... should do it. I find stack exec bash useful in order to set up a new shell with the compiler I’ve just built at the front of my PATH.

#10

Edit:

exec also needs the --profile flag:

stack exec --profile bash

Original:

I don’t know if I’m missing something. The path it adds is missing bin/purs.

$ stack build --profile
$ stack exec bash
$ which purs
/home/miles/.nvm/versions/node/v13.12.0/bin/purs
$ echo $PATH
/home/miles/temp/tt/complier/purescript-0.13.8/.stack-work/install/x86_64-linux-tinfo6/9a90079476fb9497fec8dcb8f3a5d3c3d320d86ce5c6b30f16e0e19b8c727128/8.6.5/bin:<remaining paths>
$ ls /home/miles/temp/tt/complier/purescript-0.13.8/.stack-work/install/x86_64-linux-tinfo6/9a90079476fb9497fec8dcb8f3a5d3c3d320d86ce5c6b30f16e0e19b8c727128/8.6.5/
pkgdb
#11

Should this work?:

$ spago build --purs-args +RTS -p                                                
spago: the flag -p requires the program to be built with -prof

This works and produces a purs.prof file:

$ purs --version +RTS -p         
0.13.8 [development build; commit: 9cad73ed8ea7df3011032ddbd2f3de5a0c08629c DIRTY]

The version that spago picks up appears to be the same:

$ spago build --purs-args --version        
[info] Installation complete.
0.13.8 [development build; commit: 9cad73ed8ea7df3011032ddbd2f3de5a0c08629c DIRTY]
[info] Build succeeded.
#12

I think here you’re passing the RTS arguments to Spago.
This will work to pass the profiling flag to purs instead:

spago build --purs-args "+RTS -p -RTS"

(note that we need to close the RTS because spago will have to pass the globs after our flag)

1 Like
#13

Geeze. I always forget those quotes! If only I updated to the latest-and-greatest spago v0.15.3 (released the day before) which has a note about this in the help text.

Profiling led straight to the issue. Now 2.5x faster with this fix.

I also had a pretty gnarly time getting my Haskell environment setup for this project, so I documented the steps that ended up working for me to help future users have a smoother experience.

3 Likes