Upcoming changes to error messages for large records/rows

Hello all,

In this year’s PureScript Community Survey, the pull request that received the most interest was #4421, an effort to improve the error messages caused by large records or rows. We are almost ready to merge this PR, but its original author is no longer taking an active role in giving feedback. We’d like any community members who have experienced frustration with the error messages associated with large records or rows—according to the survey, there are a lot of you!—to drop in and give the PR a try with some of their code to see if there are regressions or important opportunities we missed.

5 Likes

I’ll try to have a look at these errors in the coming days!

3 Likes

How do we try this? Do we have to build PureScript ourselves?
Could this get merged and we can try the @next version and worst case it can be patched out again?

At the moment, yes, the PR would need to be built.

Forgive the mini-rant, but as a volunteer team with finite time, we try to put our effort into doing things from which the community gets value. The survey is a good tool for learning what those things are, but costly signals are often more honest than clicking a votey button. This particular feature shakes a few things up inside the compiler and while I think they’re good shakes, I don’t want to do it for nothing—I want the community to be able to summon at least one person willing to put in the relatively minor effort of building PureScript and testing it on a project for which these changes would be relevant. Not picking on you individually—maybe you’re not the right person for this, and that’s fine!—but if the community can’t produce one single person to do this, that’s a more honest signal than the survey that this change isn’t that important after all, and then I don’t want to merge it.

3 Likes

I just tried it out and there’s no difference for my use case.

My usecase is react-basic and functions that look like:

type Allowed = (some :: Int, big :: String, row :: Number)
div :: forall given missing. Union given missing Allowed => {|given} -> JSX
div = ...

An example with a simple typo (valu instead of value) has unfortunately not improved.
I tried a contrived example with a fixed record, and that looked a bit better but it’s not my usecase.

The excerpt of the error message makes my message too long for discourse’s allowed 32000 characters so here’s a link to a gist:

1 Like

Ok, what would be a “better” error in this situation?

With a “normal” record use case, I do get a message that looks nicer to me:

  Expected type

    { onYes :: Effect Unit
    ...
    }

  but found type

    { onYess :: Effect Unit
    ...
    }

But used with HTML DSL, similar to @i-am-the-slime 's use case, I get a similar error message, except in my case it also involves some extra class resolutions.

If I try to think what would be more helpful to me in this Union situation, I think the same huge error message is fine (for completeness), but what would actually help fix my code is some edit-distance suggestions. Like, "valu" is not one of expected fields, did you mean "value"?

I understand this can’t be done in general, because it depends on the intent of Union, and it doesn’t always mean the same thing. But maybe a special-case pattern can be identified? Or, perhaps, a new specialized class could be employed - e.g. Subset subRow superRow?

3 Likes

Thanks folks, that’s exactly the kind of thing I was worried we’d miss.

2 Likes

A better error would use the type alias in the original definition and then suggest keys with a low Levenshtein distance.
From what I can tell the compiler has a few ways of solving the Union constraint based on which types are fixed. In the pattern that I use the most and that is very common in JavaScript in general it’s really always a subset so the third argument of Union is fixed.
I have no idea about the phases of typechecking and all that jazz but here’s what I think would make a great error message:

There's no key named `valu` in `Props_div` .
Did you mean `value`?

I just got a better error message but it’s still pretty hard to spot:

[2/2 PropertyIsMissing] src/DB/Context/Prod.purs:38:3

        v
  38    { signInWithOTP
  39    , onAuthStateChange
  40    , getSession
      ...
  47    , loadDocument: loadDocument client
  48    , registerPersistenceProvider: registerPersistenceProvider client
  49    }
        ^

  Type of expression lacks required label deleteDocument.

  while checking that expression { signInWithOTP: signInWithOTP
                                 , onAuthStateChange: onAuthStateChange
                                 , getSession: getSession
                                 , signOut: signOut
                                 , insertDocument: insertDocument
                                 , upsertDocument: upsertDocument
                                 , updateDocumentName: updateDocumentName
                                 , loadDocuments: loadDocuments
                                 , loadDocumentList: loadDocumentList
                                 , loadDocument: loadDocument client
                                 , registerPersistenceProvider: registerPersistenceProvider client
                                 }
    has type { deleteDocument :: { id :: DocumentID
                                 , user_id :: String
                                 }
                                 -> Om (Record ...)
                                      ( deleteDocumentError :: ...
                                      )
                                      Unit
             , getSession :: Om (Record (() @Type))
                               ( failedToGetSession :: { code :: ...
                                                       , details :: ...
                                                       , message :: String
                                                       }
                               )
                               (Maybe
                                  { user :: ...
                                  }
                               )
             , insertDocument :: { name :: DocumentName
                                 , user_id :: String
                                 }
                                 -> Om (Record ...)
                                      ( insertDocumentError :: ...
                                      )
                                      { id :: DocumentID
                                      }
             , loadDocument :: { id :: DocumentID
                               , user_id :: String
                               }
                               -> Om (Record ...)
                                    ( loadDocumentError :: ...
                                    )
                                    (Maybe ...)
             , loadDocumentList :: { from :: Int
                                   , to :: Int
                                   , user_id :: String
                                   }
                                   -> Om (Record ...)
                                        ( loadDocumentError :: ...
                                        )
                                        (Array ...)
             , loadDocuments :: { user_id :: String
                                }
                                -> Om (Record ...)
                                     ( loadDocumentError :: ...
                                     )
                                     (Array ...)
             , onAuthStateChange :: (Maybe ... -> Effect Unit) -> Effect (Effect Unit)
             , registerPersistenceProvider :: YDoc
                                              -> { email :: String
                                                 , id :: String
                                                 , user_metadata :: Foreign
                                                 }
                                                 -> Effect Provider
             , signInWithOTP :: String
                                -> Om (Record ...)
                                     ( authError :: Error
                                     )
                                     Unit
             , signOut :: Aff Unit
             , updateDocumentName :: { id :: DocumentID
                                     , name :: DocumentName
                                     , user_id :: String
                                     }
                                     -> Om (Record ...)
                                          ( updateDocumentNameError :: ...
                                          )
                                          Unit
             , upsertDocument :: { document_changes :: EncodedStateUpdate
                                 , id :: Maybe DocumentID
                                 , user_id :: String
                                 }
                                 -> Om (Record ...)
                                      ( upsertDocumentError :: ...
                                      )
                                      { id :: DocumentID
                                      }
             }
  in value declaration prodContext

Everything from while checking that expression onwards seems like verbose error output to me that mostly intimidates me (and potentially beginners) and I’d love to see hidden behind a -v flag.

Let’s say I had this code:

type Foo = { a :: Int, b :: Int, c :: Int }

bar :: Foo -> Int
bar = _.b

test = bar { aa: 2 }

What should the error message be for this? Based on…

A better error would use the type alias in the original definition and then suggest keys with a low Levenshtein distance.

… it seems the error should be:

Could not match the rows in the type alias

   Foo

with the rows

   ( aa :: Int
   | t0
   )

Perhaps these labels are misspelled?
   - `aa` might be `a`: Levenshtein distance of 1

Yes, that looks very good!
I’m not a fan of the “Could not” style of error messages where the compiler is speaking and using auxiliary verbs, but I suppose that’d be a more pervasive change.
I would refrain from mentioning the concept of Levenshtein distance but I like the
aa might be a part!

I’d try with something that keeps the number of fixed words in the message very low:

Key `aa` does not appear in `Foo`.

Did you mean
  - `a`
?
3 Likes

Ah… good point. There is a 1-to-many relationship between the label provided by the user and the ones found in the row.

Also, because the code is desugared before it’s typechecked, we don’t know that a given row has its source in some type alias Foo. My error above is what would be possible if this order was reversed.

There’s another problem with my proposed error in terms of where that label information appears. If the issue is caused by misspelling, one would still need to scroll past a bunch of rows before seeing the “Did you mean label X?” message. If the error is caused by label differences, then that’s the first action a user can take: fix the spelling mistake and before moving on to possible type errors. In that case, I’d rather see that at the top of the message rather than below it. For example:

Could not unify two rows.

   Label distances:
      Found 'label' but perhaps you meant
        - label1
        - label2
        - ...
        - label10

  Unification error:

    Could not unify 
      X
    with
      Y

… where the ‘could not unify part’ is what we currently have in the PR.

However, if the error is not caused by misspelling, then maybe putting it at the top is just noise in the error message.

Regardless, one issue with the above error is each possible label gets its own line. That’s clearer in terms of readability, but leads to a similar issue where one needs to scroll through that entire content before seeing the type unification error. Sometimes, seeing which types are associated with a label is helpful for determining which part of your code needs to be changed.

We could workaround the problem in this way. First, we determine a threshold to use for which labels are included in the distance message (e.g. a distance of 3 or less). Second, we determine how many labels to show on a single line for readability before comma-separating the rest on a single line. The example below shows a possible error with the decision of a “distance <=3” and “single-line-label-count of 2”:

   Label distances:
      Found 'label' but perhaps you meant
        - label1
        - label2
        - ... or 4 other labels: `label3`, `label4`, `label5`, `label12`

I also think it would be helpful if we could print the edits needed to get the correct label. Sometimes a typo is switching two characters around that isn’t obvious when such characters are similar (e.g. i, l, I, and 1). How long does it take you to spot the typo?:

lI1lIl1ililIil
lI1lIl1illiIil

All that being said, at what point is this too verbose for a single error?

1 Like

It all sounds very good. I would have a hardcoded limit of Levenshtein (there surely is a better metric for this but that’s the only one I know of). It’s very unlikely that something is a spelling mistake if it’s off by your suggested 3 and just listing all possible label is just noise. The user should then check the actual type (alias) by themselves by navigating to the type in question.

Also, because the code is desugared before it’s typechecked, we don’t know that a given row has its source in some type alias Foo .

This is what I thought was the source of all the bad error messages in the first place and why this problem hasn’t been addressed thus far. Isn’t this also the root cause for bad error reporting in do blocks?
It might be that somehow keeping more info about the original source is a prerequisite for this, but of course I don’t know.

As if ‘bad’ is something that has only one source…

No, these error messages are all influenced by dozens of little details where the code in the compiler makes what locally seems like a very reasonable choice, but which lead in aggregate to results that users don’t like for reasons about which they are often not explicit. I wish there were a single makeErrorMessagesBad function I could just rewrite or remove.

As Jordan said, we desugar type synonyms quite early and it would take a fair amount of effort (I think?) to rearchitect the type checker so that type synonym resugaring was available when reporting type checking errors. Doesn’t seem like that’s the only thing you think is bad about these error messages though! The more specific we can be about the things that are easier to achieve, the more progress we can make.

About do blocks, my understanding is that those errors could be improved if we changed the order in which we traversed expressions during type checking; I didn’t think type synonyms had anything to do with it but again this might be a situation where different users think the singular ‘bad’ thing are entirely separate issues from each other.

1 Like

I apologise for my messages.

Oh, you’ve nothing to apologize for. I wouldn’t expect someone who isn’t elbow-deep in the compiler to just know this stuff; that’s why I’m laying it out. Any frustration in my tone isn’t directed at you. These are just thorny problems with no clear specification for what ‘correct’ is, either in approach or in final result. But we can iterate towards ‘more correct’, and your input is valuable for that process.

1 Like

I’m glad to see some work done here, I think the error messages related to rows are ripe for improvement. If we are to simply discuss opinions about error messages I have a few, I’ll let you lot decide their validity and relevance. If anyone cares for more opinions on error messages send me an email or write to me on Discord - I unfortunately don’t hang around GitHub issues as much as of late.

But first some personal option. When I read an error message I usually want it to go away - this means I try to read as little of it as possible. If I see references to the inner workings of the compiler I usually try to gloss over it which is what happened at first when row-types started floating about. This means I value type errors that are legible and have little unnecessary information - which is unfortunately not always the case for row-type errors. These guidelines are of course very vague, and I suspect people will disagree on these points. But to examples are usually illumination. (1) I’ve never found the t64 is an unknown type variable helpful at all; the same goes for the desugared expression - this noise to me. (2) If I can fit more errors in one terminal window I can usually fix them faster - but unfortunately type errors have a tendency to cascade in PureScript; errors hide other errors further down the line which will probably force me to recompile to find them. (3) I want the information in the error messages to be relevant - if you point to a file or line - it should be the place where I can fix my error or figure out how to fix it.

(Already this points to two another solution to the problem of bad error messages: a faster compiler or better error cascading. Both of these solutions will aid other parts as well, not just row-types.)

I’ve been writing PureScript daily now for well over a year, since I am one of few people who work with this language, and I’ve seen a lot of very specific error messages. This is a few of the ones I’ve found that might be of interest. Unfortunately I’m not allowed to share explicit snippets since the code is in a proprietary codebase. These errors I’ve managed to dig up are the ones that are remembered, and that I could draw from the top of my head this Tuesday night - they are not representative of the frequency I find these errors. We also use a coding style where we use a lot of rows in our row-types (around 500 rows all with fairly large types inside them). we were thus forced to fork the compiler and it has saved us days of scrolling through the terminal (I am not kidding with days). This is also the reason for the PR being left alone for the better part of a year - I solved my urgent trouble by working on a fork of the PureScript compiler.

  1. Row unification showing rows that matches, this has resulted in multiple errors which are around 152Kb, this clogs up the terminal and makes it almost impossible to find the error messages. Here is a zoomed out screen shot of one of these errors. The worst part about them is that the type is often written twice for both sides, which is noisy and unnecessary. The screen shot is only of parts of the error message and the red box is the part of it that was actually actionable.

  2. If there are errors inside a rowtype with the same name (i.e List imported from say Erlang.List and Data.List), the unification fails and complains about the rows with an error claiming the types are identical. This might be a case of the desuggering, but the row type is relevant here and is strictly not needed.

  3. When I get a type error it was inside a record, it used to not report what row was relevant, I think (Mention which row label the type error occurs on by FredTheDino · Pull Request #4411 · purescript/purescript · GitHub) solved that for most cases - but maybe it should be considered here.

  4. It wasn’t I who got this message but a colleague got this (and I sadly don’t have the code to reproduce it) - also modified to anonymize.

[1/1 TypesDoNotUnify] Code.purs:204:15

  204              ( input # Api.parse
                     ^^^^^^^^^^^^^^^^^

  Could not match type

    E

  with type

    Q

  while trying to match type
                               ( llllllll :: String
                               , kkkkkkkk :: Array String
                               , jjjjjjjj :: Int
                               , hhhhhhhh :: String
                               , id :: E
                               , gggggggg :: Q
                               , ffffffff :: Int
                               ...
                               )

    with type
                ( ppppppppp :: String
                , qqqqqqqqq :: Array String
                , id :: Q
                , wwwwwwwww :: String
                , eeeeeeeee :: String
                , rrrrrrrrr :: String
                ...
                )

I am guessing there were some fields that matched but the records have little in-common when you look at this error.

Of course these are not all the error messages all the time, but they are definitely a source of frustration.

Those are the ones relevant to row errors that I can think of from the top of my head. If one wants to hunt other exotic errors that are quite uninitiative here is a lightning round:

a. If you use a type in a class-declaration without exporting it - the place where the class is instanced throws an error about the class not being imported. This might point to a different file completely, and the type that is not exported is not mentioned.

b. do-blocks as mentioned earlier is a problem if you mix different monads or if you write the wrong return type, since it usually points to a line in the middle which is confusing.

c. The exhaustiveness checker gives up quite eagerly - which forces you to rewrite code in strange ways to please it. For example if you want to case on a large data-enum with a lot of members you are out of luck and have to restructure your case-expressions.

d. Circular dependencies are a nightmare to fix, I audibly groan every single time I get that error. Most of the modules are not relevant to the actual dependency cycle and you end up going on a wild goes chase unless you really know where you’re going with the current change.

But these are just like, opinons, man. <3

3 Likes

Please don’t assume everyone is in discourse. I didn’t know there was a discourse until today.

I’ve forked and maintained GitHub - drathier/purserl to fix issues like these which were never merged into main. We’ve reduced compilation times by about 10x on this fork, in a 120ksloc code base.

FredTheDino, Pwnxl and I tried to get some improvements in a year ago, but they were never merged. The feedback we got in private was that forking the compiler and/or building a new compiler from scratch was a better way forward. We stopped submitting PR’s because why bother when it never gets merged?

The core team does a lot of code review at work, and doesn’t want to do the same in their free time, which makes perfect sense. However, it means that nobody new can join the core team to help review new PR’s.

We would’ve continued improving non-sexy stuff like error messages and compilation speed if we got things merged. I’m sure other coworkers would’ve joined in on the fun if it was.

2 Likes