Install state file

[crubier]

What's the problem this PR addresses?

#973

How did you fix it?

A bit more brutally than in #990

Here we save the whole project state in one big file so there are no resolution to be done at run time.

[crubier]

@arcanis as discussed there #990 (comment)

I took a more brutal approach.

The interesting stuff is in Project.ts, in the generateInstallState and restoreInstallState functions, which simply serialize and restore the whole project state in a file.

However once again I am stuck. I thought this approach would be straightforward enough for me to manage it, but nope! It seems that restoring accessibleLocators , optionalBuilds , storedDescriptors , storedResolutions , storedPackages is not enough ? Or I dont do it right.

[arcanis]

When dumping the loaded content I see that null (the null) is turned into null (the string). It seems this is because we're loading the file through the "failsafe" YAML schema, which doesn't support null as a value (TIL). Try storing it as JSON to see if that goes better.

[arcanis]

Also of note: the file is 10M for our repository, I can't imagine what it will be for yours 😬 Maybe some basic format packing optimizations would be a good idea (I think ideally it shouldn't be much more than 1MB for our repo, given that the lockfile is 800KB).

[arcanis]

is there a way to speed up the setupResolutions using restoreInstallState?

The time in Project.find is likely the one needed to find the configuration, the workspaces, and read their manifests - I'm not sure there's a low-hanging fruit here (at the very least, it should be part of a separate PR).

Implement the check on the checksum of yarn.lock to invalidate the cache if the checksum don't match => @arcanis is there a yarn official way to do this or should I roll my own way to get the checksums?

There isn't at the moment, but I'm fine if you add a private field to Project to store the checksum of the lockfile when it's loaded in setupResolutions. Oh, and don't forget to hash a constant as well that so that we can force-refresh the cache on future releases with a different cache format.

I kept the skipVirtualResolution parameter that you made in #990 => @arcanis is it still needed in this MR or should I remove these changes ?

Not needed anymore, let's remove it.

Check what I did already in this MR (the zlib and v8.serialize) => @arcanis did I do it right, or is there a more yarn-kosher way to do what I did?

That seems about right. The code is formatted very differently from the rest of the app so I'll likely make a pass to uniformize it, but the logic itself looks fine.

[crubier]

Ok, finished!

I now write/control the checksum of the lockfile + installStateFileVersion before writing/reading the install state file. It adds 1-3ms depending on the project, so nothing much.

It looks like this

this.lockFileChecksum = hashUtils.makeHash(`${INSTALLSTATEFILE_VERSION}`,lockfileContent);

Let me know what you think @arcanis , change the code style if you like, and let's get this merged!

[crubier]

Results of the stress test: It's very good! It looks like this:

Which is much better than it was before with #990 : (note the different vertical axis):

[crubier]

I had a look at the branch, it looks good to me

[arcanis]

🎉🎉🎉

Thanks @crubier!

[crubier]

Thank you so much @arcanis for supporting me through this! My first contribution to yarn, and a lifesaver for my company's tech team, reducing our install and build times by a lot! Best 🎉