Avoid yarn for packages for now and fully enjoy its benefits for application development
--
tl;dr The new yarn package manager is a fantastic tool to install your application’s dependencies. When using it for packages, that are meant to be consumed by other packages and applications, things might not work as expected 💥
The npm CLI uses the opt-in feature shrinkwrap to allow it’s users to lock down the exact versions of the dependencies installed into the node_modules folder. A lot of people think this is a useful thing, so dependency installs become repeatable.
There are plenty of reasons to enable this for application code, but up until now the community agreed upon the idea that shrinkwrap shouldn’t be used for installable packages, because it would cause a lot of outdated and duplicated modules in the final node_modules folder.
Publish a shrinkwrapped package and the npm installer will take it into account
Here comes the important part however: If you still decide to publish a module with shrinkwrap enabled, the npm installer will take that into account. This means dependencies are not only locked during the development and publishing of the shrinkwrapped module, but also for the consumers. If you install that package in your application, then npm will make sure its node_modules folder looks exactly as defined in the npm-shrinkwrap.json file.
Yarn turns locking dependencies around — it’s the default way of how things are working. Whenever you add, remove or upgrade packages the exact dependency tree is written to a yarn.lock file. The yarn installer being able to take this file into account only, enables reliable, repeatable and even offline-only dependency installs — for everyone on any machine. This is awesome.
Yarn does not care about its dependencies’ lockfiles
There is a problem though. In contrast to the npm installer yarn does not care about neither its dependencies’ npm-shrinkwrap.json, nor yarn.lock files … and this is where things are getting confusing and dangerous.
Yarn resolves […] issues around versioning and non-determinism by using lockfiles and an install algorithm that is deterministic and reliable.
I was rather surprised to find out that yarn is not doing what is described in the quote above. When yarn is installing packages that used yarn themselves, or were shrinkwrapped, it is ignoring their lockfiles. By doing this it provides the installed packages a set of dependencies that might be completely different from what they defined. This isn’t exactly what I would call predictable and reliable.
Let’s look at an example. I created a demo package called @boennemann/yarn-test-module. It doesn’t do much, but it’s depending on another demo package called @boennemann/semver-fail-module. At the time I wrote the yarn-test-module and published its version 1.0.0 the semver-fail-module was at version 1.0.0 as well, which did nothing more than exporting true. I installed that module using yarn and wrote some tested code that’s depending on semver-fail-module as the value true.
I then published version 1.0.1 of the semver-fail-module, but this time it’s exporting false. The only purpose of this is to simulate a dependency that publishes a breaking change without increasing the major version — violating SemVer.
[S]emver relies on package developers not making mistakes — breaking changes or new bugs may find their way into installed dependencies if the dependencies are not locked down.
– https://code.facebook.com/posts/1840075619545360
After the new version was published I went back to my yarn-test-module, ran yarn, ran my tests and, as promised, everything was still fine. The yarn.lock file protected me from the SemVer violation of the semver-fail-module and everything was working as expected.
After that I moved on to create a third package, called @boennemann/yarn-test-consumer. Inside of that package I installed the yarn-test-module. As yarn ignores the lockfile of the yarn-test-module it did not come with the semver-fail-module@1.0.0 though, but the very problematic semver-fail-module@1.0.1.
With this behavior yarn is creating a very unintuitive and dangerous situation where it leads package authors to believe that their dependencies are locked, while in reality they are only locked during development. End users might always end up with a completely different set of dependencies. Detecting and debugging these problems will require constantly upgrading or resetting the yarn.lock file of the package, defeating its original purpose.
You may verify this behavior. Clone yarn-test-module, run yarn and npm test. Things will work. Clone the yarn-test-consumer, run yarn, execute node index.js and things will blow up.
One reason why npm is slower than yarn at installing dependencies is that it does take shrinkwrap files into account. In order to do this it has to look for the npm-shrinkwrap.json file, which is only stored inside of the package’s tarball. To fully resolve the dependency tree it can’t just look at the package.json and the metadata stored in the registry, to then download/copy packages in a fully separated and speed optimized step — something yarn is doing.
Yarn could change their installer to take yarn.lock files into account, but that would come with a performance hit, as described above, and a lot of different versions of the very same module. Additionally that would introduce an ecosystem divide, where only yarn can install yarn managed packages well, and only npm can install shrinkwrapped packages well.
As always, this is about tradeoffs. As much as npm would love to offer a speedy installer, ignoring shrinkwrap files isn’t something they could do, without breaking everything. Yarn does not come with 6 years of history — they can choose whatever tradeoffs they like. In this case it means heavily optimizing for speed and repeatable installs. As long as that works well for their use-cases I’m all for it. However it does break in other cases and I hope amidst all the excitement the community becomes aware of them.
For now I recommend to avoid using yarn to install package dependencies and to experiment with its benefits for application development. Note that installing shrinkwrapped packages like hapi might still break.
Edit: Yarn does offer a no-lockfile mode, which I previously didn’t realize and mention. If you’re using yarn for your packages, then please turn it on. It will take away many of the benefits, like speed, repeatability and offline-installs however.
Read my follow-up post: “Questions I wish Yarn had answered on day one”.
