elm-tooling with Simon Lydell

elm-tooling with Simon Lydell

elm-tooling helps you manage versions of tools like elm-format and elm. It downloads them more efficiently and securely. We discuss with the author, Simon Lydell.

PublishedJanuary 18, 2021

Episode#22

Simon Lydell (twitter) (github)
elm-tooling-cli
elm-json
Install elm-tooling into your npm dev dependencies
npx
elm-publish-action
NPM's package.json scripts section
NPM postinstall scripts
Richard's recommendation to use npm config set ignore-scripts true, which can cause issues because it also skips the postinstall from your package.json
elm-tooling-cli docs website

Elm tooling in ci

GitHub Actions
Simon's example GitHub Actions workflow with elm-tooling
Separate steps for Elm tooling install and npm install to optimize caching
npm run --silent (or -s) to reduce noise (more ways to do it in this StackOverflow answer)
npm test and npm start to run start and test from your package.json scripts
elm-tooling quick start page

Transcript

Hello Jeroen. Hello Dillon. And today we've got another special guest to greet. Simon,

hello, welcome to the show. Hello, thanks for letting me be on the show. Our pleasure. So for

those who don't know you, this is Simon Lydell. And do you want to give a quick introduction and

tell us who you are and a little bit about what you do? Yes, so I'm Simon and I'm from Sweden and

I'm a developer and I love Elm. Awesome. That sounds very good to us. And you've got a tool,

a meta tool. So, you know, that's near and dear to our hearts. Jeroen and I love tools for helping

you use tools. So you've got a tool called Elm Tooling CLI. And so, well, let's get into it. So

what the heck is Elm Tooling CLI? Like what actually is it? Let's define what it is before

we dive into the details of how to use it. Yeah. So Elm Tooling is a command line program that

manages your tools. It's a drop in replacement for installing Elm, Elm format and Elm JSON using NPM,

but it does it in a faster and more secure way. That does make sense. Although the more secure

part was not on my radar. What do you mean by that? So if you've ever installed Elm or Elm

format with NPM, you might know that you don't actually install Elm. You just install some

JavaScript code that eventually downloads Elm for you. That code doesn't actually verify what you

downloaded. It just downloads a binary from a URL and hopes that it's what you thought it was. So

Elm Tooling also downloads the same URLs, but it has SHA256 hashes for everything. So it should be

able to verify that you got the Elm binary and not some Elm binary with a virus in it or whatever.

That's very interesting. And it also is able to cache the binaries in a way like if you have Elm

in your dev dependencies in your package.json in an Elm project, which is a common practice. You've

got Elm, Elm format, Elm test all in your package.json dev dependencies. Whenever you do NPM install,

it's going to go download those, whether it's installed that exact Elm binary version, Elm

format binary version in a different project that you NPM installed. Is that right?

Yeah, exactly. So the point of installing things with NPM locally is that you want versions for

every project, but it feels very wasteful to me to have a copy of the entire Elm executable over

and over on your hard drive. So what Elm Tooling does is that it saves all of the executables in

one place in your home folder and only links to them in each project. This way you save a lot of

the space. It saves them in the Elm home directory, right? Wherever that is.

Yeah, exactly. So the Elm home directory is usually in your home folder and called.elm.

And the Elm compiler saves all the Elm packages that it downloads in there. And I know that

another tool called Elm JSON also saves a couple of things in there. So Elm Tooling follows that

pattern and also saves the Elm binary and the Elm format binary and stuff like that in there.

I think Elm Review also downloads things in there, but I don't remember actually. But yeah,

that's usually where any Elm data goes.

Right. And then if you're caching that in your CI process, however, whether it's Netlify or

custom CI or whatever it is, if you cache that directory, which you should on your CI,

then it's going to work. You don't have to have specific caching rules for these specific tools.

Yeah, that's one of the things I liked with that approach. I'm thinking that people are

already caching that directory, so they don't need any extra work if they want to use Elm Tooling.

So what do you gain when you cache those binaries? Maybe let's go over that.

Yeah, you gain two things. It's performance and reliability. So performance in that it's faster

to restore things from a cache than going out on the internet and downloading things over again.

And reliability, that's maybe more for Elm packages than for Elm Tooling. So if you were

to install Elm packages every time in your CI, that means your CI is talking to the Elm package

site every time. And if that site goes down, then your CI won't run. But if you have them cached,

there's no need to talk to the server. You already have everything. So you can just restore the cache,

your CI keeps working until the site is up again. But for Elm Tooling, well, I guess a lot of people

are using GitHub actions these days, and the binaries are also downloaded from GitHub releases.

So if one is down, maybe the other one is down too. I guess that if you have another CI, it could

probably help. But on the other hand, you might have your code on GitHub, so your CI might not

run anyway. Okay, so we'll get more into some best practices and all of that. Let's make sure we've

gotten all our definitions here. So Elm Tooling CLI is an NPM package that you install. So to

install Elm Tooling CLI, you can't manage that tool with Elm Tooling CLI yet. Maybe someday.

No, that wouldn't work very well. What comes first, Elm Tooling or Elm Tooling?

It's definitely a chicken and egg problem. So you have to get that tool. So you NPM install that.

Most likely, you're going to want to NPM install Elm Tooling into your dev dependencies so you can

manage the version. And this is all this philosophy of locking in specific versions

in your environment. So somebody, they don't have their machine specific setup where they're saying,

oh, why isn't the code compiling on my machine and it is on another person's? Well, one person has

Elm 19.0 installed and another has 0.19.1. And it's locked in through that specification. And

you can also lock in the specific version of the Elm Tooling CLI in your dev dependency. So that

would be the recommended practice there, right? Yeah. Another point about the versions, you talked

about Elm and having different Elm versions in different projects. So that hasn't mattered in

quite some time now because Elm 19.1 came out over a year ago. But something that does matter more

is Elm format, actually. So if someone is on Elm format 0.8.3 and then one product has updated to

0.8.4, you wouldn't want two contributors being on different versions and just keep formatting

the code back and forth all the time. But as you said, it's also important to lock your Elm Tooling

version in every project because the Elm Tooling program actually contains all the links to all Elm

tools and all Elm tool versions that it knows about, which might sound strange at first,

if you think about it. What if NPM worked that way? What if the NPM program itself had the entire

registry in it? That wouldn't work because the registry has tens of millions of packages

and you would need to update that every day, I guess. Every second? Yeah, probably every second.

But for Elm, the tooling changes so seldomly that there was actually no need to complicate the tool

and try to have an online registry of all versions and all tools. I just put it all into the code

instead, which is a very reliable way of doing it, if you can, because you don't have a dependency

on a registry and you don't need a lock file because the tool itself contains those

SHA256 hashes that I was talking about. So maybe let's go through how Elm Tooling chooses

one version or another. How do you say, please Elm Tooling, can you install Elm 0.19.1 or 0.19.0?

How does it do that? Yeah, that's the second part of this project. I said that Elm Tooling

was all about managing your tools. That doesn't mean just installing them. It also means managing

the metadata around your tool. So you can run Elm Tooling in it to create a file called elmtooling.json.

And this is a file where tools can share configuration, but it also contains the versions

of the tools that you use. So if you run Elm Tooling in it, it's going to have a field in there

that says tools. And in there, there's going to be Elm 0.19.1 and Elm format 0.8.4 or whatever.

So you have this elmtooling.json file that specifies all the tools that you use in this

project and their versions. So when you then go ahead and run Elm Tooling install, it looks for

an elmtooling.json, finds the tools and the versions you use and installs those. Yeah, and

there are specific versions that are not version ranges like Elm usually does. Yeah, exactly. It's

specific versions because you depend on one version in your project and that's it. Right, and there's

this thing that we see often with npm packages where it says there are 271 security vulnerabilities

in your npm package configuration. And so the npm binaries wrapping these Elm tools, which are really

just a simple shim that download and executable, which is actually like Haskell binary in the case

of Elm format and Rust, I think in the case of Elm JSON. So it's not JavaScript code like regular

npm binaries. It's just wrapping, downloading them and you get these security vulnerability

warnings because it's npm and you have a different sort of technique that you use to

download it. And so you get these versions like the Elm binary on npm, I think it's like elm

at version 0.19.1 dash something, dash one, dash two. And sometimes people get confused and they

think is there a patch release for Elm? Why is the npm binary changing? But it's actually not. It's

just the wrapper for downloading it and there's a new best practice for avoiding the security

warnings on npm. So using the Elm tooling CLI bypasses those warnings and really just lets you

do it in a much more direct way. Yeah. And as a side bonus so that if you've ever installed Elm,

you might have noticed that it said like installed 70 packages from 69 different people. And when I

see that, I'm like, why? I just wanted Elm and you gave me 70 packages. What are all these packages

doing? And none of them including Elm itself, because it's just an installer for Elm, right?

Oh yeah, yeah, exactly. Yeah. When you npm install Elm, there's like a message that says,

this is just downloading this binary for you. You can also download it from this URL or use these

instructions to set it up on your machine without npm. So it's almost like recommending that you not

use npm to do it because it's not great. But it is really important to lock in the specific versions

because I got my first development job out of college doing like a Ruby on Rails job back just

as Bundler, this sort of Ruby library version manager started coming into use. And it was like

a big deal because it was like, it was absolute chaos. It was the wild west in the days before

that version management system. Because what would happen is you would globally install these Ruby

gems, these Ruby libraries on your development machine. And you'd say, why is there this bug?

Why is this SQL query blowing up? And it's like, oh, you're using the wrong version of active record.

You have to install this one, check out the dev instructions for this. And then somebody installs

the latest version on their machine. Everybody has different versions. There's a different version on

the actual server that's running your production code. It's absolute madness. So it was a big deal

when Bundler came out and there was a way to lock in the versions. It was like a revolution.

And so I don't know, people who are coming onto the scene these days and starting to program might

be like not aware of the absolute madness that is working with globally installed binaries that can

be different on different machines, including the production machine. It is not good and you do not

want to be there. So you need some way to manage binaries and just globally installing versions

is at some point going to bite you and it's not going to be fun. And so you've got to manage it

somehow. Managing it through a package JSON works and it's ugly in some ways, but it locks in the

versions. But if you can do it in an even simpler way and more performant, all the better. End PSA.

This message brought to you by. That's what a PSA means. Public service announcement.

So I think we've sort of laid out what Elm tooling CLI is. It's an NPM package. There's a

specification for sort of how it looks up where binaries are. It finds them in a file called elmtooling.json.

You can initialize that file using Elm tooling init on the CLI, elm dash tooling space init,

and that's going to pull binaries from your package.json. It's going to see that you have

elm version 0.19.1 dash four installed and figure out that that represents elm version 0.19.1.

Put that in your elmtooling.json file and now you're using the same binaries across machines

and your production environment. You don't specify the dash four in Elm tooling, do you?

No, what Dillon meant was that Elm tooling init looks at your package.json to figure out what

versions to put in your elmtooling.json. But if you want to be really pedantic, it doesn't actually

look in your package.json. It looks in your node modules folder and sees if there's an Elm in there

and what version it is on. And if you can't find that, it actually falls back to looking at your

elm.json because you specify the Elm version in there too. Oh, that's a cool detail. I didn't

realize that. I like that. I think, yeah, these solutions are very elegant. I think you've put

a lot of thought into these details and it makes it a really smooth experience. So I think those

are sort of like the key building blocks, but there's one last thing that we haven't touched on

that I think is really essential, which is how once you've used elmtooling.json to

init your Elm tooling, once you've used Elm tooling CLI to init your elmtooling.json file,

you've got the binaries, it's downloaded them, you know, so you run elm dash tooling space

install to install the binaries. Now, how do you run the correct version that it has downloaded?

Great question. So the way it works is that Elm tooling install, it mimics how NPM works.

So if you install a command line tool locally with NPM, it's going to create links to those

command line tools in your local node modules slash dot bin folder and elmtooling does the

same. So it creates a link in, for example, node modules slash dot bin slash Elm to dot Elm in

your home folder slash Elm slash the version you want and slash Elm again. And the reason it does

this is because this way everything just works out of the box. All tools that support local

installations of Elm and Elm format by looking in that node module slash dot bin folder, they're

going to find the links that Elm tooling made, but being none the wiser, they won't know that

it was an NPM that created them. And this means that if you want to run the tools from the command

line, we can also reuse the way you run command line tools installed by NPM, which is using

NPX. So you can type in the terminal NPX space Elm space make or something. And this NPX tool

will look in that local node module slash dot bin and see if there's an Elm in there. And if it is,

it will execute that. So Elm tooling is not only piggybacking on the NPM way of installing local

command line tools. It's also using its execution program or whatever you should call it, NPX.

I really like that approach. So NPX essentially, is it as simple as NPX just looks in your node

modules slash dot bin slash. I guess there's the additional step of if it's not installed,

by default, NPX will attempt to install something. So sometimes people will say,

run this command. If you want to set up a new Gatsby website, then you do NPX Gatsby init.

And it will actually, if there's no Gatsby in your package dot JSON dependencies or dev dependencies,

it will actually install it and then execute the version that it locally installed.

Right. That changed in NPM version seven. Now it doesn't do that anymore. You need to do dash dash.

Yes, I think to for to install something from the internet. Oh, that's great. I didn't know about

that. Yeah, I'm still using NPM six and some colleagues starting using NPM seven. They were

like, hey, this is not working for me. That is confusing behavior because you use NPX

to just say, I want to run the locally installed version of this binary in this projects package

JSON. And then it's like, OK, I'm installing this thing. You're like, what? No, I didn't mean to do.

So I noticed in Simon's instructions and examples for the Elm tooling CLI, you use the flag NPX

space dash dash no dash install to say, which I do not mean. That's really interesting. I didn't

realize that. I wonder what would happen in your build system, though. Is it going to stall out

because it's waiting for an interactive prompt or is it going to fail because it knows it's in a

non interactive environment? Might have to look that up. But very good question. But I've never

liked the behavior that NPX first tries to find something locally and globally and then tries to

download it. It often happens to me that I type NPX space something and I'm in the wrong folder.

So it starts downloading something and I have to press control C. And I wonder, like, what did it

download now? So it's great that they they're changing that. But for now, I'm using NPX dash

dash no install, just like you said, in CI, just to be sure that it's not trying to download

anything, which makes the CI a lot slower. So you mentioned another detail there that I

wasn't aware of. You said that it looks locally then globally, then tries to install it. So you're

saying if I had let's say I had like let's say I'm not using Elm tooling CLI and I have globally

installed Elm. So I did npm install dash dash global or dash g Elm. So I have it installed in

my global npm. And then if I do NPX Elm in a project which does not have Elm in the dev dependencies,

do you know is it going to then try to run my globally installed one and not tell me that it's

not using a local version? Because I don't like that. I'm pretty sure that's the way it works.

Yeah, it's confusing because if you make a mistake, you might think that you're running

your local version, but actually it's running something else. Well, I quite like it for some

things like for instance, I install Elm review globally, and then I run it on projects where

it's not installed yet. So I just do NPX Elm review all the time. And I don't have to think

about it a second time. It would be nice if there was at least like a flag to say like,

I expect this to happen to exist locally. If it doesn't, don't install it. Don't look for it

globally, fail. I wonder if there's an option for that because that's what I want it. I want to be

able to do that at least. So, okay. So I did have one question, Simon, around this NPX. So I really

like the way that the Elm tooling CLI uses NPX, I think. And I really like the way you've sort of

piggyback off of these existing standards and tools to sort of build something simple that can

fit into existing workflows. I'm curious, what do you recommend if you're running a tool that expects

to find the Elm executable, the Elm binary on the path? How do you get it on the path for that tool?

Because it might not be expecting to look it up with NPX. It might be just expecting Elm on the

path. Great question. So that's another benefit from using NPX actually, because when it executes

the command that you wrote on the command line, it also adds everything in that node modules

slash dot bin folder to the path of just that command, which means that if you say NPX,

let's say Elm review as an example, then it will add Elm to the path if you have it locally as

well. So Elm review can just pretend as if Elm was installed globally because it's going to look

like that to Elm review, but your global path variable does not contain Elm. And so because

it also looks at the global packages, like if you haven't installed Elm in this project, then

Elm will still be available. So this is also quite useful for NPX. Very interesting. And what if

you're running a tool that... So you're saying you could run something that's an NPM binary with NPX

and then it will include the entire node modules slash dot bin in the path. What if it's not a tool

that is installed as an NPM package? What would you recommend for making sure that that's in your

path? So what if Elm JSON needed to have access to Elm, but Elm JSON isn't written in node and

doesn't use NPM? Is that what you mean? Yeah. And for example, I have a GitHub action that I've

published called Elm publish action, which is for publishing an Elm package in your GitHub action

script. So you can run your test suite, you can run Elm review, you can do whatever steps you

need to make sure things check out and then publish it automatically at the end if you've

bumped the version. And I provided a way to pass in a path to the Elm executable. So you could

pass in the version, you could pass in the path to node module slash dot bin slash Elm,

but it looks on your path. Or actually do I use NPX? I don't remember, but that's one of the

confusing things is when you use a tool, you don't know if it's looking in the NPM installs or if

it's looking on the path. You could be running a tool that you installed through NPM or you could

be running a different tool like this GitHub actions Elm publish action that I have is not

something that you are running through the typical NPX, it's running through your GitHub action. So

are there other practices you can use to get it onto your path or have you not encountered that

use case? I don't think I've encountered this use case, but if you need to execute something and

that thing needs to find what's in your node modules dot bin, for example, and nothing is

doing that for you, you could always extend the path when you execute that thing in whatever way

you want, I guess. Right. Yeah. You could say path equals dollar path, colon node modules slash dot

bin. Yeah. I guess something like that should work. I guess there will be like tricky cross platform

things. Think about there if that's important to you, but I'm sure there are ways to do it.

Yeah. I was just running something that uses the node Elm compiler package. It's an NPM package

that allows you to compile Elm code locally. And I was testing a little script that I have that

actually compiles and runs some Elm code and it's a JavaScript file, but I was running it through a

local file directly. And if I ran it directly, it doesn't look in the node modules slash dot bin

folder for that executable. It looks on the path. And so what I was doing for that was I was adding

a little script in the package dot JSON. There's a script section. You can say, you know, you can

define a script for test is Elm test and it will you don't have to put NPX in front of Elm test

in your package dot JSON scripts because similar to NPX, it includes all those NPM executables

in your path. And so I was able to use it to create a little script that I run as an NPM script. And

that that solved that problem well. But it's it's sort of a there are so many different variables,

you know, it's hard to find the best practices. Yeah, that's that is definitely an intuitive

behavior of NPX until you learn it. You think it's just a shortcut for executing a binary,

but it's also adding all these things to the path. And once you know that things start to click,

but if you don't know that it's it's a bit magic almost how it can find things that are installed

locally. Yeah, I actually had no idea. I thought it was just looking in your node modules slash dot

bin for I thought it was equivalent to just, you know, besides trying to install it and looking

for things globally, I thought it was equivalent to just doing like dot slash node modules slash

dot bin slash executable name. But that's that's a really important distinction. So what one benefit

of having everything in your package, Jason, like having Elm and formats installed and Elm test is

that whenever you do NPM install, all of those are installed at once. But if I go through Elm

tooling, I'm tooling is installed, but it won't install Elm and Elm formats for you. So how do

you make that work seamlessly? Yeah, correct. So to solve that, you can add a script in the script

section in your package dot Jason, which is called a post install. And NPM will automatically run

this script post install. And in there you can put Elm tooling install. So that means that anytime

you write a type NPM install, it's going to install all of your dependencies and then execute

the post install script, which runs Elm tooling install. This way you can use Elm tooling, but you

don't really need to tell anyone because they won't notice any difference. Yeah. Yeah. So even

my colleagues who don't work with Elm, if Elm somehow needs to be installed and run, as long

as there is a post install step, they won't notice that it's installed differently. Right. Exactly.

Unless your colleagues have listened too much to Richard Feldman. I know you're referring to.

Yeah. Because Richard, he likes to talk about the setting in NPM, which is called ignore scripts.

And if you enable that setting, NPM won't run any scripts automatically anymore. And the point of

that is that you don't want to trigger post install scripts for any random NPM package that

you install, which could do almost anything. It's a bit of a security thing. Because post install

does two things. It installs. So whenever you do NPM install, it runs the post install script

that you have defined in your package JSON. Yeah. Package JSON, it's like a JSON field.

And then there's the, then there's the script key. And then you have a list of key value pairs. It's

test colon test script post install colon some arbitrary command that you do for your post install.

Yeah. But then there's, it also runs the post installs for every package that was just installed,

right? Yeah. And that's like the scary part. Yeah. But it's arbitrary script execution for any NPM

package that you have installed. It's weird that there's no two different ones for the package when

you install it and one for your own project. Exactly. They should be separate because you,

of course you trust your own scripts. Why wouldn't you want them to be run?

We should, we should just send this podcast episode directly to the NPM team and say,

here are our feature requests. I really hope they're aware of this problem.

Yeah. But I, I've tried to document this Scotch as good as I can. So there are some tips on how you

can work around this problem. Yeah. Yeah. So you just created this website, which has a lot of

great tips and some more details on a lot of the things we're discussing. So this is the,

we'll have a link in the show notes, but it's elmtooling.github.io slash elmtooling CLI.

And there's like a quirks page that describes the ignore scripts and post install and all those

details. And this is, this is, this is a really good conversation. I'm glad, you know, I'm glad

that we've got you here to discuss this because you've thought through all these ugly details of

the, you know, sort of NPM binary execution ecosystem. And, and there's a lot, there's a lot

to wrap your head around, but I definitely have some like key takeaways here. And I think just

having some of these, you know, some, some of these gotchas and some of these sort of concepts

in mind is really helpful. Like the idea that NPX puts the node modules binaries into the path, or

even, you know, for, for some people who may, who may not know the, the facts that there is a node

module slash dot bin folder is a really good thing to know because we rely on that all the time. And

it's just like this little bit of magic that if you understand that things sort of fall into place

a little bit better. So it's, it's really nice discussing all these details with you. So one,

one other topic. So I think we've sort of covered using the elmtooling CLI pretty well when it

comes to local development. Let's talk a little bit about what that looks like in a build environment

or in a GitHub action. And, and how does it differ the way that you would run your elmtooling

install step in a, in a CI process? Yeah. So I mentioned that you could put a post install

script in your package.json to make it easy for local development. You just need to run

npm install and you get all your tools at once, but in CI, it's actually better to split them up

to install npm packages in one step and your elmtooling in another step. And that's because

of caching. So what I recommend is caching both your node modules folder and your dot elm folder

in, in, in your home folder, the elm home. And these will have different cache keys. The node

modules folder, it depends on your package lock.json. Package lock.json changes, then your

node modules will change. So that means that you cannot use your cache anymore, but the dot elm

home folder, it does not depend on package lock.json. It depends on your elm.json and your

elmtooling.json. So in order to be able to have these separate cache keys, you also need to

install them separately. So for this reason, I added a bit of a hack. You can set an environment

variable called no elmtooling install. And that disables the elm install command that you probably

have in your post install script, which means that you can run npm install without triggering

elmtooling install. And once that's done, you can move on to the next step and run elmtooling install

separately. And that's for the optimum caching strategy. If this is super difficult for you to

set up, you can still install both in one step, but you won't get the perfect caching anymore.

Right. So essentially, it may overfetch these binaries that elmtooling.json manages rather

than fetching them from cache? Yeah. For example, your package.json is much more likely to change

than your elmtooling.json because you have a lot more stuff in there and npm packages change all

the time. And you don't actually need to reinstall all the elmtools just because of some npm package

update. So by splitting them, you can allow for installing just the npm packages and keep using

the cache for the elmtools. So it's an optimization to get more cache hits, but you're not going to

have any sort of incorrect cache fetches where it gets the wrong thing from the cache. You would

just have cache misses that could have been cache hits where you didn't have to go redownload it.

That's good to know. So okay, so to summarize, because this is a lot to wrap your head around,

but essentially, first of all, you have an example GitHub action script, which I found really helpful

for setting this up on my projects. And you have some nice comments in there that sort of describe

some of the reasoning here. So you can find that in the show notes. You recommend using the npmci

command, which is a slightly different version of npm install. Do you want to talk a little bit

about why you recommend using that instead and how it differs? It's basically because npm themselves

are recommending it. It's called npmci because it's optimized for CI. And npm install is a bit of a

mixed bag command. It looks at your package.json and sees if anything needs updating. And it might

update your package.lock.json. But npmci, it says, I'm just going to read package.lock.json

and install everything that's in there. And if there's anything wrong, I'm just going to error

out. So yeah, it avoids unnecessary work. And it's a little bit faster. Which the sort of dependency

version resolution is one of the most expensive parts of an npm install. So it completely bypasses

that and just looks for what's in the package.lock.json. So that's a performance boost in itself.

And then you pair that with being able to use the cache key of the package.lock.json. And if you do

that, whether it's with GitHub actions or whatever else you do, you specify that your package.lock

is the cache key. And if the package.lock changes, then you use a fresh cache. And that means you're

going to have to run a fresh npmci. It's going to do all the installs. But otherwise, you don't have

to pay that cost. And you can fetch it from the cache. So that's a super performant way of doing

that. So OK, so first step, you do npmci. And as npm recommends, you use a cache key of your

package.lock.json. So you don't need to do a fresh npmci install. You don't need to run that command.

Or you don't need to fetch those binaries or executables at all if it's a cache hit. You

bypass running the elmtoolingcli postinstall step when you're running that step in your CI.

And then as a separate step, you run elmtooling install explicitly. And you run that with a cache

key of your elm.json and your elmtooling.json files. There's a lot going on there. But I mean,

essentially, you're just separately caching and executing npmci and elmtooling install to make

sure that you optimize the cache hits. And there's documentation about this on the website. And the

example GitHub actions workflow, like you mentioned. So it might sound very complicated, but you can

look at the examples and remember that we have the word cache in this discussion, which of course

makes it much more difficult to reason about. Great point. And then the sort of last point on

running elmtooling in your build step is if you, again, look in this example GitHub workflow script,

you can see that it's just running elm make with npx. And then you use dash dash no install to

make sure it doesn't do any of that funny business. And then you just run these regular commands,

elm make, elm test, elm review. You're running all of those with npx. Elm review is not currently

managed by elmtooling. But in general, you're just running everything through npx, which could also

be an npm script. You could have a script for each of those things in your package.json. And then it

would have the same effect where it's executing those with the right path that includes all of

the executables from node module slash dot bin. Yeah, exactly. The key to success is to use either

npx or npm run. Then you get the path set up for you. It's just that I like using npx if possible,

because then you have less indirection. Instead of having to look at your GitHub actions and see that

it runs npm run lint and then go to your package.json and see that lint runs elm format,

you can just see that it runs elm format directly. And also as a bonus, you don't get these super

long npm messages at the end. If you've ever noticed that when a command fails, you don't just

get the output from the command, you also get like five or 10 lines of information from npm.

Yeah, I actually prefer to have npm scripts, because that way I can run the same test locally

and in the CI. And to remove the noise you were just talking about, you can run

npm run dash s or dash dash silent, and that removes all of those. Yeah, that's a great trick.

Yeah, those 10 lines of like, this is not an issue with npm, please look at this,

please don't blame us and file issues with us. I swear it's not my fault. It looks like you.

Yeah, npm run dash s dash dash silent is a very good trick that you're in shared with me. Yeah,

I think that I use npx quite a lot as well, but I also do kind of like using the script

section in a package.json to manage sort of the knowledge of how do I build this, how do I start

up a server. I really like having like for all of my projects, I like to be able to run npm test,

which is sort of weird because for start and test, and there may be a couple of others,

you don't have to do npm run script name, you can do npm test or npm start. But piggybacking on those

conventions, I really like to have a script in any project. And I'm talking about Elm projects here

for start and for test. So I always run my test suite and I want it to be, you know,

this is what my CI is going to run. So if I run npm test, then I want my CI to succeed. And if I

run npm start, I want a dev server. And if I run npm run build, then I want my production build

step. That's just the conventions I follow. But I also use NPX heavily, both locally and sometimes

in my CI and it's very useful as well. I usually duplicate everything that the GitHub actions

workflow runs into my npm test, which feels so so because there's a risk they'll run out of sync.

But my dream setup would be to have a tool that runs the GitHub workflow locally. So yes,

just say execute whatever GitHub actions would do, but locally and quickly.

Would you like it to do the caching too?

No, I would like to skip over things like choosing the operating system, caching,

stuff like that, installing things, just run like my own things like linting tests and building.

You sure don't want it to install Windows for you real quick?

Yeah, I mean, I think, you know, the common thread, the common thread in all the things

we're discussing here is sort of having a declarative environment agnostic way to run

scripts. Like I want, you know, my production build to end up the same, no matter where or how

I run it. I want like one way to run it in different environments and I'll get the same thing. And I

want one way to run my tests and I'll get the same result.

I think that some people will tell you that's what make files are for.

They don't work on Windows.

And to that I have nothing to say. They don't really.

I might be misremembering. It's been a long time since I actually developed on Windows. But back

then it was so annoying when some NPM product used make file because I couldn't use it. So

for Elm tooling, I've really gone out of the way to make sure that everything works perfectly on

Windows as well.

From the community, thank you.

Yeah, yeah. It's annoying to do.

And yeah, it is very interesting to be, you know, be in this position, you know, as a community

where we have these lovely tools that we all love. You know, we I mean, Elm format, Elm review,

Elm compiler, there are so many awesome tools and we live in this happy little bubble of the pure

Elm space. And some of some of us venture outside of that bubble to build really cool things to

make it more awesome working in that bubble. You know, we build stuff in JavaScript and in

these ecosystems. But we leverage the JavaScript ecosystem so much. And I'm very grateful to have

all these resources because I mean, imagine if Elm had to figure out the story for how do you

install binaries? We've got a huge thriving community. You know, it may have some sort of

design choices that aren't in line with sort of the style that that, you know, as sort of pure

functional aficionados we might we might want, right? We might want we might be more more our

values might align better with tools like Nix that, you know, give us declarative pure builds

and stuff. But the fact is, we've got this huge vast ecosystem. It's well documented, it's well

supported, it's very feature rich. It's got, you know, all these executables, you want to run

multiple scripts in parallel and fail on the first failure or continue on the first failure,

whatever, there's some like cross script NPM package for that. So it's it's definitely a

love hate relationship where the design decisions sometimes aren't aligned with with what we would

want in a very like declarative pure world. But there are so many great tools out there at our

disposal. Yeah, definitely. So I was wondering, why do you not look at the version in the Elm

JSON file? So when you so to install Elm, the applications they have Elm version 0.19.1. Why

don't you use that one? I know it's not available for packages. Is that the reason? Yeah, I've been

thinking about that a lot. It's like one of those all details and it's so hard to make up your mind

what way do you want it to be. But ultimately, because of that, it wouldn't work for packages,

like you said, because in a package Elm.JSON, you just you don't say Elm version 0.19.1, you say

0.19.1 up to but not including 0.20.0. So you get a range, which means that the tool wouldn't know

which version you want. And so I decided let's put all of the versions in one place in Elm tooling

dot JSON. And you will always find them there. It means a tiny bit of duplication, but I think it's

fine because the Elm versions don't change very often. No, for better or worse, they don't change

often. All right, so there are other fields in that Elm tooling dot JSON than tools. Yeah,

exactly. Which are they? So far, it's just one other field. And it's called entry points. And

this is used by the Elm language server, which powers the VS code extension and the vim extension.

And what the language server uses it for is to find which files to compile to compile your entire

project. And that is your entry points. So sometimes you just have one entry point, maybe

in main dot Elm. And if you compile that, then it will import other files which imports further

files and so on until you have gone through every file in your whole project. But if you have another

entry point, maybe you have like two apps or two separately built pages or something, then you

would actually need to compile both those entry points to cover all of your files. So this way,

you can specify all of your entry points. So tools can know how to find errors in your entire project.

And other tools than the language server are free to use this field, which is kind of the point with

this Elm tooling dot JSON. It's supposed to be a shared place to put all of your configuration,

both to like keep it all in one place. And so that tools can collaborate on reusing the same

configurations. You don't have to specify it several times. So if there's any tool that wants

to put something in the Elm tooling dot JSON, just open a discussion and we will try to figure out

what we need and what it should look like. Yeah, I really like that entry points field and that

general philosophy of having like a common, you know, specification and place to store this

project metadata that tools can share. I find myself pretty often, I mean, we may all be sort

of unique here in that we're very focused on building tools. So our use cases may be different

than the standard use case. But I do find myself frequently having a lot of entry points for,

you know, my examples folder in an Elm package and, you know, some test case. And it's really

a pain right now and a little bit scary in case I might miss compiling something. I might have an

example that isn't compiling and publish a version of a package or push a commit and not know that

it's broken. So I try really hard to write scripts in my GitHub actions CI where I, you know, compile

every entry point. But I would love to have like a more declarative way that tools can share. So

instead of it just being, I mean, I think Simon, this is sort of the sort of vision that you were

getting at that wouldn't it be nice if there was a declarative what you were talking about? Wouldn't

it be nice if we could just run our GitHub actions so we could share everything. But the more

declarative you can be about things, the more simple you can be. So if you just declare, these

are my entry points, then tools, you don't, it's more imperative to say, okay, go into this

directory, come run this command to compile this module, then go into this directory entry points

is declarative. It's just saying these are my entry points. And if a tool says, Oh, I'll look at your

entry points and make sure they all compile, it can figure out how it wants to actually execute that.

So you can, you can share something among tooling, the more declarative it is, the easier it is to

share between tools because imperative stuff, it's hard to get to the bottom of what is it actually

trying to do. There are all these low level details of how it does it. It's hard for tools to share

that because it's not like a simple specification that different tools can leverage. So, so I really

like that concept. Yeah. I might actually use it for Elm review because Elm review doesn't know

which files you're going to compile. So that could be interesting. I'll have to think about use cases

for that first, but that's definitely interesting. And then there's another, another thing you've

added to the Elm tooling CLI project, which is there's so as we've discussed, it is an NPM package,

which gives you an executable called Elm tooling, but it's also a node package and you can

programmatically as a tooling author, you can programmatically invoke Elm tooling to execute

binaries that are installed with Elm tooling. You want to tell us like a little bit about that?

Yeah, sure. So both Elm test and Elm review use Elm JSON when they compile things to figure out

how the dependencies work for your project. That's pretty complicated to calculate. So they use this

tool called Elm JSON, which you might've used yourself when you update your Elm.json file.

But that means that that Elm JSON tool needs to be available, which is a REST executable that you

can install with NPM install. Exactly. So in order to not put the burden on the user to install

Elm JSON themselves, what the Elm review does is that it depends on the Elm JSON NPM package.

And my goal has been kind of to get rid of all these Elm binary NPM packages,

but I still want to use Elm review. Thank you.

So it annoys me that it depends on Elm JSON, the NPM package, because now I get all of the

dependencies and slower installation time whenever I install Elm review. And this is why I made this

API in Elm tooling. So Elm review can import Elm tooling slash get executable, which is a function

and you say to the function, what tool you want Elm JSON in this case, and which version you want.

And then you can say, I want correct sign 0.2.8. And the correct sign is just like in NPM,

it gives you a semver range and the Elm tooling get executable function will then try to find

a matching version, see if it's installed. If it isn't, it downloads the binary and then it returns

the absolute path to this tool, which Elm review then can execute. Yeah, that's it.

A way for NPM packages to use Elm tooling to depend on some Elm binary.

If the user had happened to install that in their local dev machine and they're running

Elm review, which is using this Elm tooling API, then it would not need to install that as with

any other Elm tooling install, or if it needed to, it could install that.

Exactly. So if the user themselves has a Elm tool, they can use that Elm tool.

If the user themselves has installed Elm JSON using Elm tooling before, get executable does not need to do much really.

Or if you've run Elm review once and it installed Elm JSON then, you probably will never need to

download Elm JSON anymore because it's already in that shared.elm home folder.

Oh, that's wonderful. So I have run into a similar use case in Elm GraphQL.

The command line tool for Elm GraphQL, which runs the code generation,

it would be very difficult. I tried to get it perfectly formatted without Elm format,

not to mention the fact that people might use different versions of Elm format for their project,

which I can't target arbitrary versions in the code generation tool.

At some point, it just became clear that I need to just run Elm format on the generated code rather

than trying to get the perfect output that will succeed on people's Elm format steps or look the

way they want it when they're browsing the code. And so now would that use case apply to the Elm

tooling node API? Would I be able to say, give me whatever Elm format version the user has?

I mean, I guess if I needed to, I could read their elmtooling.json and look for that or otherwise

fall back to a specific version. I think that like your goal here is that generated code is

supposed to be readable, right? Because you sometimes go into it to find out what the types

look like and stuff like that. So in an ideal world, you should be able to use whatever Elm

format version you want. But I guess that it's good if you use the same version as the user

themselves do, just in case they run Elm format on all of their files, included your generated files,

then you wouldn't want Elm format to say like, up, up, up, you need to format all of these files,

according to my style. So it could be beneficial to use the same version as the user. So they don't

need to like ignore those files. And that's not what this API is for. So then you should try to

invoke the user's Elm format binary. And I think this is what Elm review does, right?

Uh, yeah, yeah, we try to get the format and the Elm compiler that the user chose.

And the way that works, I think is that Elm review or Elm GraphQL in your case,

it just assumes that Elm format is on your path and tries to execute that. And if you have run

Elm GraphQL or Elm review with MPX, then Elm format will be on your path because MPX adds

the local node module stop in to your path. And as a fallback, it's good to have a flag that says

like dash dash Elm format path or something. So you can pass in your own as a last resort.

Yeah. Elm review tries both with MPX and without MPX and hopefully it will be there. Not a lot of

people have complained, so I'm guessing that works well. Interesting. In Elm GraphQL, I run Elm

format just with MPX and the, you know, for a short time I got like one person saying, hey,

it doesn't work. And I said, Hey, you have to install at least NPM version 5.7.2 or something,

because that's when MPX was introduced. But otherwise that hasn't been problematic, but it,

it's, it's good to discuss these things and these, these sort of best practices, because I think that

both for tooling authors like ourselves and for tooling consumers, I think it's good to sort of

understand a little bit of the magic behind how these things work. So if something doesn't seem

to be working correctly, you can say, Oh, maybe I should be running NPX and things would magically

be on the path using the versions that I've specified in my Elm tooling dot json or my

package dot json. So it's good to sort of look at how the sausage is made a little bit,

even though it's messy and there are so many best practices and so many things to think about.

So I think a lot of people wonder why is there Elm and Elm formats, but not Elm tests and Elm

review as we already mentioned. Yeah. You mean like why Elm tooling does not support installing

Elm tests and Elm review. Yeah. Elm tooling only supports tools that compile to a single executable

that are cross, no, there aren't cross platform while Elm test and Elm review are Node.js packages

and they depend on other NPM packages. And like, there's no way to beat NPM or YARN for installing

Node.js tools. So I think it's important to use the right tool for the right job. So it's much

better to install Node.js based tools using NPM and let the Elm tooling deal with the stuff that

NPM is bad at, which is installing different binaries for different operating systems.

Right. That's a great point. That's the specific problem that it's solving is there's a single

binary that Elm, the Elm Haskell code base compiles to a single binary for different

platforms. There's a Mac 64 bit binary, there's a Linux binary and Elm tooling manages fetching

the correct version of that single binary, which is not a bunch of JavaScript NPM packages. It's

a single binary and it manages fetching it for the correct platform and that's the specific task

it does. So, okay. So I think is there anything else that people should know to understand what

the heck is going on when they're installing things to have best practices? Are there any

tips or concepts that are important that will help people sort of master their tooling setup better?

Not sure if there's much more to say there. Maybe that if you already use NPM to install

an Elm format, you should be able to just switch to Elm tooling and get all the benefits without

much work. Right. So, okay. So let's kind of talk a little bit about how someone would get started

and just summarize. I think we've sort of covered it, but if somebody wants to start using Elm

tooling CLI right now, what are their first steps? What are some resources that are going to help

them? What are some things they should keep in mind? Yeah. So there's a getting started section

on the website and it mentions like a super quick start if you already feel comfortable with setting

things up and just want something quick. And then what you basically do is that you say NPM install

dash dash save dev Elm tooling to get a local copy of Elm tooling. And then you run npx Elm tooling

init to create an Elm tooling dot json and then npx Elm tooling install to install everything in

your Elm tooling dot json. And then you're done. Then you can start using like npx Elm help or

npx Elm format to format something so on. And apart from that quick start guide, there are two

more detailed sections, one for adding Elm tooling to an existing project and one for making a

product from scratch with Elm tooling. And it's basically still the same steps. I've just fleshed

out with some more details and like tips that you need to go into your package dot json and remove

Elm and Elm format from there if you have installed them with NPM previously. But it should be a pretty

streamlined experience to do it. That's the goal at least. Yeah, the documentation is very thorough

here. Would you recommend uninstalling all of your global Elm executables, your global Elm

binary and your global Elm format binary to ensure that you don't accidentally use the global one by

mistake? That's a very good question. So there's nothing saying that you have to do that. It's

perfectly fine to have them as they are. But as you say, you can accidentally call your global

version by mistake. But on the other hand, right now the Elm ecosystem is so stable. So even if

you accidentally call the global version, it's likely going to be the same, the correct version.

So yeah, it's unclear if you're used to having a global version and you use that for like most of

your projects, then by all means continue doing that if you like. And you can use Elm tooling just

for certain projects. Myself, I have global versions of Elm and Elm format, but I've noticed

that I've almost never used them. Right. And it's kind of nice to know that you're not by mistake

using those, right? Because you want things to fail if you do something unexpected, just like

when you run NPX, you don't want it to be like, oh, I didn't find that. Let me start installing

this thing for you. And you're like, what? What are you doing? So when I'm thinking about what

would you use a global Elm binary for besides running Elm test and it needs to have an Elm

version, which we've kind of talked about, if you do NPX Elm test, it will find the version that Elm

tooling has installed for you. If you're running a REPL, the thing is a REPL depends on a specific

Elm.json file because it's going to, you can't just run an Elm REPL anywhere. It's going to

actually look for the Elm package dependencies in your Elm.json. So more and more, I think of

an Elm project as an atomic unit that has a package.json and that has an Elm tooling.json.

Like to me, that is like an atomic thing that Elm does not exist outside of that. And so I more and

more lean towards, I don't want to accidentally fall back to a global binary. I don't want to

run NPX and think that it's using a locally installed version, but it's actually reaching

out to the global binary. I want the global binary to not exist. So more and more, I'm thinking that's

the way to go for me. But yeah, all good tips here. Cool. Any last tips or resources for getting

started? This should be it, I think. So if people have problems or questions or want to contribute,

where do they go to do that? Yeah, you could always open an issue on the GitHub repo. There's

also a Discord channel in Dillon's Discord. And I guess that we could also make a Slack channel

eventually. So you can choose whatever you prefer, I'd say. Yeah, Simon jumps on and gives a lot of

thoughtful details if you ask him a question. Not to invite people to bombard you with questions,

but I've appreciated that. Same here. I want to really thank you, Simon, for coming on the podcast

and talking to us about this. It's been very illuminating. And I want to thank you for the

thought you've put into this tool. It really shows. It's one of these things where you can see

how much work it took to put in all this thought and consider all these options to then go to the

simplest possible path, which then looks like, oh, you run it with NPX. So it wasn't much work,

right? But you can see all the thought that went into, oh, if you leverage this one thing,

all these things fall into place. And it really shows. And discussing it with you,

it shows even more. And it was great to sort of pick your brain on that. Yeah, Simon was the one

who made the pull request for Elm Review to use Elm tooling. And I had a lot of questions, and

he answered all of them flawlessly. And there was no problem at all. I raised so many issues,

and he was like, no, that's fine. This is working this way, and it's working great. And well, okay,

I just merged it then. Yeah, as a tooling author, as fellow tooling authors, I think we can really

appreciate what a feat it is to accomplish to get to a point where it's like, wow, that lines up

really nicely. And you know that it's not luck. It's a lot of careful thought and design detail

attention. So thank you, Simon. And it's been great chatting with you. Thanks for having me

at a great time. Our pleasure. And Jeroen, until next time. Until next time.