spotifyovercastrssapple-podcasts

Elm Code Generation

We discuss different use cases for code generation in Elm applications, and our favorite code generation tips.
May 24, 2021
#31

Watchers for rerunning codegen

Scaffolding

Transcript

[00:00:00]
Hello, Jeroen.
[00:00:01]
Hello, Dillon.
[00:00:02]
Well, what are we talking about today?
[00:00:03]
Today we're talking about code generation.
[00:00:04]
Uh huh.
[00:00:05]
Well, have you ever done code generation?
[00:00:06]
I have done my fair share.
[00:00:07]
Uh huh.
[00:00:08]
I'm guessing you too.
[00:00:09]
I've done a lot of code generation, and I really enjoy it.
[00:00:10]
It's very cool.
[00:00:11]
So like code generation, well, as is our tradition, let's lay out some definitions.
[00:00:26]
So what is code generation in the context of Elm?
[00:00:30]
Well, code generation is just writing Elm code.
[00:00:33]
So you are a code generator, Dillon.
[00:00:36]
That's true.
[00:00:37]
You are a very good code generator.
[00:00:38]
I'm a code generation machine.
[00:00:41]
And if you want to become even more productive at generating code, what if you wrote code
[00:00:47]
that could write code?
[00:00:49]
Wouldn't that be cool?
[00:00:50]
I thought you were going to say, oh, what if you want to become even better just by
[00:00:55]
my book?
[00:00:56]
Maybe.
[00:00:57]
Yeah, no, I mean, I mean really, uh, you know, Elm is a code generator.
[00:01:04]
A compiler is a code generator.
[00:01:07]
When we're talking about code generation right now, what we're really talking about is doing
[00:01:12]
things that you wouldn't be able to do just by writing some simple Elm code or, you know,
[00:01:18]
perhaps adding some safety mechanisms.
[00:01:20]
So like, yeah, basically generating code through automated scripts or tools and for different
[00:01:27]
purposes that we will go through.
[00:01:28]
Right.
[00:01:29]
So it's like, I mean, one of, um, you know, obviously one of, one of the examples that
[00:01:34]
is top of mind for me would be Elm GraphQL.
[00:01:36]
And I think that that's a pretty good, pretty simple example in a way, because, so like
[00:01:42]
one of the things that I think is really interesting to think about for code generation is what
[00:01:48]
is the source of truth and what's the mental model.
[00:01:51]
And so for, for Elm GraphQL, what's the purpose of the code generation?
[00:01:56]
What is it giving you that just writing code by itself couldn't give you?
[00:02:01]
Why not just use a plain old API, right?
[00:02:03]
Like if you can write an API without code generation, that's far better.
[00:02:08]
You shouldn't reach for code generation unless you really need it.
[00:02:10]
So first of all, why would you go through for code generation and not write the code
[00:02:15]
yourself?
[00:02:16]
Right.
[00:02:17]
So if you take the example of Elm GraphQL, you've got this external thing, the schema
[00:02:22]
and Elm doesn't know about that at compile time.
[00:02:26]
At runtime, you can teach Elm about your schema, but you want compiler guarantees.
[00:02:33]
So that's one of the really cool things you can do with code generation is you can sort
[00:02:36]
of bring this information so that not only is Elm aware of it at runtime when your code
[00:02:42]
is running, but the compiler that you can teach the Elm compiler about external things
[00:02:48]
because the Elm compiler, you can't just tell the Elm compiler.
[00:02:52]
Now with, with certain fancy magical languages, you can do this, but Elm is not a fancy magical
[00:02:59]
compiler.
[00:03:00]
It's a very simple, predictable, boring compiler, which is really one of the things we like
[00:03:04]
about it.
[00:03:05]
But we also like our compiler to know about the types in our GraphQL API.
[00:03:10]
That's very nice.
[00:03:11]
Right.
[00:03:12]
So you can try to get a field and the compiler says, uh, there's no such field, or, uh, you're
[00:03:18]
treating this as if it was an int, but it's actually a string.
[00:03:21]
We like that.
[00:03:22]
We like the compiler to know that external information.
[00:03:24]
So that's one reason that you would do code generation is to bring in information about
[00:03:30]
some external source of truth.
[00:03:33]
And well, you know, types without borders, you're erasing the borders between the type
[00:03:37]
system of the Elm compiler and your GraphQL schema.
[00:03:40]
Yeah.
[00:03:41]
If you were to write it yourself, then you have a potential bug because you're saying
[00:03:48]
that this field is an integer and actually it's a float or something like that.
[00:03:53]
So at runtime you would have some kind of error, an HTTP error or...
[00:03:58]
Right.
[00:03:59]
It's a good way to keep two sources of data in sync with the Elm compiler because you
[00:04:05]
can, you know, you can figure stuff out by running Elm code, but we like to do better.
[00:04:10]
So that's one thing that like some languages have macros and other metaprogramming tools.
[00:04:17]
And now, so like, let's talk about that a little bit.
[00:04:21]
So like, like what is a macro?
[00:04:24]
Which Elm doesn't have macros.
[00:04:25]
Yeah.
[00:04:26]
It doesn't have any macros out of the box.
[00:04:29]
And there's, I don't think there's a tool out there at the moment that has macro support
[00:04:34]
or something.
[00:04:35]
Basically your macros are a way of changing your code.
[00:04:41]
Probably at compile time or maybe at runtime in some cases, depending on, yeah, I don't
[00:04:48]
know, like configuration.
[00:04:50]
I think that's a good way of defining it.
[00:04:51]
It's basically like a way of programmatically changing code.
[00:04:55]
So you like essentially take, I mean, sometimes people talk about it as like code as data.
[00:05:02]
So like a lot of Lisp variants are known for being able to use a macro where you say this
[00:05:10]
function is going to take this code as an argument.
[00:05:13]
So whatever code you pass to it, you can sort of go in and tweak the abstract syntax tree
[00:05:19]
or inspect it, which is quite handy, but it can make it difficult to understand things
[00:05:24]
and it can make it harder to predict like buckle script slash rescript, whatever they're
[00:05:32]
calling it these days.
[00:05:33]
What was the old name?
[00:05:35]
Reason ML was the old name.
[00:05:36]
Now they call it rescript.
[00:05:38]
They've got a feature, these PPX macros.
[00:05:42]
And I mean, that's extremely powerful.
[00:05:44]
You can basically like go do whatever side effects you want to make code available.
[00:05:51]
But if there's a problem, how do you go figure out what the problem is?
[00:05:55]
Or if you're trying to understand what your code can do, where do you go to figure that
[00:05:59]
out?
[00:06:00]
Whereas if you're just using vanilla code generation, which is what Elm gives us, then
[00:06:03]
you can go and you can look at that generated code.
[00:06:06]
It's in the folder somewhere.
[00:06:07]
Your IDE will even help you command click into the code and see exactly what the code
[00:06:12]
does.
[00:06:13]
Yeah.
[00:06:14]
Most of my experience with macros are from C from a very long time ago where you basically
[00:06:18]
just replace constants with other values, like for enums, mostly.
[00:06:24]
That's what I use it for.
[00:06:25]
I'm guessing it's way more powerful than that, but not to my knowledge.
[00:06:29]
Yeah.
[00:06:30]
I guess the word macro is used and it can mean very different things.
[00:06:34]
So I think C macros are precompiler, like a compiler preprocessor macros, which is very
[00:06:42]
different than like a Lisp macro, which is arbitrarily allowing you to modify the code.
[00:06:48]
And the other one is Babel.
[00:06:50]
So Babel has, that's for JavaScript.
[00:06:53]
You configure Babel with a bunch of plugins and you tell it what transformations to do.
[00:06:59]
And basically it allows you to write a new JavaScript version code and transform it to
[00:07:05]
ES5 or something.
[00:07:07]
Or it can allow you to do pretty much anything.
[00:07:10]
Like if you want to add a macro that says all the strings in the source code, make them
[00:07:17]
uppercase, that could work.
[00:07:19]
And it can be very powerful, but as you said, it can be a big pain to debug when something
[00:07:24]
goes wrong.
[00:07:25]
Right.
[00:07:26]
And that's actually something that I think it's quite nice to embrace as Elm users that
[00:07:35]
this is a benefit of Elm that it doesn't have this complexity to deal with.
[00:07:40]
If something goes wrong, we don't have to dig through some cryptic messages or just
[00:07:46]
not have any way to understand what's going on.
[00:07:48]
We can see exactly the code that was generated and that's quite useful.
[00:07:53]
Because what you have in mind is generating Elm files, right?
[00:07:57]
Exactly.
[00:07:58]
That's all it is.
[00:07:59]
It's writing an Elm file to a disk.
[00:08:01]
So one of the things that, I mean, we're recording this episode about code generation and I want
[00:08:08]
to emphasize, I don't think that this concept is only useful to library authors.
[00:08:14]
I think that this concept is useful.
[00:08:17]
It's extremely useful for library authors and sure, code generation sounds scary, but
[00:08:24]
I would like to make it sound less scary.
[00:08:28]
And I think that code generation should feel like something, again, you don't want to,
[00:08:33]
if you can do something, if you can accomplish a goal without code generation, you by all
[00:08:37]
means should.
[00:08:38]
And it does add complexity.
[00:08:40]
It's funny what you said, because to me, code generation is mostly for applications.
[00:08:45]
I would have almost said, you know what, code generation can also be useful for libraries.
[00:08:51]
That's interesting.
[00:08:52]
Well, then I guess you're already thinking the way that I'm hoping people will.
[00:08:59]
I always get the feeling that code generation feels intimidating to people, but maybe, I
[00:09:03]
mean, I don't know, maybe that's not.
[00:09:06]
I guess it's something that you have to get used to, because people who have used ElmGraphQL
[00:09:12]
or similar projects, they've already bought in the idea of, I have my Elm project and
[00:09:19]
I have a set of build tools that generate the code that I need to run before I run Elm
[00:09:24]
make or before I need to push something to production.
[00:09:28]
I think that's most of the upfront cost that you have or most of the setup costs that you
[00:09:34]
have.
[00:09:35]
So once you've bought into that, you're free to do a lot more code generation.
[00:09:39]
Yes, absolutely.
[00:09:42]
And it is extremely useful for applications, as you say.
[00:09:47]
And I think that code generation seems very intimidating.
[00:09:55]
One of the things that I always like to say about code generation is that code generation
[00:09:59]
makes it sound like such a difficult, complicated thing.
[00:10:03]
But you could do a code generation script that is, you know, like code generation could
[00:10:09]
be as simple as what is the build time?
[00:10:11]
Like you want a timestamp for when you build a script and you want that available as a
[00:10:17]
value in Elm.
[00:10:19]
So it's going to be like just a POSIX time value, right?
[00:10:23]
Yeah.
[00:10:24]
Or a string like for the build version or something.
[00:10:26]
Yeah.
[00:10:27]
Right.
[00:10:28]
Exactly.
[00:10:29]
Exactly.
[00:10:30]
So that's going to be some little piece of metadata that you have that, you know, that
[00:10:31]
timestamp represents your build.
[00:10:33]
So every time you do a new build, you have a new timestamp.
[00:10:36]
So how would you go about that?
[00:10:38]
It's a very simple example.
[00:10:40]
So let's go through all the steps together.
[00:10:43]
That sounds great.
[00:10:44]
Yeah.
[00:10:45]
So what I always like to say about code generation is that really it's just a type of templating.
[00:10:50]
And you can use different tools to template.
[00:10:52]
If you want to get really fancy, you can pull in some abstract syntax trees and output those
[00:10:58]
to strings.
[00:10:59]
But why would you do that if you're just generating a module, you know, build module build, exposing
[00:11:06]
timestamp and then timestamp colon POSIX dot time timestamp equals one, two, three, four,
[00:11:13]
five from POSIX time or something like that.
[00:11:15]
Right.
[00:11:16]
That's the code we're generating.
[00:11:17]
Why wouldn't you just do string templating?
[00:11:19]
So what I would say is just templating.
[00:11:23]
It's just templating.
[00:11:24]
Just write a little node JS script.
[00:11:26]
So just write like generate build module dot JS.
[00:11:33]
And then, you know, you're just going to you know, if you're not familiar with node, you've
[00:11:36]
got a couple of things to learn.
[00:11:38]
You can use whatever your preferred tool is.
[00:11:40]
If you prefer to use Ruby or Python, you're not going to do it in Elm because how are
[00:11:45]
you going to write to a file with vanilla Elm?
[00:11:47]
That's not really going to help you out.
[00:11:48]
Well then you use node.js.
[00:11:49]
Oh, yeah.
[00:11:50]
Right.
[00:11:51]
Exactly.
[00:11:52]
So at the end of the day, all you're trying to do is write out this file.
[00:11:57]
In fact, if you want, just use bash.
[00:12:00]
That that would be fine, too.
[00:12:02]
That's not what I would reach for.
[00:12:03]
But let's just say we're using node.js.
[00:12:05]
So we're just going to, you know, say just write a little string template and we're just
[00:12:10]
going to do exactly that string I mentioned, except the one, two, three, four, five part.
[00:12:15]
We're going to do, you know, date dot whatever.
[00:12:19]
Turn it into POSIX time using some fancy JavaScript thing.
[00:12:23]
And then we're going to do Fs dot write file sync, probably.
[00:12:27]
And that's it.
[00:12:28]
Now we've done some code generation and then we're going to hook that into our build.
[00:12:32]
And every time we run a build, we run that code generation thing.
[00:12:35]
And then we're probably going to want to get a little bit of extra confidence that it's
[00:12:42]
a fresh generated file.
[00:12:45]
And therefore we'll probably get ignore our build dot elm module or put it in like a gen
[00:12:50]
folder.
[00:12:51]
Maybe we'll output it to a gen folder, which is ignored.
[00:12:55]
Which is usually what I go for.
[00:12:57]
Yeah, because that's very nice, because then you know that you're not accidentally forgetting
[00:13:01]
to to generate it on on your build server and it's going to fail to compile.
[00:13:07]
If you do elm make, it's going to say, what is build dot elm?
[00:13:11]
And you say, oh, whoops, I forgot to run my build script.
[00:13:14]
That's it.
[00:13:15]
Now we've just done your first code generation.
[00:13:17]
You do need to change your project's source directories to include that new folder.
[00:13:24]
Yes.
[00:13:25]
If you generated it in a gen folder or if you, you know, you could just generate it
[00:13:28]
right in the source directory and add source slash build dot elm to your to your get ignore.
[00:13:35]
But in general, it's I think it's a nice practice to have a separate source directory for generated
[00:13:41]
code.
[00:13:42]
I don't even see what a drawback for that could be.
[00:13:45]
No, I think that's that's a great practice.
[00:13:49]
So that's it.
[00:13:50]
We just did code generation.
[00:13:51]
Yeah, I think my biggest drawback for or I think my biggest pain point for code generation
[00:13:57]
would be having to set up a new kind of dev server.
[00:14:01]
Like when you are developing with this project, you need to run the code generation before
[00:14:07]
every compile before every before you start developing whatever, whichever makes sense
[00:14:12]
for your use case.
[00:14:14]
And you probably won't use like vanilla elm make might use Webpack to trigger the code
[00:14:20]
generation or it's just like post install scripts, whatever makes more sense.
[00:14:25]
But you have to set something up so that people in your team don't clone the repo and do elm
[00:14:33]
make and then they get a compiler error.
[00:14:36]
You need to make it easy for them to to get started with the project.
[00:14:39]
Yes.
[00:14:40]
Right.
[00:14:41]
So, yeah, there's sort of like the lifecycle of these generated files because so like there's
[00:14:47]
a particular like conceptually pretty much any any I'm going to go ahead and say any
[00:14:53]
generated code has a source of truth.
[00:14:55]
There's some that's sort of the point of generating code.
[00:14:58]
There's some source of truth that you're mirroring.
[00:15:01]
And depending on what that source of truth is, you that's going to influence when you
[00:15:06]
want to generate it.
[00:15:07]
So like in the example of like a sort of build timestamp, that one is the lifecycle is simpler
[00:15:13]
because you just want to generate a unique one for every build.
[00:15:17]
So, you know, maybe for your dev server, you say, OK, fine, the file needs to exist.
[00:15:22]
But the important thing is just that the file exists and that every time I ship a new thing
[00:15:27]
to production, it gives me unique build ID or timestamp.
[00:15:30]
That's that's the only thing that matters.
[00:15:32]
So for that source of truth, the for that mental model, you just need to be sure that
[00:15:37]
you're generating a fresh one for each build, which the gitignore sort of takes care of.
[00:15:42]
If the source of truth is like, let's say, for example, a GraphQL schema, then in order
[00:15:48]
to mirror that source of truth, you want to rerun the code generation.
[00:15:53]
You want to make sure that it's generated initially upon starting up a dev server or
[00:16:00]
running your production build.
[00:16:01]
But you also want to make sure in dev mode that it's being regenerated as needed ideally.
[00:16:08]
So that does get a little bit trickier in cases like that where you're mirroring a source
[00:16:12]
of truth.
[00:16:13]
You can you know, you could make a ChalkiDAR.
[00:16:16]
You know, ChalkiDAR is like a an NPM tool which gives you a file watcher.
[00:16:21]
They provide there's a ChalkiDAR CLI, which we'll link to in the show notes.
[00:16:25]
That can be a handy way to just say, hey, anytime these files change, like these files
[00:16:30]
are the source of truth.
[00:16:31]
These GraphQL schemas falls.
[00:16:33]
Exactly.
[00:16:34]
Maybe you've got your, you know, your server of choice set up to output a new GraphQL schema
[00:16:40]
file every time that changes.
[00:16:42]
So then that's your source of truth.
[00:16:43]
And you set up the ChalkiDAR watcher to watch that schema file.
[00:16:48]
Anytime that schema file is touched, it's going to rerun your code generation.
[00:16:51]
So that definitely is a challenge.
[00:16:53]
But what you can also do is just have either NPM scripts or make file scripts that every
[00:16:59]
time you rerun the, you start your dev server, it recompiles or regenerates the code that
[00:17:07]
is needed.
[00:17:08]
And then you just tell your teammates that every time your GraphQL schema changes, you
[00:17:14]
need to rerun the dev server.
[00:17:16]
And as long as everyone is aware of these, I guess it's a fine trade off.
[00:17:23]
Right.
[00:17:24]
Right.
[00:17:25]
And yeah, I mean, even with like a GraphQL schema, if it's not changing that frequently
[00:17:30]
or for whatever reason, it's, you know, or maybe you typically make non breaking changes
[00:17:35]
or, you know, in that, in that case, you could say, you know, we're not even going to bother
[00:17:40]
with a file watcher.
[00:17:41]
We're just going to leave it to the individual developer to manually run the generate script.
[00:17:49]
And we're going to guarantee that it's generating it from scratch for every production build.
[00:17:53]
So it's always up to date for production.
[00:17:55]
And if something were to go wrong, we would know because it wouldn't compile.
[00:17:59]
So worst case scenario, it doesn't compile, then the build fails, the developer goes and
[00:18:05]
fixes the problem.
[00:18:06]
So in some cases, you might not need to go to the trouble of, or you could start with
[00:18:11]
something simple.
[00:18:12]
Yeah, but at least consider your teammates and your own productivity.
[00:18:16]
So just like you don't want to make mistakes because your GraphQL schema is out of date
[00:18:22]
with your generated code.
[00:18:24]
So make it easy for all of you.
[00:18:27]
Right.
[00:18:28]
And if you're not going to get a compiler error, if something goes wrong, but instead
[00:18:31]
you're going to have some different values at runtime that weren't what you were testing
[00:18:36]
against when you were developing locally, you know, that's something considered too,
[00:18:40]
that that could cause cause bugs because you're looking at something different in your development.
[00:18:46]
So then what you're shipping to production.
[00:18:48]
So these are things to keep in mind.
[00:18:51]
One of the biggest challenges that I've seen with like doing code generation for LM applications,
[00:18:57]
not libraries is just like coming up with the right mental model.
[00:19:02]
So I think one of the most important things is you really want to have a very simple mental
[00:19:08]
model as much as possible.
[00:19:10]
I think it's really valuable to piggyback on other mental models.
[00:19:15]
So like, like with, with a GraphQL schema, you know, it, it, it seems, it seems obvious
[00:19:22]
if you're not like designing the source of truth, but GraphQL is a very simple mapping.
[00:19:29]
You've got this concept, there's a GraphQL schema, you can make requests to it.
[00:19:33]
And you know, the developers already have a mental model of this GraphQL schema that
[00:19:37]
they can make queries against.
[00:19:39]
And so they can take those concepts and they can map those concepts into a slightly modified
[00:19:45]
concept of, okay, there's this GraphQL schema.
[00:19:49]
I can make HTTP requests like this.
[00:19:52]
And they map that concept to, okay, there's this Elm code and it maps onto the GraphQL
[00:19:58]
schema in this clear one to one mapping like this, right?
[00:20:01]
So that is, um, they do have to sort of map their mental model there, but it's a clear
[00:20:07]
mapping.
[00:20:08]
Another example would be with, um, with Elm SPA, you know, you've got like this file based
[00:20:14]
routing and there's a clear mapping between the wiring that you would do for a single
[00:20:19]
page app in Elm and, uh, the, the module, uh, module names on the file system.
[00:20:25]
Yeah.
[00:20:26]
Elm SPA is a very different story though, uh, because it's not a matter of a source
[00:20:31]
of truth.
[00:20:32]
It's a matter of boilerplate mostly and maybe conventional or making it easy to, to develop
[00:20:38]
it, but it's not a matter of source of truth in this case.
[00:20:41]
I mean, in a sense it is a source of truth as well because it, it enforces the mapping
[00:20:46]
where if you have, yeah, but just like any Elm code in any way, it's like normally in
[00:20:53]
a single page app, the source of truth is whatever you do in your top level main.elm,
[00:20:58]
not your like pages main.elm.
[00:21:02]
But with Elm SPA, you can, you can map what's happening in your file system to, to know,
[00:21:08]
okay, this is going to map to a page that's going to get wired in in this way with this
[00:21:13]
URL routing and you can map all of those concepts to what would actually be happening.
[00:21:18]
So normally in a single page app, the source of truth is what happens in main.elm, which
[00:21:22]
could or could not follow convention.
[00:21:25]
And you don't know until you, until you look at the code and you could make a mistake there.
[00:21:29]
But if you have a code generation tool, it helps keep you honest about that source of
[00:21:33]
truth.
[00:21:34]
So it can actually simplify the mental model because it can express a certain concept more
[00:21:40]
concisely.
[00:21:41]
Yeah.
[00:21:42]
But to me, that is point of tools like Elm SPA is like removing potential failures, potential
[00:21:47]
bugs because you've, yeah, because you forgot to wire things, a page with the main or you
[00:21:54]
did it wrong.
[00:21:56]
And code generation is a pretty good tool for making sure that something that is very
[00:22:00]
similar for plenty of things is done in a very consistent and well done way.
[00:22:06]
Right, right, right.
[00:22:08]
So I think we're, I think we're talking about two different sources of truth here because,
[00:22:11]
you know, my assertion was that anytime you're doing code generation, there is a source of
[00:22:16]
truth and what you pick as the source of truth is going to be important for like understanding
[00:22:22]
like how people using that code generator are going to map that source of truth onto
[00:22:27]
what's actually happening.
[00:22:28]
They need to create a mental map.
[00:22:29]
And what you're talking about is what is the value where Elm GraphQL has the value of like
[00:22:34]
taking the source of truth and letting the Elm compiler know about it and Elm SPA, it's
[00:22:39]
more convenient than keeping these two things in sync.
[00:22:42]
Yeah, exactly.
[00:22:43]
Like Elm SPA, you have a source of truth, which is all the different files in the file
[00:22:49]
system, but they are not a source of truth in themselves.
[00:22:54]
You make them into a source of truth.
[00:22:56]
Whereas GraphQL schema is the source of truth.
[00:22:59]
It is something that can be made out of sync with the Elm project.
[00:23:05]
Right, right.
[00:23:06]
Yes.
[00:23:07]
So there's this book, I actually have to admit, I sort of read half of it and didn't finish
[00:23:13]
it.
[00:23:14]
I really liked the principles in this book, the design of everyday things.
[00:23:19]
It's a really cool book.
[00:23:20]
Are you familiar with it at all?
[00:23:22]
You've read 50% more than I did.
[00:23:28]
And one of the things, like I think about this sometimes when I'm using everyday things,
[00:23:35]
the author, Don Norman talks about sort of the mappings of these mental models.
[00:23:41]
So like, you know, how you can have like a door, basically, he says, if you have a door
[00:23:46]
that, like, have you ever gone into a door and you try pushing it and then you realize
[00:23:51]
you have to pull it?
[00:23:52]
Yeah, like, is it something like, if a door has no handle, then you are expected to push?
[00:23:58]
Right.
[00:23:59]
So those are like the affordances of the door.
[00:24:01]
So there are these like clear physical cues that show how to interact with it.
[00:24:07]
And like, they're not even conventions.
[00:24:10]
They're like, just here's what exists.
[00:24:13]
And so what are you going to try to do with a door with no handle?
[00:24:17]
You're going to push it.
[00:24:18]
Yeah.
[00:24:19]
It's like, what will your instincts tell you?
[00:24:22]
Right.
[00:24:23]
Exactly.
[00:24:24]
Exactly.
[00:24:25]
Because just looking at it, you're like, what are the possible interactions I can do with
[00:24:28]
this?
[00:24:29]
Well, I'm not going to touch the handle because there's no handle.
[00:24:32]
Yeah, there's nothing that you can pull.
[00:24:34]
And so that's, there are these affordances that give cues to how something can be used.
[00:24:39]
If something has a handle, then naturally you want to pull it.
[00:24:42]
If something has like a little metal bar, then you're like, oh, push on the metal bar.
[00:24:46]
And I think it's important to consider like what people's intuition will be interacting
[00:24:51]
with your sort of code generation interface.
[00:24:55]
And that's the source of truth, right?
[00:24:57]
So like consider how your source of truth is giving cues that make it easy for people
[00:25:03]
to understand how to interact with something and what the implications are going to be.
[00:25:07]
Another thing he talks about is like sort of these mappings.
[00:25:11]
Like if you have like two different lights on two different sides and one switch is on
[00:25:20]
the right side and one switch is on the left side and one light is on the right side and
[00:25:24]
one light is on the left side, you could map either switch to either light.
[00:25:29]
But if there's a clear physical mapping, it's going to be easier to interact with that and
[00:25:33]
intuit what the behavior is going to be.
[00:25:35]
So you can create a clear mental model.
[00:25:37]
And I think that that's like I think about this with my with my garage door opener.
[00:25:44]
I always walk into the garage door to the garage to open the door and I want to open
[00:25:51]
the door.
[00:25:52]
I'm looking on the right and there's a button on the left and a button on the right.
[00:25:56]
And I have to always flip it in my head.
[00:25:58]
I'm like, oh, it's the opposite of the intuitive one.
[00:26:00]
I press the left button and I do and my brain always goes through that process.
[00:26:04]
So if you can avoid having, you know, the consumers of your code generation interface
[00:26:10]
go through that mental mapping by creating like my point is just like you are you are
[00:26:17]
creating a mental model that users are going to have to learn and interact with.
[00:26:22]
So be aware of that that mental model that people are going to have to interact with
[00:26:27]
and what the affordances are, what what cues you show how to use something, you know, make
[00:26:32]
impossible states impossible.
[00:26:34]
But make the source of truth have a very clear mapping so that users don't have more concepts
[00:26:41]
to learn.
[00:26:42]
And so it's predictable what it's going to do.
[00:26:43]
It's easier said than done.
[00:26:45]
But ideally, if you can try piggybacking on a mental model that people already have.
[00:26:51]
For instance, for GraphQL, we already have something like decoders.
[00:26:54]
Is that what you're thinking?
[00:26:56]
Yeah, yeah, we have we have decoders and and we have the you know, I mean, also, like,
[00:27:03]
you know, with Elm GraphQL, it's not a perfect one to one mapping of GraphQL.
[00:27:08]
You know, there are certain concepts like field aliases that don't exist in Elm GraphQL
[00:27:14]
because it abstracts those away.
[00:27:16]
But there are other concepts, you know, like objects, interfaces, unions in GraphQL that
[00:27:24]
have a predictable mapping in ways that you know, objects, the interfaces, you know, the
[00:27:30]
APIs for objects are generated in a specific way.
[00:27:33]
And so there are certain conventions that I'm sort of configuring the right switch with
[00:27:40]
the right light and the left switch with the left light where I'm trying to like find a
[00:27:45]
clear mapping to make it intuitive for these concepts to go together.
[00:27:49]
So what are other approachable usages of code generation?
[00:27:55]
So one that I can think of is like generation for images.
[00:28:02]
Like static assets sort of static assets.
[00:28:04]
So often you have like images, logos, icons that are in your file system, your assets
[00:28:09]
folder or something, and you want to reference them in your Elm code.
[00:28:14]
So what you can do is just have a have a string pointing to their location, which you can
[00:28:20]
also do in order to make it more type safe and prevent errors from typos is to create
[00:28:25]
an API for it using code generation.
[00:28:29]
So you create a script which loads the source of truth, which is listing all the icons in
[00:28:36]
your assets folder and then generating a Elm file using that.
[00:28:41]
And then one nice thing that you can do is when you say icon.add button or whatever,
[00:28:48]
you have a code completion with your editor and you have something a lot more type safe.
[00:28:54]
Right.
[00:28:55]
You can even get inline documentation in your IDE if you generate doc comments, which is
[00:29:01]
pretty cool.
[00:29:02]
Yeah.
[00:29:03]
I often think of like code generation is really just you're instead of like writing an API,
[00:29:09]
you're writing something that generates an API for a very specific thing.
[00:29:13]
Yeah.
[00:29:14]
Yeah.
[00:29:15]
I saw on Twitter if people have like small in house code generation tools that they use
[00:29:21]
and there were a lot of interesting use cases people mentioned like internationalization
[00:29:28]
is a cool one to and you can actually you can enforce certain things in in your code
[00:29:35]
generation as well.
[00:29:36]
So you can make impossible states impossible.
[00:29:38]
You can say, you know, if there's a certain like key for your internationalization dictionary,
[00:29:45]
you can say, well, if that key is missing in a particular translation dictionary, then
[00:29:51]
that should be an error.
[00:29:52]
For example, build error or right.
[00:29:54]
Exactly.
[00:29:55]
You can actually write because code generation introduces a build step.
[00:29:58]
So you can actually introduce not just compiler errors, but builders, which is so I get, you
[00:30:04]
know, the concept of making possible states impossible applies not only within the code
[00:30:09]
that you generate, but the process of, well, I'm only going to generate code with a certain
[00:30:14]
known valid structure.
[00:30:16]
And if I can't produce that, I'll give a builder.
[00:30:19]
Yeah.
[00:30:20]
Another common one seems to be just taking like you're describing sort of mapping assets
[00:30:26]
to a particular URL.
[00:30:29]
I think, you know, another common one is environment variables.
[00:30:33]
Yeah.
[00:30:34]
Mm hmm.
[00:30:35]
Like secrets that are injected as a build time in the CI.
[00:30:39]
Although you shouldn't inject secrets because it's going to be in your bundle.
[00:30:44]
That is true.
[00:30:45]
Well, not secrets, but it's production.
[00:30:49]
Everything else.
[00:30:50]
Yes.
[00:30:51]
Mm hmm.
[00:30:52]
Yeah.
[00:30:53]
Wait, I should probably go back to work and remove a few things.
[00:30:56]
You should go remove those secrets.
[00:31:00]
So we've already covered a few projects that use code generation.
[00:31:05]
So we've covered LMSPA.
[00:31:07]
We've covered LMS tailwind modules.
[00:31:09]
Yeah.
[00:31:10]
I've seen a few other libraries which do code generation as you are probably familiar with
[00:31:17]
because that's your primary intents for code generation apparently.
[00:31:21]
So chat tech has an Elm vector package, which is a vector is like a list or an array of
[00:31:29]
a fixed size.
[00:31:31]
And it has a package which has vector one, vector two, vector three, up until vector
[00:31:38]
50.
[00:31:39]
So all of those are modules with functions that go well with each of those.
[00:31:46]
Sorry, until 60.
[00:31:48]
And they're using the same code for every one of those.
[00:31:51]
And it would be tedious and error prone to write them by hand.
[00:31:56]
So what Chad did there is write a script that generates those modules.
[00:32:01]
And then he stopped at arbitrarily 60 modules.
[00:32:06]
So Rupert Smith created a few packages for working with AWS.
[00:32:11]
So he uses Elm, sorry, he uses Elm AWS code gen, which in turn uses one of his packages,
[00:32:19]
which is called Salix, which I don't know much about it, but he seems pretty excited
[00:32:24]
about it, which takes the, not the documentation, but the API definitions from AWS services,
[00:32:31]
and then creates packages to work with that.
[00:32:34]
So he has one to work with S, no, not with S3, with authentication, with the Cognito
[00:32:42]
service and Elastin containers.
[00:32:45]
So you can, he did a good job apparently from, for generating APIs that work well with those
[00:32:52]
services.
[00:32:53]
I don't know if it's well made in the sense like it has the Elm philosophy of preventing
[00:33:03]
impossible states or if it's just a type safe way to create AWS service APIs.
[00:33:10]
But I think at least as a base tool, which allows you to do anything, it's very interesting.
[00:33:17]
Interesting.
[00:33:18]
Yeah, I haven't used that one.
[00:33:21]
So another code generation tool, I've done so much code generation, I'm like losing track
[00:33:27]
of all the code generation I've done.
[00:33:30]
But Elm TS Interop is a code generation tool.
[00:33:33]
It happens to generate, rather than generating Elm code, it's actually generating TypeScript
[00:33:38]
declaration files.
[00:33:41]
But it's that same sort of principle of having a common source of truth and keeping those
[00:33:46]
in sync.
[00:33:47]
It's just that it works the other way around.
[00:33:49]
Instead of taking some external schema and making the Elm compiler aware of it through
[00:33:55]
generating Elm code, it's taking your Elm decoders and encoders and it's making TypeScript
[00:34:03]
aware of those so you can use those ports.
[00:34:07]
It actually, the pro version uses some code generation to make it a little bit more convenient
[00:34:14]
so you can have a module where you just declare these top level values in your sort of port
[00:34:19]
definitions file and then it takes those and for every top level exposed encoder, it generates
[00:34:28]
a port from Elm to JavaScript and for every top level decoder, it generates from JavaScript
[00:34:36]
to Elm ports unless its name is Flags, in which case it generates your Flags decoder.
[00:34:42]
So now there's another type of code generation which we haven't gotten into yet, which is
[00:34:47]
for scaffolding.
[00:34:49]
I was expecting that one.
[00:34:51]
Yeah.
[00:34:52]
So these are like, there's like, what's the mental model?
[00:34:55]
And then there's like, what is the purpose of the code generation?
[00:34:59]
So we've talked about keeping your sources of truth in sync, but scaffolding is another
[00:35:05]
really interesting area.
[00:35:07]
So you've got some examples written down of scaffolding here.
[00:35:10]
Yeah.
[00:35:11]
So Elm SPA is one of those.
[00:35:13]
It creates a new project with a bunch of predefined pages and files and Elm JSON.
[00:35:21]
So a lot of things already.
[00:35:22]
Elm review also has a new rule and a new package command or commands.
[00:35:28]
So a new package creates a new project entirely.
[00:35:31]
It's not just a git clone.
[00:35:33]
It's really generation because you are giving an information and then it generates a project
[00:35:39]
based on that.
[00:35:40]
And same thing for the new rule, you give it a rule name and generates some scaffold,
[00:35:45]
some base files with the information that you gave it and changes the rest of the files
[00:35:50]
that are expected to be there.
[00:35:52]
So yeah, scaffolding in this case is more about getting started rather than having a
[00:35:58]
source of truth, I think.
[00:36:00]
Yes, right.
[00:36:01]
It gives you a convenient starting point, but it's for the scaffolding approach, you're
[00:36:06]
generally using it, like you said, to get started, but you're not going to rerun it
[00:36:10]
every time to keep it in sync.
[00:36:12]
So the source of truth, like, so like html to elm.com, which we talked about in our Elm
[00:36:18]
Tailwind Modules episode, it's this tool that I built that generates Elm code from HTML
[00:36:23]
and it also can parse out your Tailwind Modules to generate code for that.
[00:36:29]
So it's a code generator for a code gen tool.
[00:36:34]
Because why not, turtles all the way down.
[00:36:36]
But yeah, that's, you know, the source of truth for that, you know, the mental model
[00:36:41]
is here's this HTML, it's got a clear mapping to Elm code and my Elm review rule as well,
[00:36:48]
it's sort of, you know, able to hook in even more to the source of truth of your, like
[00:36:53]
your imports in a particular module and take that context, which is cool.
[00:36:57]
But it's not something that you're generally going to keep around that starting HTML and
[00:37:02]
then edit the HTML anytime you want to change your templates.
[00:37:05]
It's like, hey, let me help you get started with this view.
[00:37:09]
But then you own it from there, rather than, hey, let me keep around this HTML file.
[00:37:15]
I'm going to run the code generator on that every time to get this Elm code.
[00:37:19]
And I actually never touch that Elm code.
[00:37:21]
If I want to change the view, I go change the HTML file, which is the source of truth.
[00:37:26]
That's something different.
[00:37:27]
That's not the scaffolding approach.
[00:37:29]
So when would you go for the scaffolding and when would you go for code generation?
[00:37:33]
That's a good question.
[00:37:34]
I think it's really like, if you can accomplish a task without human intervention, then you
[00:37:42]
may as well just code generate it.
[00:37:43]
But like with a view, you know, if you were to just like, you couldn't just say, here's
[00:37:49]
some HTML, please map this into Elm code, it would defeat the purpose.
[00:37:53]
Because the point is like, you want Elm code.
[00:37:55]
So you can say list.map over these things to create a list item for each of these things
[00:38:02]
from the model or whatever.
[00:38:04]
And well, so what are you going to do?
[00:38:05]
Create like a templating language within your HTML code?
[00:38:09]
I mean, you could if you wanted to.
[00:38:11]
But in that case, most of the time, you're just going to want something to help you out
[00:38:15]
and say, oh, here's some HTML that I'm copying from this handy online template.
[00:38:22]
It's like tailwindui.com.
[00:38:24]
You've got a nice template I want to use, I want to copy paste it as my starting point,
[00:38:27]
and then take ownership of the code.
[00:38:30]
So at Humia, we also used code generation, kind of like Elm SPA to generate all the boilerplate
[00:38:37]
for linking the main file with the individual pages.
[00:38:41]
And one reason for that was sure to remove any errors that could be made because you
[00:38:48]
forgot to wire something.
[00:38:50]
But mostly because it was much more performant in a sense.
[00:38:54]
When you add a file, you had to add a new constructor to a custom type, which was like
[00:39:01]
type page equal homepage, etc.
[00:39:05]
And that was actually pattern match in 2030.
[00:39:10]
I don't know how many pattern matches.
[00:39:12]
So you would have to add a constructor and then spend 10, 20 minutes just fixing compiler
[00:39:19]
errors because, hey, you forgot to handle this constructor here and there and there.
[00:39:25]
For code that was very similar to what was next to it.
[00:39:29]
So you would pretty much copy paste some code, go to another place, copy paste some code
[00:39:34]
and adapt it to use the constructor.
[00:39:36]
And that 20 or 30 times.
[00:39:38]
If you were well versed with the code base, that would take you 10 minutes.
[00:39:43]
If you weren't, then it would take longer.
[00:39:46]
And that was just a lot of time not well spent.
[00:39:50]
So by refactoring and making it work with code generation, now adding new pages just
[00:39:58]
takes a few seconds.
[00:39:59]
We even have like a script that creates the file and now we can focus on more important
[00:40:06]
things.
[00:40:07]
Cool.
[00:40:08]
Yeah.
[00:40:09]
When you do code generation, you can abstract certain details.
[00:40:12]
So again, it's an important distinction if you have code that a human is then modifying
[00:40:21]
versus the human never touches this code.
[00:40:24]
It's generated and machine maintained.
[00:40:26]
I think it's very important to have a clear distinction between those.
[00:40:31]
So for example, the code that Elm GraphQL generates for your GraphQL schema, I even
[00:40:37]
put like auto generated code, don't modify.
[00:40:40]
And in fact, I think I have an error there.
[00:40:44]
If you somehow went in and tweaked it, it would say, looks like a human changed this.
[00:40:49]
So delete the file and then I'll go regenerate it.
[00:40:53]
But I'm not going to touch this file because it looks like a human modified it.
[00:40:56]
Wait, how do you do that?
[00:40:57]
Like when you regenerate, you say, oh, this file has been changed?
[00:41:00]
I can't remember exactly what I did.
[00:41:02]
I mean, obviously it's sort of like a heuristic, but I don't know.
[00:41:08]
If the file doesn't have that comment that I generate on top of every file or something
[00:41:12]
like that, then I say, it looks like this isn't the file I created.
[00:41:16]
And I don't want to touch, like I just am going to blow away anything.
[00:41:20]
So make sure that this folder is a clean working space for me before I do my job, because I'm
[00:41:27]
just going to blow everything away with disregard for everything that a human touched.
[00:41:33]
But I think that's very important that there should be a clear separation.
[00:41:37]
Now, obviously humans use that code, but they shouldn't touch the generated code.
[00:41:44]
You should either have code generated that's entirely maintained by the code generator,
[00:41:49]
or you should have code that's scaffolded and then you take ownership over it.
[00:41:55]
Or parts of it.
[00:41:56]
Like you can generate part of it, scaffold another, and then use those together.
[00:42:00]
What would be an example?
[00:42:01]
I don't know.
[00:42:02]
I haven't encountered that.
[00:42:03]
I mean, like if there's something, if there are parts of it that you, that could make
[00:42:10]
use of code generation and parts of it that need to be changed by someone, then you generate
[00:42:16]
the one.
[00:42:17]
I see.
[00:42:18]
Right, right, right.
[00:42:19]
You generate the former and scaffold the latter.
[00:42:22]
Yes, you can certainly mix them within a project.
[00:42:25]
But it's good to be clear about like a file should either be generated or not.
[00:42:32]
Like it should either be maintained by a code generator or not.
[00:42:34]
And it's good to not mix those up.
[00:42:36]
Now it is, and another interesting thing that can happen is you can use actual Elm code,
[00:42:44]
like parsing the AST as the source of truth.
[00:42:47]
But that can definitely get complicated, but that can be a good way to keep the, that can
[00:42:54]
be a good way to piggyback on top of a certain mental model.
[00:42:56]
Cause you say, okay, well I know when I, when I write a decoder that does this, I expect
[00:43:02]
it to work this way.
[00:43:03]
And then you write some code generation that generates something to in support of that.
[00:43:07]
But that can definitely get messy because now it's starting to feel more like metaprogramming
[00:43:14]
where you change some Elm code and you're like, Oh, I can rename this.
[00:43:18]
I can change this and things start to break and unexpected ways.
[00:43:21]
So it can remove some of the guarantees in a certain way, which is why the mental model
[00:43:26]
is really important because you, you, if you can, if you can make it so you can map very
[00:43:31]
directly onto an existing concept, then somebody could say, well, if I make this change, I
[00:43:38]
expect it to not break.
[00:43:39]
If I make this change, I expect it to break.
[00:43:41]
And you could actually retain that property, that guarantee because you've mapped onto
[00:43:46]
that mental model.
[00:43:47]
So what are your tips and tricks that you've learned with code generation?
[00:43:52]
One of my favorite tips, which I've probably talked about it some, actually maybe I haven't
[00:43:57]
talked about it on Elm radio yet.
[00:43:58]
My biggest tip is end to end testing your generated code.
[00:44:02]
It's awesome.
[00:44:03]
Um, I've definitely, you know, given, given you my rant on this before, but, um, I love
[00:44:10]
it.
[00:44:11]
Uh, it's so easy to get confidence about your generated code.
[00:44:15]
Basically the workflow is, you know, write an end to end test, which is some sort of
[00:44:19]
snapshot test.
[00:44:20]
So somehow you're saying, um, like with Elm GraphQL, for example, you've got this, this
[00:44:25]
is the code it generates given this schema.
[00:44:28]
And so in your, you know, in your build, you, you have like a test schema.
[00:44:33]
You say, you know, I mean, with Elm GraphQL, I started with like the, the GitHub schema,
[00:44:39]
the simple Star Wars example that they use for all the GraphQL stuff.
[00:44:42]
I had a few schemas and I said, okay, I'm going to actually run on my build server.
[00:44:47]
Um, I'm going to generate these schemas and I'm gonna, I'm gonna compare the code that
[00:44:53]
was generated before with the code that's generated for this build.
[00:44:57]
So every time it changes, I need to check in the changes.
[00:45:00]
That's kind of the snapshot testing approach.
[00:45:02]
And you're also running actual tests with the generated code, I'm guessing.
[00:45:07]
Sometimes, uh, yeah, sometimes I do that as well.
[00:45:10]
And do you generate the code and do you generate the test code?
[00:45:13]
Well, I do that sometimes.
[00:45:15]
With the Elm GraphQL generated code, I essentially just, uh, what I do is I, when, when I was
[00:45:22]
initially building it, I started by first, I would manually generate the code, quote
[00:45:28]
unquote, generate, uh, that I wanted it to look like.
[00:45:31]
So that was sort of the code gen target and then use that.
[00:45:35]
And then I, what I did is I wrote examples in the examples folder and those were sort
[00:45:40]
of manual tests.
[00:45:42]
So you know, every time I'm like adding a new feature or changing something, I go through
[00:45:46]
and make sure all the examples are working as expected.
[00:45:49]
And not only does, am I testing the behavior, but I'm testing the, I'm testing the user
[00:45:56]
experience.
[00:45:57]
So I'm making sure that interacting with it feels nice.
[00:46:00]
The types look good.
[00:46:02]
Could I get it any simpler?
[00:46:03]
Is it intuitive?
[00:46:04]
And then once I've got that working with the hand generated code, I check that in as my
[00:46:09]
snapshot and then I iterate on that failing test until I get, I get it green with the
[00:46:15]
actual code generation target being what I'm generating.
[00:46:18]
So approval testing or snapshot testing is awesome for, for code generation.
[00:46:23]
Right.
[00:46:24]
Yeah.
[00:46:25]
I also really like, so this is almost like a whole separate type of code generation,
[00:46:29]
but as you were hinting at testing generated code is really nice.
[00:46:33]
Like for, for LMTS Interop, I've got, I've got this tool for the pro version, which is,
[00:46:41]
you know, you, you write some TypeScript type definitions in a little sort of VS code TypeScript
[00:46:49]
embedded editor in the browser.
[00:46:51]
And it generates LMTS Interop, you know, encoders and decoders based on that, that are going
[00:46:58]
to yield the same TypeScript types.
[00:47:00]
And you know, there's a lot going on there.
[00:47:02]
It's non trivial, like that project is non trivial and it would be really easy to mess
[00:47:07]
something up.
[00:47:08]
So I, I generate a test suite.
[00:47:11]
So I for, for, I have some like sample input, which is like TypeScript files and I generate
[00:47:18]
a test.
[00:47:19]
So I actually generate like an Elm test test suite for each of those examples.
[00:47:24]
So I just have an examples folder.
[00:47:26]
I write it, you know, I write a dot TS file and it actually takes that code.
[00:47:30]
It feeds it into this code, which is, you know, Elm code to generate this stuff.
[00:47:36]
Make sure that it, make sure that it runs, it does a reversible test and feeds input
[00:47:40]
in and it says that if you run the encoder and the decoder, you get the same value.
[00:47:45]
So it's generating an Elm test test suite for each of those.
[00:47:47]
So, so generating Elm test test suites is awesome.
[00:47:53]
That's that's a technique I like to use.
[00:47:55]
I do that for HTML to Elm.com as well.
[00:47:59]
I'll put a link to the source code for that example.
[00:48:02]
So I what I do is I, I generate for a bunch of different HTML inputs.
[00:48:08]
I generate a test suite and I, and I make sure that it's compiling.
[00:48:12]
So the generated code should be compiling with the Elm tailwind modules default published
[00:48:17]
package.
[00:48:18]
So that's pretty handy.
[00:48:19]
And I mean, we should point out like Elm test itself is just a code gen tool actually.
[00:48:25]
Yeah.
[00:48:26]
People don't, people don't think about it, but.
[00:48:28]
Yeah, it generates a an Elm file and then compiles it.
[00:48:33]
Yeah.
[00:48:34]
Right.
[00:48:35]
Yeah.
[00:48:36]
They're like Elm explorations tests does have some kernel code, I think, but a lot of it
[00:48:40]
is just using code generation.
[00:48:43]
Yeah.
[00:48:44]
Which is the same for like for Elm verify examples, which I think we've talked about.
[00:48:49]
It's a really handy package that lets you write your, your examples in your Elm doc
[00:48:55]
comments that it will actually execute and compare your expected output that you write
[00:49:01]
with a little comment notation in the docs.
[00:49:03]
It creates a test for each of those, right?
[00:49:05]
Yeah, exactly.
[00:49:06]
That's all it does.
[00:49:07]
You can go in and look at it and then it just runs it with, with Elm tests.
[00:49:10]
So you know, that's, I would definitely recommend considering, considering that technique if,
[00:49:16]
if an opportunity presents itself, like I've, I've used that both in libraries I maintain
[00:49:21]
and applications.
[00:49:23]
It's a really handy technique sometimes.
[00:49:25]
Yeah.
[00:49:26]
I've wondered about doing something like very similar to that for Elm review, like give
[00:49:31]
it, give us a set of examples and then compare the error that you get with what you would
[00:49:36]
really get.
[00:49:37]
Yeah.
[00:49:38]
But it's a bit more tricky than I think like, do you really want to compare the errors?
[00:49:46]
Yeah.
[00:49:47]
Interesting.
[00:49:48]
Code generation can be really nice for end to end testing things.
[00:49:51]
So for for Elm pages 2.0, I've been considering taking the, the routes.
[00:49:59]
So I'm trying to, I've gotten rid of routes in this generated code.
[00:50:03]
I used to have generated code that had all of the routes, but then if you have a thousand
[00:50:08]
routes, then you've got something in your generated code that would you, if you ever
[00:50:12]
refer to it anywhere, now you've pulled all that into your bundle.
[00:50:16]
So it's quite nice if you can avoid pulling that into your bundle and instead avoid incurring
[00:50:21]
that cost at runtime and in your bundle size and incur that cost at build time.
[00:50:27]
Well, it's a minimal cost to having a thought, you know, reference to a thousand files by
[00:50:32]
using Elm review.
[00:50:33]
So I've been thinking about adding some sort of hook that basically gives here's what I've
[00:50:39]
been thinking about.
[00:50:40]
Actually, tell me what you think of this.
[00:50:41]
What I've been considering is, so there's like, there's a route type that's generated.
[00:50:45]
So Elm pages 2.0 similar to Elm SPA has this file based routing.
[00:50:49]
It generates a route dot Elm module, which enumerates, you know, you've got your blog
[00:50:54]
route, you've got your index route.
[00:50:57]
You know, some of the routes have route parameters.
[00:50:59]
Some of them don't have route parameters.
[00:51:00]
If it's a single page for the ones that have route parameters like blog, well, if you've
[00:51:03]
got blog, slug, post one, slug, post two, that are your valid static routes.
[00:51:11]
And you want to say, well, I don't want to link to post negative one because that doesn't
[00:51:16]
exist or post zero that doesn't exist.
[00:51:18]
So I want to use an Elm review rule to check all of the static routes.
[00:51:23]
If they're static pages, then I want to check if, if it exists.
[00:51:27]
Right.
[00:51:28]
So, well, I, before with Elm pages 1.0, I was generating, you know, I was generating
[00:51:34]
a record that referenced all of those URLs and you could use that to in a type safe way
[00:51:40]
refer to static routes.
[00:51:41]
Now I just have these route variants.
[00:51:43]
So the blog route is going to be a single route and it's not going to, if you have a
[00:51:49]
hundred blog posts, it's not going to generate something for each of those.
[00:51:53]
So what I'm thinking is generate, uh, like have some sort of, um, CLI command for Elm
[00:51:59]
pages where I can generate a module that, that you can use in an Elm review rule to
[00:52:05]
verify that the routes are static.
[00:52:07]
So it would be, so basically before running Elm review, you would first have to run this
[00:52:13]
generated code because the Elm review rule would depend on that generated code.
[00:52:17]
Yeah.
[00:52:18]
I think that's a, that's a good idea.
[00:52:19]
I actually did something very similar recently to, for, for work where I basically detect
[00:52:26]
unused CSS project.
[00:52:28]
So we have a bunch of CSS files and we have a lot of dead code in there because it's CSS.
[00:52:35]
So what I did is I go through all of those, find all the classes and then generate an
[00:52:40]
Elm file.
[00:52:41]
And then I use that as a source of truth for a rule that I called no unused CSS classes.
[00:52:49]
Boring.
[00:52:50]
And then wherever it would find a class, it was marked as used and the ones that it couldn't
[00:52:56]
find marked as unused.
[00:52:57]
So using that and another rule where I disallow dynamic classes like homepage dash dash plus
[00:53:06]
plus modifier, like homepage constructing CSS classes, I disallow.
[00:53:12]
So with those two, two rules in place, I was able to remove all the unused CSS classes
[00:53:17]
from the project, which was about 4,000 lines of code.
[00:53:19]
So that was nice.
[00:53:20]
Wow.
[00:53:21]
Oh, that's fascinating.
[00:53:22]
Okay.
[00:53:23]
Yeah, that's cool.
[00:53:24]
That's a, that's a good use case for, for code generation.
[00:53:27]
So yeah, I mean, basically like Elm review could get arbitrary access to files and look
[00:53:34]
at those and parse the, but I mean, honestly, then you'd have to write like a CSS parser
[00:53:39]
in Elm and et cetera, et cetera, which you could probably easily find an NPM package
[00:53:44]
that parses CSS and does all those things.
[00:53:47]
So yeah.
[00:53:48]
And then the watch mode can be a bit different, like, Oh, when this, which files should I
[00:53:54]
check?
[00:53:55]
Right.
[00:53:56]
Yeah.
[00:53:57]
Yeah.
[00:53:58]
But th th those are questions that I have, uh, at the top of my mind, but yeah.
[00:54:02]
Yeah.
[00:54:03]
I'll also, I also want to like generate files using Elm review so that you can like, yeah.
[00:54:10]
Like you say, this is a safe, unsafe value type of thing.
[00:54:14]
Yeah.
[00:54:15]
For instance, or I can auto fix the CSS files, like give me the CSS files.
[00:54:21]
I will parse them and look at the ones that the classes that have been, that are not used
[00:54:27]
and I will remove them for you.
[00:54:28]
And I just write those back to a file to the original file.
[00:54:33]
But that has a lot of challenges, but, um, it's something that I am interested in exploring.
[00:54:38]
So a few tips that I have for when you do code generation is one, I would move all the
[00:54:45]
generated files to a generated folder, like source gen or generated slash, whatever with
[00:54:52]
Elm review.
[00:54:53]
Uh, since we're on the topic, a lot of people don't include the generated codes in the files
[00:54:58]
that Elm review should look at.
[00:55:00]
So they do Elm review source slash and test slash through that.
[00:55:05]
It only looks at those files because they don't want errors for the, for the generated
[00:55:09]
code.
[00:55:10]
And that makes sense.
[00:55:11]
But the problem is that you, you're then limiting the amount of information that Elm review
[00:55:15]
has.
[00:55:16]
So it will not know the contents of some generated files, which it will need for some rules.
[00:55:23]
So what I recommend is never calling Elm review with arguments like source or tests, but instead
[00:55:29]
to use ignore errors for directories in your review config and ignoring the generated code,
[00:55:36]
same for vendor code.
[00:55:38]
Like you don't want to report errors for those.
[00:55:41]
And this is like almost half of the bug reports that I get are false positives because they
[00:55:47]
didn't, they limited the amount of information that the rules could have access to.
[00:55:53]
Another thing that I would really recommend is like at the top of every file that you
[00:55:59]
generate, add a comment saying, this is how I generated the file.
[00:56:04]
Like to generate this file, I used this script or this file.
[00:56:09]
That way people who will look at the generated file will know, Oh, if I need to change something,
[00:56:14]
I need to go there because sometimes like it is not obvious where it comes from.
[00:56:19]
If you can indicate the source of truth as even better.
[00:56:22]
So if you are generating like icons for your, that are in your assets, you can add a comments
[00:56:28]
or documentation for each of the functions that you generate that says, Hey, this represents
[00:56:35]
icon, blah, blah, blah, which is at assets slash whatever.
[00:56:40]
So I think that's always pretty nice to have and it really doesn't cut costs a lot.
[00:56:46]
Right.
[00:56:47]
Yeah.
[00:56:48]
It's just, it's just one of those things that's sort of a habit because while, while you're
[00:56:50]
in there, you've got the context, you've got the data you need, just output it in the generated
[00:56:55]
code in a comment.
[00:56:56]
Also like do you format your generated code?
[00:57:00]
What I try to do usually is to make the generator code look kind of like Elm format and that
[00:57:06]
is usually enough.
[00:57:07]
I find.
[00:57:08]
Yeah.
[00:57:09]
That's usually the first thing I reach for, but it, it, it, it depends on the, on the
[00:57:15]
use case.
[00:57:16]
Sometimes I, I format it.
[00:57:18]
I mean, you can get pretty close to Elm format.
[00:57:21]
Right.
[00:57:22]
Well, depending on how simple it is, if it's like our build timestamp, then it would be
[00:57:26]
overkill to run Elm format because you just, you know, you're just generating a simple
[00:57:31]
hard coded thing.
[00:57:32]
Just use your Elm format target.
[00:57:34]
It is fine to run Elm format, but it's, it is not as fast as just not running it.
[00:57:39]
Yeah.
[00:57:40]
And it's another moving part.
[00:57:41]
And yeah.
[00:57:42]
Yeah.
[00:57:43]
What do you think about things like, you know, if you wanted to generate an enum, so you've
[00:57:47]
got like an Elm custom type and you want to have access to, you know, let's say like colors
[00:57:55]
and you want to have all colors is a list of that enum, which is like the colors in
[00:58:01]
your app.
[00:58:02]
And then you want to have from string to string.
[00:58:05]
What do you think about using that for, for code generation?
[00:58:08]
You mean generating that, that file or?
[00:58:11]
Generating that file.
[00:58:13]
And what would you use for the source of truth for that?
[00:58:15]
Well, it really depends on what your source of truth is.
[00:58:18]
Like if your source of truth is what your designer gives you, like a Figma file or,
[00:58:23]
or similar, then if you can get the source of truth as a file or as an API call, I guess,
[00:58:30]
then sure.
[00:58:31]
That sounds like a good idea.
[00:58:35]
When you have something like all colors, like a list of all colors, I would worry about
[00:58:40]
the order.
[00:58:41]
Like it, does the order matter?
[00:58:43]
If so, in which order should you put it?
[00:58:46]
And yeah.
[00:58:47]
Right.
[00:58:48]
Yeah.
[00:58:49]
That's a good reminder of just like, it really does depend on the use case.
[00:58:54]
And, and that, I think that's all the more reason to, you know, take advantage of this
[00:58:59]
for your own use cases to do code generation, because if you're just relying on community
[00:59:05]
code generation tools only, they don't know about your data source and your specific domain
[00:59:12]
and you do, and you can build in those, that understanding of those constraints.
[00:59:17]
So take advantage of that.
[00:59:20]
And you know, I mean, build simple abstractions reach for non code generated solutions first,
[00:59:25]
but if there's some source of truth, you can keep in sync with, if there's some helpful
[00:59:29]
scaffolding tool you can build for yourself, go build it and, you know, take advantage.
[00:59:34]
Okay.
[00:59:35]
So there's one more thing we didn't touch on.
[00:59:38]
We sort of hinted at, you know, doing simple string templating to generate, you know, our,
[00:59:44]
our build timestamp.
[00:59:45]
So you know, you can do that with a simple node script.
[00:59:48]
Yeah.
[00:59:49]
Like if I write it in, in Elm, I even just go for string concatenation.
[00:59:54]
Like I don't even go, I don't even go for like a string where I replace things.
[01:00:00]
I just go for string concatenation.
[01:00:02]
It's just very easy because Elm is a simple language, so it's easy to generate also.
[01:00:07]
Right.
[01:00:08]
Well, yeah.
[01:00:09]
And that, you know what I find extremely helpful.
[01:00:12]
This is going to shock you, Jeroen.
[01:00:14]
But what I find very helpful for keeping myself honest about using the simplest thing that
[01:00:20]
could work is starting with hard coding.
[01:00:22]
Oh, I'm not surprised.
[01:00:23]
I was teasing.
[01:00:25]
Yeah.
[01:00:26]
I'm sure, I'm sure our listeners are bored of me saying this at this point, but I find
[01:00:31]
hard coding extremely useful for code generation as well.
[01:00:35]
Just because validate that the thing you're generating, I mean, first of all, don't even
[01:00:40]
generate something, just write a module.
[01:00:42]
Does it work the way you want?
[01:00:43]
Oh, nice.
[01:00:44]
It would be great if I had this module generated.
[01:00:46]
Okay, well, write a JS file, generate it.
[01:00:50]
Or if you're going into Elm to do it, set up something that runs some Elm code, generate
[01:00:55]
it from Elm just as a single hard coded string, and then pull out the pieces that you need
[01:01:00]
to remove the hard coding from.
[01:01:02]
And well, do that the simplest way you can.
[01:01:04]
If that's with string concatenation, great.
[01:01:07]
If you want to use string interpolation with a library for that, like Luke's string interpolation
[01:01:13]
library, great.
[01:01:15]
If you want to get really fancy, you can use Rupert's Elm syntax DSL to generate code,
[01:01:21]
but it really shouldn't be the first thing you reach for.
[01:01:23]
At least that's not how I like to do it.
[01:01:25]
I like to do the simplest thing that could possibly work.
[01:01:28]
If the particular problem you're solving, it would be helpful to generate it through
[01:01:32]
an AST DSL, then pull that abstraction in when you need it.
[01:01:37]
And if you've got approval tests helping you, you can tweak the way you're generating your
[01:01:41]
code and if you haven't changed your generation output, it doesn't matter how you arrived
[01:01:46]
at it.
[01:01:47]
You just rerun your approval, your snapshot test, and it tells you nothing's changed and
[01:01:51]
you're good to go.
[01:01:52]
I'd like to touch on one last thing.
[01:01:54]
So a lot of people want to use Elm review to forbid writing code one way.
[01:02:01]
For instance, the page boilerplate that you have in your main code.
[01:02:07]
Sometimes I hear people saying, oh, it would be useful to have an Elm review rule that
[01:02:12]
checks whether the boilerplate is well written and does not have an error that you didn't
[01:02:17]
forget to do this and that you did this exactly that way.
[01:02:22]
And this is one instance where code generation can be useful, as I explained before, because
[01:02:27]
like Elm review rules are very good at finding things that are wrong, but you need to specify
[01:02:34]
which ones you're expecting.
[01:02:36]
Like it needs to look for one specific thing, another specific thing, but it cannot tell
[01:02:43]
you whether things are okay as a general thing.
[01:02:47]
Maybe if you multiply the number of rules or the number of checks, but if you want your
[01:02:51]
code to be written in one very specific way or maybe a few, but a limited number of ways,
[01:02:59]
then code generation is probably a better bet.
[01:03:01]
Right, because it can enforce consistency in a different way, in a simpler way.
[01:03:08]
And also it will be less painful to write.
[01:03:11]
It's easier to just run a script that generates a code rather than writing the code yourself,
[01:03:16]
which takes time, as I mentioned before, and then have Elm review tell you, oh, you did
[01:03:22]
this wrong.
[01:03:23]
Oh, you did that wrong.
[01:03:24]
Oh, and then you got a compiler errors.
[01:03:25]
Oh, well, yeah, just run the script and you're done.
[01:03:28]
If you can do that, I think that's a good idea.
[01:03:31]
Right, right.
[01:03:32]
And if you have to remember to do one thing, one place, another thing, another place, another
[01:03:36]
thing, another place, that might be a cue to consider code generation.
[01:03:40]
Although there's certainly a cost and we should reiterate, code generation does add complexity.
[01:03:46]
It does add moving parts.
[01:03:48]
It does make it harder to trace why something's behaving a certain way.
[01:03:52]
If something goes wrong, do we remember to generate this code?
[01:03:55]
Is the source of truth being pulled in correctly?
[01:03:58]
Or just more things, more moving parts?
[01:04:01]
Also, it works really well for things that are very similar.
[01:04:05]
When I did the code generation for the pages, one problem is that a lot of pages were working
[01:04:10]
slightly differently.
[01:04:12]
Their update function took another argument.
[01:04:15]
Their view function didn't take that argument.
[01:04:18]
And I had to do a lot of refactoring work to make them all consistent, to have them
[01:04:22]
all use the same abstraction.
[01:04:25]
But once I got that, generating the code was very easy.
[01:04:29]
But it didn't work.
[01:04:30]
I couldn't have made that code generation work until I got to a place where all the
[01:04:35]
pages were using the same abstraction, where they were very similar to each other.
[01:04:40]
Yeah.
[01:04:41]
And I think it takes experience to be able to see, number one, where you could make those
[01:04:46]
things become similar first.
[01:04:48]
And number two, when they won't then later start diverging and needing to be different
[01:04:54]
again.
[01:04:55]
Because if you start doing code generation and now you're like, oh, well, yeah, the code
[01:04:58]
generation worked really well for this simple case.
[01:05:01]
But for this more nuanced case, there's not a clear source of truth.
[01:05:05]
So I can no longer generate this thing.
[01:05:07]
You can run into problems there.
[01:05:09]
Sure.
[01:05:10]
But the thing is, if you want to opt out of code generating, well, you have Elm code.
[01:05:16]
So you just move it and you check it in your Git repo.
[01:05:21]
True.
[01:05:22]
Yeah.
[01:05:23]
And probably the biggest thing to watch out for is if your code generation is creating
[01:05:28]
a whole set of abstractions that people have to learn, that's a red flag.
[01:05:33]
Because now you're getting bound to this other abstraction.
[01:05:37]
Unless it's a better abstraction.
[01:05:39]
If it's a better abstraction.
[01:05:40]
But I've seen code where there's code generation happening from something where you have to
[01:05:47]
piece things together just right.
[01:05:49]
And if they don't get pieced together just right, the generated code is not going to be happy.
[01:05:53]
Oh, yeah.
[01:05:54]
And that's a bad time.
[01:05:56]
And now, well, how do you back out of that and eject that?
[01:05:59]
Well, you're depending on this extremely complex generated code that you're not going to go
[01:06:05]
maintain by hand.
[01:06:06]
And actually, that's another tip too, is try to keep the generated code as simple as possible,
[01:06:11]
ideally, if you can.
[01:06:13]
You don't want to have to go and debug generated code.
[01:06:16]
Because that would mean that you have to debug the code generation scripts, which is not
[01:06:21]
as easy as debugging Elm code.
[01:06:23]
Right.
[01:06:24]
And oftentimes, if you can, generate intermediary little helper functions or APIs for the generated
[01:06:31]
code to use.
[01:06:32]
So that doesn't vary.
[01:06:34]
So you can just say, here's some code that's only used by the generated code, but it reduces
[01:06:39]
the actual code that we're generating, because it can leverage those APIs.
[01:06:45]
All right.
[01:06:46]
So you covered everything about code generation?
[01:06:49]
Well, we've certainly covered a lot.
[01:06:52]
And yeah, hopefully people will give it a try and let us know how it goes.