Hello Dillon.

Well Jeroen, there are some topics that are hot off the presses and you gotta hit them while they're hot.

And some topics never get old.

You might even call them evergreen.

And I'd like to think that our topic today is a bit of an evergreen topic.

So, Mario Rojic, welcome back to the show.

Hey guys, thanks for having me.

And what are we talking about today?

So yeah, I guess we're talking about the idea of evergreen elm.

So, type safety between versions of your application.

Or at least that's my interest in evergreen elm.

Maybe there's other kinds of evergreen elm that people are interested in.

That's a nice tagline for it.

So, evergreen elm.

And we were talking before the recording here about different species, different variants of elm tree.

And apparently there are some variants that are deciduous elm trees and some that are evergreen.

So, perhaps lamdara is an evergreen variant of an elm tree.

Yeah, I'm down for that.

I'm really keen to dig into the philosophical implications and ideas behind evergreen migrations.

And what makes it such a core part of lamdara and the guarantees that it provides.

So, I'm kind of curious, Mario, could you tell us about the motivations and contrast it to deciduous migrations in other contexts?

Maybe non-elm contexts.

What was the pain point and problem you were trying to solve and the guarantees you were trying to provide?

Yeah, sure.

I think the driver for evergreen was a practical one and the name came afterwards as I kind of expanded that practical problem to the front end.

So, where the concept arose from and kind of what led to my original presentation called Evergreen Elm at Elm Europe in 2019.

I had started working on the ideas behind early versions of lamdara.

So, this idea of full stack elm and what that would look like if the elm architecture was on the back end as well.

As opposed to previous incarnations of back end elm and some current incarnations of back end elm.

I kind of like, well, what if we just use elm as a language instead of JavaScript as a language, but otherwise have all the same concepts.

Back ends obviously have a server with HTTP endpoints.

And so, one of the approaches was like, okay, let's model all of that in elm and see what that looks like.

Whereas, the projects that I'd been working on and the thing that I was kind of reaching for was,

I wanted an experience on the back end that was the same as the experience on the front end.

And so, part of that crazy idea initially was, well, what if the database wasn't like a separate thing?

Like, I like the elm model. It's nice that it just, you know, while you're operating in the front end as a front end developer,

you don't think about the persistence of the front end model.

As long as the user's browser is open, it's quote unquote persistent, right? You're not thinking about it.

And if the user closes their tab, well, then it's gone.

And so, for the back end, I was like, okay, that's great. But, you know, if the server crashes

and all your state disappears, like that sucks.

And so, step one was like, okay, can we solve that problem?

But then step two was several of them being like, okay, well, you know, now there's a new thing that we don't really have.

Or at least I hadn't heard talked about a lot in elm or just in front end in general,

which is what happens when we have a new version of our app and the types have changed.

Now, I think the general approach on the front end is like, I don't know.

Cross your fingers and yeah.

Maybe they'll click a link and the page will refresh. We'll be lucky.

Or, you know, like, I suppose if you're, I suppose most companies would be like, I don't know.

We'll just, hopefully it doesn't happen that much. And if something goes wrong and they contact support,

we'll tell them, hey, have you tried refreshing the page? Oh, the issues have gone away. That's cool.

Great. Glad we solved your problem. And no one thinks about it again.

I guess the problem happens more with single page applications rather than.

Yeah, definitely.

More traditional way.

And I had heard, I can't remember who I heard this from, but I had heard anecdotally that like someone was,

told me, I was like, oh, they once received like their company used like versioning or something.

Right. So for the front end comes. So they're like, oh, you know, I don't think we have this problem because we use versioning.

And I was like, okay, cool. What's the oldest event version that you've gotten?

They're like, oh yeah, we got like, someone must have kept a tab open for like a year or something because we got events from like a year ago.

I was like, right. And what do you do with those? And they're like, I don't know if you drop them on the floor.

What can you do? You can't do anything right. In the normal thinking.

So I was like, what would be a really elm way to solve this?

And yeah, the only thing I could think of was, well, it should just upgrade.

Like it should just work. It should just continue.

And even more so than that, it's like, I want it to type check. That's what I want.

Like I want to go from one version of app to a new version of an app.

And I want the compiler to be like, yeah, you, this is good.

Like however you're going between these two, that type checks, that makes sense.

So that was the first motivation. It was like that, that thought of like, okay, well, if we have data that's alive in one app and we're releasing new app with new types, how do we get from one app to the second app?

And the reason name came about.

I feel like you're intuitively going to the more like subtle edge case of it, but there's also like the core meat of what most people would think of as a migration that people wouldn't gloss over how to handle these edge cases in their application and just be like, Matt, I hope it works, which is migrating your backend, migrating your database.

Right. And so of course that's a less subtle case, but evergreen migrations also provide a type safe way to migrate from one version of your backend model to a new version of your backend model.

Yeah, absolutely. So that was the practical driver.

It was like, yeah, obviously the concept of migrations of your backend is already a thing and it's important.

So I wanted to facilitate that and for it to be type safe.

And then where the name came from is I realized like, oh, now that like if we have this, well, we could do this on the front end as well.

And so instead of the front end effectively like having to drop it state every time there was a new app or you hope the user clicks, which for me had that notion of like, you know, the leaves falling off the tree, kind of being deciduous.

Then I thought, oh, well, if the front end can always stay alive, then I was suddenly like, well, that's kind of like how browsers now, they call them like evergreen, right?

Like browsers that continually update themselves. And so, yeah, that's, you know, elm and trees and evergreen. I was like, oh, this is nice. This is, yeah, this is the way I'm going.

So, yeah, I'm not sure how well though that term translates. A lot of people get confused.

I had imagined evergreen would like perfectly like encapsulate and explain everything. I'd be like, yeah, it's evergreen, but it doesn't, I'm not sure that in practice that's translated.

So I've definitely found that, that, yeah, people, it feels to me like, or at least the way that I experienced it, I'd be curious how you guys have experienced it.

My experience is that people go, oh, that's some black magic. Like there's something crazy going on with evergreen. There's something magical happening.

When in reality, it's actually really dumb. Like it's, it's the dumbest possible thing. It's like, how do you migrate with a function?

Like it takes value A and it returns value B. But I think, yeah, the, the grokking, like the, the, the details of what that actually means has interestingly been, yeah, a stumbling point, which I found interesting.

Yeah, I was there. Well, we were both there actually with them. Churchalk in Europe, 2019. So yeah, we, we got the explanation right off the bat.

So you've been biased. Yeah, very biased. So for me, it's clear. It's, it's a function. And I've also played with them there a little bit. So yeah, for me, it's pretty simple, but I can understand why people would be confused or scared of this because yeah.

Like imagine doing this in any other system than Elm or Lamdara and like, yeah, no, like that has to be magic.

Yeah, yeah, yeah. So maybe, and I think it was also funny because like at, I was devising this idea for Lamdara, which didn't quite exist yet. And so I kind of wanted to do a talk on evergreen Elm, but I couldn't really mention the backend.

So I had to reframe my whole Elm Europe talk as, well, yeah, like this concept on the front end. And I think for some people, maybe it was like, well, why would you go to all that effort in just refresh?

So it was kind of like this idea was there and I was like, this idea is so cool, but then I couldn't talk about it in the context where I thought it was super cool.

So yeah, I think I kind of demonstrated it backwards, but yeah, maybe, maybe if we just went very briefly for those listening to understand or people who may not have a concept or maybe aren't tracking,

we could go through like a couple of types of state and what a migration would look like verbally. I think that might give an indicator. What do you reckon?

I just want to correct something because, which is my fault because I induced you into giving an error, saying something wrong. Evergreen Elm was from 2018.

Oh, I see.

Lamdera.

Lamdera was 2019.

That's the one I was in the audience for. But I really enjoyed watching the Evergreen one on YouTube.

Yeah. Okay. So let's take a model. So let's say you have a, I think explaining it on the front end is probably the easiest. So let's say on your front end, we have the counter app. Right.

And I think I did this in my talk in France. So say the counter app starts with like counter of, it's a record with one field, field name counter, and the field type is int. Right.

And let's say for some silly reason, we say, okay, well, we now want to release a new version of the app, but we've changed the counter field to type float.

And the question would be like, how do we migrate from one to the other? And so, yeah, when we say it's just a function, what we're saying is like, yeah, we want a function that takes a record with a field counter type int and returns a record with a field counter type float.

So returning the new record, you basically just construct the record. But for the value of counter, right, you've got an integer and now you need a float.

So you kind of go off and you find a function, you know, what does like what function takes an integer and returns a float. And that's the function that you would use. So I can't remember off the top of my head. Is that from int or to float?

One of those two. Probably have to look that up.

To float, I'm pretty sure. Is what I would try. And then if I get a compiler error.

This is a guess driven development is how we operate. Yeah, so to float, to float, right from basics. So it's in the call.

So yeah, that is in an essence it, like that's the end of Evergreen migrations at like a simplified level.

So basically any model that you have in your back end or front end in Lambda, it's both. But if you're just thinking of Elm, like in the Elm architecture, whatever your model type is, if you change that type, the question is, well, what is a sensible transition?

What is a sensible transformation between whatever it used to be and whatever the new value is?

And on the front end, there is some interesting stuff, like some of the problems I posed in my talk was like, you know, do you need to tell a user that things are upgrading?

Like sometimes for a minor change or maybe for new features, additive stuff, you don't need to.

But, you know, maybe it would be kind of weird if the user was halfway filling out a form and then suddenly a form field like disappears.

But maybe it's not, you know, maybe it's not a big deal. Like is it any more or less confusing than the page breaking and them having to refresh?

But yeah, the cool thing is that like once you have this idea and then that evergreen setup is possible, it becomes a bit magical.

And it's something I love about Lambda and I really love it.

You know, occasionally people will come and tell me, so someone at Elm camp came and told me like, oh yeah, you know, I was making a game and some friends were testing and they were like, oh, this thing doesn't work.

And he was like, hold on, I'll fix that. And they're like, all right, that's deployed.

And like all of their apps like updated, you know, without them doing anything. And he's like, cool, that's done.

And they were like, wow. And yeah, that's that. I'm like, yeah, that that feels right to me.

I feel like that's how it should be. That feels awesome. So yeah, I think that's the joy of it.

That is very unique. So you're doing a hot reload on the client and applying the new code changes with that safe migration that migrates the front end model and then migrates any incoming front end messages.

So it hot swaps that in. Is that correct?

Yeah, that's right. So basically we take the concept of it. So the theoretical concept of evergreen is really just a function, right?

It's saying we've got this old type with an actual value of that old type that's live.

And then, you know, we want to go to this new type in the new app.

So we write a function that helps us migrate that live value from one to the other.

And then once you have that tool, it's like, what do you do with it?

And so in Lambdaera, what we do with it is we apply it to every value primitive that you have in Lambdaera.

And there's six of them. There's the front end model, there's the back end model, there's a front end message, the back end message.

And then the two messages that connect the two, two back end and two front end.

And so we apply this in Lambdaera, this idea that any time you change a type in any of those six core types, we go, cool, we can see you've changed this type.

You know, please write this migration. You know, here's the harness for it.

And, you know, if you do and it type checks, then we go, great.

If you deploy this, then, yeah, we can do exactly what you've said.

On both the front end and the back end, load the new version, take the model from the old app, migrate it into the new app.

And now you've got the new app running and it was all kind of smooth and hot and live and delightful.

So, yeah, that's the deal.

Yeah, that's amazing.

And I mean, if you wanted to, like, so you have, you know, as you said, it's a function from one version of a model to another.

And we should probably just to paint a picture here more explicitly, you have your types module in a Lambdaera app,

which defines, as you described, those six core types, back end model, front end model, back end message, front end message,

and then the two back end and two front end custom types, which define sending data back and forth.

But then those essentially get versioned under these namespaces V1, V2, V3, as you change your types.

So you have that module with all those six core types defined and any supporting types that you might have that are used in your back end model, in your front end model.

Yeah, the subsumed types.

Right. And that all, so those are all versioned.

So now instead of just types, being some fuzzy thing that you're like, Oh, yeah, I think it was different in a different version.

You actually have every version of your types for your Lambdaera application that you've ever deployed.

Yeah, absolutely. So the, it's funny, it's like the fundamental idea is really, really simple.

But then like the practicality of the tooling to implement that simple idea ended up being quite complicated.

And I think this is kind of where, and I talked about this as well in the talk, like part of the pitch for Lambdaera was that this idea behind this kind of live reload migrations.

I think it was, it seems to me that it is most compelling when your entire system has this philosophy behind it.

Because if you can, like if you're tracking your entire system, your front end and your back end and everything between, it's then that the tooling is really nice.

Right. Without that setup, you would have kind of what you've just described, which is like, you know, there's no good way to know what the authority is on these types.

So you can't, like, it's not very easy for you to, you'd have to use discipline to be like, okay, you know, I've changed the type, therefore let's go and copy paste our types and let's try and name space.

You know, you'd have to do this. Whereas with Lambdaera, because everything's integrated, like the local tooling, you know, when you run a check, like a pre-check for a deploy,

it knows to talk to your production instances and go, okay, well that's what's deployed currently.

That's what the type hashes are. Do we have a change? Okay, we do. Let's automate the snapshotting.

Let's automate, you know, giving you some feedback about what's happening, what types have changed so that, you know, the developer experience that, you know, I've been chasing in Lambdaera is that you don't have to think about it. Right.

So in the early versions of Lambdaera, you had to think about it, you had to track it.

And yeah, as of the latest release, we are now trying to make that as seamless and kind of carefree as possible so that you can, with confidence, be like, you know what, I'm going to re-architect my whole backend.

I'm going to change the whole backend model. I'm going to move things around. I'm going to change dictionaries to sets and nest things into custom. I'll do whatever I want.

And the idea is that you can kind of do that and then Lambdaera will be like, cool, that's cool. Here's what changed. All right. Like this is now you need to tell me how to get from where you were to where you are.

And if you can figure that out, if you can make it compile, then cool. Like probably this is going to, well, from a type perspective, it's going to migrate.

But obviously we can't guarantee that you haven't put dic.empty or set.empty where you shouldn't have and that you'll lose some data. So you can still write the wrong migration in the business sense, but you can't forget to migrate a column.

You can't delete a column and still have code that's referring to it. Right. So in that sense, I think it's a step up from the traditional transactional database migrations that I think most of us are used to.

Have you heard of people writing unit tests for their migrations or is it usually straightforward enough where people are?

Yeah, no, that's a good question. I think the question's been expressed before. I don't think there's any reason you couldn't. There's a slightly awkward technical reason in Lambdaera why it would be a bit weird.

It's because Lambdaera, yes, actually it's not specific to Lambdaera. There's a question of, you know, we've got these type snapshots, right? So we snapshot our types every time things change.

But we want to snapshot anything that's changed and we deploy. We want to snapshot that. However, because of the way that Elm's namespacing works, we want in production for our migrations to result in values of the type of our current types file, not of values of the type of the last snapshot file, if that makes sense.

So maybe to put this another way, if I define a custom type called ice cream with the variants chocolate, vanilla and strawberry, and I put that in my types.elm, and then in Lambdaera I go to deploy, Lambdaera will go, ooh, let's say it's our first deploy.

Lambdaera goes, ooh, it's your first deploy. I'm going to snapshot those types into evergreen slash v1 slash types.elm. So now in this v1 types.elm, there's an evergreen.v1.icecream type that has the exact same variants.

That is what's going to be deployed, the ones with v1.

Yeah, so both actually get deployed, but the v1s there, the v1s on the first deploy only exist to be referenced in future deploys when the types change.

But in the current version, we still want ice cream values of the types.elm, right? Because that's what our whole application uses.

But in the future, when we now deploy version two, and let's say for ice cream, we had chocolate, vanilla, strawberry, and let's say we add mango, right? So now we're going, okay, this type is now different.

It has an additional variant. The migration that gets generated is a migration from version one to version two, right? And so we decide what we do with those values.

Probably we do nothing, or maybe we say, hey, you know, actually, everybody that told us that they put strawberry in the first version, they all emailed us and they said, you know, we hate strawberry, we actually want mango.

So can you please add a mango option, but also migrate any of our strawberry choices to mango, right? So that would be an example where that function when you migrate the value, you might case the old type, you know, on chocolate, vanilla, strawberry, and chocolate and vanilla, you would map to chocolate and vanilla in the new type.

But maybe strawberry variants, we would map to mango in the new type, right? So that's how you could do data changes or data transformations in your migrations.

But yeah, we're getting a bit into the weeds here. But the trick is that that new type will become a snapshot version two, right? And your migration has to type check between version one and version two.

But actually, when we deploy, we get sneaky. And we don't map it to version two, we actually map it to types dot elm, because that's what you need in your application. So we do this little we do this little swapsy, so that you're always getting the types that your application actually uses.

But at the same time, we're preparing that snapshot, that's that snapshot in time to go, okay, when we deployed version two, this is what the types were. So that's ready and committed and in your repository, such that if you change it in the future, we don't have to try to go back and remember what they were, we already snapshotted what they were when you deployed.

Is that making any sense? I realize that's a bit that's a bit squirrelly.

It makes sense to me.

Nice.

So you said that every time there's a change in the the types in those six core types, you need to write a migration. One, it's only when they change, right? If nothing has changed, and you don't need to write a to write a migration?

Yeah, that's correct. So that's a that's a little optimization that okay, yeah, lambda has, which is that even though in elm, those types aren't the same. If when we snapshot the types, the type tree that gets generated, generates identical encoders and decoders. So we can keep, you know, if, if you have a version two snapshot, right, and then you deploy version 34567, and nothing's changed, we can keep decoding straight into your types dot elm type.

Because we know that those encoders and decoders are identical from the binary format, right? So it's only when it's only when the types do change that we know, okay, that will mean that the encoders and decoders will become different, which means we need a function to get between the old value and the new value to keep everything sane.

And the second question is, could you write a migration, even if there's no type change, for instance, everyone hates strawberries. So let's write a migration that goes to to mango without having a type change.

Yeah, so you can't. But with an asterisk, you can't with an asterisk, you can. So you can't change nothing and ask, like evergreen and lambda to be like, please let me do a migration anyway. But you can just add a dummy field. And then evergreen will happily be like, Oh, you need a migration, right?

And so the trick is that when this is a technical, maybe a shortfall that we might tighten in the future. But right now, whenever you get asked to write a migration, you actually get placeholders for all six types.

But by default, it'll tag the migrations for the types that haven't changed as unchanged. So you could if you wanted to still apply a migration whenever there is a migration, if that makes sense. If there's a migration to any type, there's a migration to all six types.

That's just the way that it's implemented in lambda right now. So you could do data transformations like across the board. But yeah, yeah, it's you have to get lambda lambda is evergreen implementation to think that something's changed in order for it to be like, fine, okay, let's get let's get some migrations involved.

If the current version is v6, and I manually create a v7 folder, would that trick lambda into generating a migration?

No. So if you manually create a lambda right now, what would happen if you manually created a version seven, and then you try to deploy it in production, it would say, Hey, I wasn't expecting there to be a migration, but I see a v7 migration file, I don't know what's going on. Like something something's bad. So it'll it'll try bail out.

So let's say that you wanted to do the the strawberry to mango migration in a more polite opt in way. So the migrations files, they give you the opportunity to I mean, to arbitrarily change the model, the back end model, the front end model. So you could, you could make whatever changes, you know, as you're in is describing, you know, changing a model, when the data hasn't really changed.

And you need to sort of get lambda to create a migration file for you. But once you do that, you can make any data change, it doesn't necessarily have to be to get the types to fit together for the new format. That's correct. Yeah. And you also get a command for your for your back end and front end, you can trigger a command as part of that migration.

So so let's talk through that a little bit. If we wanted to do the polite version of the strawberry to mango, where we're saying, hey, unfortunately, strawberry is no longer an option. But so we are inviting you to opt into this. And now when you log in to the ice cream shop, you're going to see a, an announcement banner that says, we need you to choose a flavor. So how would you model that with a with an evergreen migration?

Yeah, absolutely. So I think 90% of the answer to that question has got nothing to do with evergreen. The question first is how would we model this in an L map? So kind of what if I reflect back at you, what I'm hearing is in terms of state. So there's something now on the user profile, right, which I would say a Boolean, which is something like requested, what would we call it? Like, please, revalidate ice cream preferences question, right?

Or show revalidate ice cream preference question, right? And that's a Boolean true or false. And so we add this to our front end model. Actually, we add it to our back end model. Let's see, because we've got accounts here. We've got user accounts, right? We have only saved preferences for users that have logged in, right? Otherwise, how will we know who they are when they come back? So we say, okay, every user profile has this new flag, right? And we add that to the back end model. And then we say, okay, in the front end model. Now, suddenly, we get type errors, right in the front end model, because in this case, it's lambda error, our types are shared.

You know, so when a user logs in on the front end, the session hydrates their account. And so that value will come into the front end. And we go into the view and we say, okay, cool. If user.showRevalidateIceCreamPreference is equal to true, then show them this little bit of UI that asks them for this question. Else, let's just show nothing. We'll just leave it, right?

And then maybe as part of that UI, we say, hey, like, it looks like you've chosen, you know, strawberry in the past. And, you know, you told us that you wanted to choose a different flavor when we had more available. That's now available. Like, click here to go to your profile page and edit your preferences. And I think, is that it from an app perspective? I think that's probably it.

Yeah. And I guess you could decide whether you want your custom type to include strawberry and have it be deprecated or remove strawberry and then set it as mango. But then you're going to need to have some application logic in the appropriate places so you don't accidentally ship them their monthly flavor of mango before they've chosen.

And maybe, you know, the monthly shipment process, instead of automatically sending it, is going to send an email and say, hey, you need to log in to change this because we no longer have this flavor. If you want your shipments to resume, then please log in and select something. So, you know, as you say, a lot of it is just modeling the problem in a sort of high-level way.

Yeah. So let's say you wanted to, so we're still, we're nowhere near evergreen yet, right? We're still in our application realm. So if you wanted to apply, like, make impossible states impossible, you know, if you wanted to be like, actually, you know what, like, what's the business rationale? Like, so let's invent some. Say the business rationale is enough people told us that they hate strawberry, that we're discontinuing strawberry. Right? So maybe we go, okay, well, we need to remove the strawberry variant, but we need somewhere to migrate these people to. Right?

So maybe we turn, you know, ice cream preference into a custom type itself. Well, maybe we turn it into a maybe, right? And nothing means they haven't selected. So we can't send them any shipments. Right? Because we don't know what flavor. Well, maybe we change it into a custom type where we say, you know, not selected or ice cream selected in this new type that only has the variants that we offer. And then a third state, which is like needs revalidation or, you know, ex, ex strawberry lover or something like that. Right?

Yeah. I thought there was no one who likes strawberries.

Yeah.

It's an impossible state.

Yeah, absolutely.

So yeah, we, we, we kind of do whatever modeling and I think this is the nice bit. And this is a nice bit that we go, we at this point, I think, unlike the way that you do it in, in kind of traditional full stack applications is like you're, you're simultaneously thinking about your data structure and the migration at the same time. Whereas in, in Elm, like via Lambda or specifically in that context, the idea is like, well, just forget about it. Just, just model what new state of affairs you would love to have in your app.

Like what is the actual value set up? That makes sense. And then we go great. Now that we've done, we've chosen whatever one of those it was. Now we go, okay, Lambda or deploy or Lambda or check like Lambda or deploy invokes Lambda or check first.

And Lambera goes, Ooh, I can see your types have changed. Okay. I'm going to go and attempt to make some migrations for you. I'm going to do my best. So that in the latest version, it goes, not only am I going to generate the scaffold, I'm going to do a best effort generation for you. I'm going to look at the tree of your types in the old version, the tree of your types in the new version.

I'm going to try diff them, like zip them together and you know where they don't look like they've changed. I'm going to try and just intelligently make all those, those choices for you to keep everything the same.

Mainly the concern is migrating custom types. Cause as we know in Elm, two equally semantically the same custom types in two different modules aren't actually equal. Right. So unlike two, two semantically identical record types in two different modules are equal. Right.

So evergreen goes, okay, I'm going to automate all of those kind of crud transformations for you. But in the bits where things have definitely changed and I can't, you know, I can't do anything reasonable.

I'm going to put placeholders for you and go, okay, here, I've gotten to ice cream preference. This has changed. You know, it used to be this custom type from version one. Now it's this different custom type from version two.

How do you want to do with this? You know, how do you want to get this value across? And so you write the code that says, great, I've done all my business logic. Everything makes sense. And you know, I'll, I guess we've, we've invented some sort of ice cream subscription store here in this, in this, in this analogy.

Sounds like a great business. Yeah. Except for the free frozen shipments. That could be. Yeah, that's tough. Tough. But yeah, I want, I definitely want ice cream now. But anyway, yeah.

So we've done it. We've done all the business logic. And then now it's kind of like, cool, how do we want to get from one value to the other? And the nice thing is once we've done all of this, you know, it type checks. So it goes, yeah, cool. Okay. That makes sense. You've, you've successfully done the transition.

And I suppose going back to the, to the unit test question, it's like, yeah, if, if that was a very complicated transition and you wanted to have certain invariants that hold, yeah, there's no reason why that function that you wrote in there for that particular part of the migration, you can put that function to the side.

You could put it elsewhere. You could include it from the tests and then you could do all your scenarios and tests. You know, I put in these old versions. I expect these new versions is my migration kind of making sense. And I think that's nice as well, because you get to stay in Elm, right? If you think of that in any other system, now you're mocking perhaps the database.

You have to, you have to, if you're not mocking the database, you actually have to boot a database in your test setup. You have to set the first versions. You have to do the migration of the schema. Then you have to load the value. You know, there'd be a lot to get that working.

Whereas in Elm, we get like a lot of confidence from the type checking and then you can cover the rest of the ground in tests if you need to.

Whenever there's a type change, you copy everything into like a V2. You copy all the types or do you copy all the code?

No, just the types. So what Lambda, so the implementation currently is Lambda kind of recursively trolls through your types from their known. So in Lambda you have to put the core types in a specific location, right?

So Lambda goes, okay, I know where they have to be. I start from there and I kind of recursively troll through the type definition and any type definitions it references.

And it kind of progressively sucks those out. And as it goes, it namespaces them and it writes them into the snapshots file. So you get, yeah, like an extracted copy of your types tree for every single core type.

Okay. Now there's a really good feature in Elm, which I don't know if you've ever heard us talk about, which is opaque types. How does it work for opaque types? Can you migrate them? Can you not?

Yeah, this is a great question. So sadly right now, there is a way around this. We could add compiler support and do some magic. But right now you sadly need to open up the constructors within your code base.

So there is a risk.

Do you mean only in migrations or always?

Always. So the reason for that is if you consider like, so let's go back to our ice cream flavors. Say you made that an opaque type, all right?

Obviously.

Yeah, obviously. Nobody is allowed to specifically say what flavor they like. They have to go through the flavor constructor function, right?

Okay, so maybe that's what you love. You're like, yes, this is the best way to do this. So in this maybe convoluted example, think about now how you express a migration for this, right?

So I'm asking you, I've written the function signature. So the function signature is from v1.icecream to v2.icecream, right? And into the value, into the migration, you get a value called old.

That's what I call it by default, right? Old is the name of the old ice cream flavor. So this is going to be a specific instance, like a specific value instance of that type, right?

The first thing you would normally do in migrations of a custom type is you would be case old of, and then you would pattern match on all the variants, right? And then you would return new variants.

But in your case, if you've made an opaque type, what do you do now?

Yeah, exactly. Yeah, that's why I was wondering, like, I think there's some problem with the opaque types. And I seem to remember that, yeah, opaque types didn't work well with Dundara or with the migration system.

If you really, really wanted to, you could write specific code to try. Yeah, so you would have to write like special functions that were like basically constructor functions for your new types.

And then maybe you would inside your actual normal elm code, like within that module that has access to the constructors, you might write like a deconstructing or put the migration file directly in there, right?

So you could try and keep all of your types opaque and hidden and put the actual migration function in the file somehow, right?

The thing that gets really weird and what I discourage with Lambda, although it is possible, is that obviously migrations are like a point in time thing.

And so part of the reason we do snapshots is as your app continues to evolve, your code is going to change, right?

So the weird thing is, if you put the migration function into the file that defines those constructors, in the next version, you're going to have to change the type that's in that file.

And suddenly those migration functions that are referring to the types that are in the same file are wrong.

So the question is like, okay, well, now where do I put them? You want to put them into the history, but the history can't access the constructors, right?

So the problem today is, okay, with those types, Lambda forces you to open it up. That's kind of the easy way.

Long-term, is there a way around it? Yes, the way around it would be at the compiler level for us to go when we're compiling,

when we see that we're compiling a module that is within the Evergreen namespace, magically the compiler unhinges its export restrictions and pretends like as if everything is exported.

So then only in the migrations context, you can reference a opaque type constructor and you won't get a type error saying, oh, this is, you know, unexposed or hidden.

But then in your main code base, any way you referenced one, you would get the type error as usual.

So that's the idea. I don't know how difficult that would be. I haven't delved into it yet.

But at least in theory, I think we could improve those ergonomics and, you know, get back the same kind of opaque type protections that we have today.

So yeah, to lovers of opaque types, I'm sorry, Jeroen, there's a little bit of a compromise there from, you know, ergonomics and language restrictions.

But yeah, I think for now it's probably not the end of the world, but we have a way to improve that in the future.

Not too many people have complained about it so far. And by not too many people, I mean zero people.

I'm complaining. Here's my official complaint, Mario.

Okay. So far we have officially had one person complaining. So there is one people demanding.

First the strawberry, now opaque types. What next?

Yeah, but also like with opaque types, what we tend to represent with the opaque types is invariance, right?

So there's the actual data and there's the invariance. So like, even if the types matched, if we have an opaque type that makes sure you have selected three types of ice cream,

and then in another version, well, it's only supposed to be two now. Like, yeah, there's, you have some kind of revalidation to do anyway, or which, yeah, not sure how you would do it.

Yeah. So there's one interesting part of this where this actually has come up. So there's some kind of, there's some a little bit undocumented auto-generation support for specific package types that are opaque and people kind of commonly use.

So for example, something like non-empty. Now, the thing I don't think we've spoken about yet is when I say that LambdaEra extracts the type hierarchy, we only do that to the extent of user-defined types.

Right? So if you're referencing like non-empty string, or like the string type from the non-empty package, we won't...

Which is an opaque type?

Yes. Yeah. So you have a constructor where you have to give it a string that has something in it, and it'll only return a non-empty string value if you give it a non-empty string.

So once you have a value of that type, you know that it's definitely non-empty. So yeah, in that case, LambdaEra isn't snapshotting the package type.

There's a few reasons for that, that I'm not sure entirely worth going into, but long story short, it focuses just on the user types now. Mainly, actually, the best reason for it is the opaque type discussion that we're having.

The best reason for it is we can't really do migrations on package types, because if it's something like non-empty, they don't offer us the internals of that package. Right?

So what I do is I generate some code that basically does what the sensible default would be if you knew you had a...

Well, the other thing is that package types don't change, at least if the package version hasn't changed. So if we already have a value of non-empty, we don't need to migrate it.

But there has been some cases where... Okay, so a better type than non-empty string, which is kind of monomorphic, or one that isn't, one that would be polymorphic, would be like, say, the anydict.

One of the anydict packages, right? Like, so packages that let you define a custom type as your key, and then it lets you do lookups and inserts on this dictionary that you can't do with the vanilla dictionary implementation, because it requires keys to be comparable.

So if you use one of these types, now you've got like a parametric type, right? You are putting your own user-defined custom type into this third-party package.

So if you've changed that type, when it comes to a migration, you may have anydict icecream version 1, and now you need to go to anydict icecream version 2.

So you need to extract all of the values from that first dictionary, and then do the migration, and then create a new dictionary. So for some of these types, it's kind of like, that's annoying, and it's mechanical.

And we know pretty much what people are going to do, especially if that custom type hasn't changed.

So lamdara in some cases will detect certain common library types, and it'll try and do the sensible migration for you if your custom type that's being used there hasn't changed.

Is that making sense? Am I tracking?

Yeah, does that mean that if I were to make a new package with a new data structure, like anydict, that it would be better if I contacted you to add support for that?

I definitely wouldn't want to encourage a perspective in the ARM community that everyone should be thinking of lamdara concerns in their packages.

I'd rather tackle it when it came up. But yeah, what would make it easier for a lamdara user to use your type if it's a type that doesn't contain user types, then I don't think it matters.

Because they can just put the old ones in the new one. But if you are publishing a package that does contain the ability for users to put in custom types, then putting in the ability to migrate.

If there was a map function that sensibly made sense, where they could give you a function of type A to B, and that let them migrate your type A to your type B in the package, then that would definitely make it easier for them to construct migrations.

But otherwise, yeah, an ability to somehow exhaustively deconstruct values in a meaningful way and exhaustively construct values in a meaningful way of your package type, I think would be the key primitives that people would need to be able to express this idea of going from an old type to a new type for any conceivable type change, if that makes sense.

Yeah, I could also imagine for some use cases, instead of directly having an opaque type, if you really wanted to model your application logic with an opaque type, especially in the front end, with the back end, there might be more performance concerns in some instances.

But you could have your raw type stored, let's say, in your front end model. And then you could have a wrapper that takes it from the raw type to your opaque types.

So you could have a function that transforms the raw types to opaque types. So immediately, as soon as you're actually working with those types, it's turning it into some opaque type that you can work with those guarantees.

But then, as you're modifying it, you also need a way to transform that too. So it's a challenging problem. But that idea you have of being able to reach into the opaque types in the context of a migration is very intriguing.

And it does, I mean, it's a, it's a, an interesting philosophical place to be. But overall, the feeling that I'm getting what I'm realizing with evergreen migrations is that so much of what happens with, you know, the traditional way that many people may have worked with migrating, you know, between a some sort of JavaScript front end and a Rails back end, or whatever it might be,

the traditional experience that I've had doing conceptually migrations, maybe there are some, you know, there are probably some back end migrations involved. And then you need to handle that with different versions of the front end is you, you very carefully queue up the changes you're going to make, you very carefully, you know, test your actual migrations on some test data.

And maybe you sort of hope that that in between state works out okay, and don't think too carefully about it. Or maybe you think very carefully about it. But even so, even if you're thinking very carefully about it, you're thinking very carefully about an implicit contract.

Whereas what I'm realizing is that what lambda gives you is an explicit contract for all of these pieces, and they're all living in the same ecosystem. And, you know, if you, you could make something implicit in a lambda app by depending on something in the outside world, but for the things that are self contained within a lambda back end and lambda front end code base, it's an explicit contract.

And it has the elm type system and all the guarantees that come with that of type safety and purity and no escape patches and immutability and all these things. So you put all those pieces together and what you get is an explicit contract for managing the entire migration, which is a really interesting feature to have. I mean, that's kind of a game changer.

Yeah, absolutely. So for people thinking about like their traditional migration setup, I think it's kind of the pitch of evergreen is similar to the elm pitch against JavaScript. It comes up in a lot of different areas, but let's take like JavaScript, you know, decoders, right? So the idea is that you go, okay, well, there's like a whole, you know, we know, that first name is always going to be there. Like we know this, in quotes, know this, right?

Then at some point in the future, someone changes it and now our code's broken. Right? So like an elm, it's like, yeah, yeah, I know, you know, this, but no, like, you've got to, you got to tell me how does this decode? I'm going to, I'm going to validate it. Okay. We validate it with the decoder. Now we know, now we know for sure. Right? Like, it's not just a, like you said, we've turned that implicit thing into the explicit.

And so I think there's a lot of stuff with migrations. And part of why I was really excited about this is like, there's all this, there's all this cognitive overhead that you have to keep. Right? Because there's nothing snapshotting, nothing keeping your types for you. You have to remember when you're building this new thing, like exactly what you've changed, exactly what you've added, exactly what you've removed to the point where, like, I think it's, it's kind of nuts that the industry response seems to me, the industry standard response is like, oh, well, obviously the sensible solution is the one that's going to be the most effective.

The sensible solution is to never remove anything ever again. And that's like, I get it. Like, that's, that's probably the only way that we can cut that, at least that part of the problem out. Right. In like the traditional kind of stack, or if you're working at scale, right, you go, okay, we never remove everything, never deprecate it. That's why you have stuff like, you know, protobufs. So it's like, yeah, you've allocated a field. Well, that's there forever in your payloads, every future payload, even if you never use it, that byte range is now allocated. Right?

You'd have to do a major version deprecation or shift your, your schema to a new endpoint to change that or to optimize it. And so the other nice thing that we get there is that all of that stuff becomes explicit. But also, on the other hand, something I think we haven't talked about is that gap problem. Right?

So I talk about in my talk, which is what if you have, if you don't have a system where everything's integrated together, and especially if you work at a company where potentially different teams can deploy different things at different times, right, you merge this change in for your backend, maybe if you're lucky, I mean, not if you're lucky, that's the wrong word. Let's say your company has chosen chosen to have a mono repo, maybe you've got one pull request, right? So the thing you merge is the front end and the backend changes at the same time.

So at least that step is synchronized. But worst case, you have two different repositories, right? So those things could merge at different times. In both cases, regardless of whether you merge at the same time, or whether you merged at two different times, the deploy could still happen at two different times, right? And in most systems, they're not necessarily synchronized from a user perspective.

So it is very possible that you have a situation where you have version one front end and version one backend in production. And you get a version two front end, that's trying to talk to a version one backend for a moment in time, or you could have the version two backend launches faster. So you have version one front end that's trying to talk to a future version two backend. And that inverses as well, right?

So you could have a version two backend that's trying to respond to a version one front end or a version two front end that's trying to respond back to a version one backend. So there's four variations there, right? There's the outbound payloads, and there's the inbound payloads, and you could have failures on both sides.

And I think generally, the thing that like, we kind of think about, okay, what tools or techniques can we apply to solve that? And I think generally, the answer is none. Right? Like there isn't really anything we can do to get explicit guarantees. One thing we can do is say we never remove fields, right? But we can still get the wrong semantics, right?

Like it may be compatible, that a version two front end message gets ingested by the version one backend, but it may process the wrong business logic, you know, stuff that we've said, or should now change, right? It should be version two. Version two should be handling, you know, strawberry responses coming in and becoming mango, but we've actually gotten, you know, a mango response in the future, or we've gotten a strawberry response in the future, and a version one backend has created yet another subscription to strawberry, which shouldn't be possible anymore.

So yeah, one thing. This is why I mentioned this before about, I think evergreen works really well in a system where you have full and total control of both the front end and the backend migration synchronicity, synchronicity, I'm not sure the correct pronunciation of that word. So that, you know, we apply this evergreen concepts to everything in lockstep. Right? So we know that you only there's never going to be a scenario where we're getting events from the future to older versions.

Everything's always being pushed forward. And so yeah, lambda has like, like a migration, kind of like a staging thing where it basically hot loads everything and everything's prepped and ready, all the new versions are ready, everything's live. And then there's like a sudden lockstep where everything in the same instant as much as possible, kind of all slides into the future. But if we've missed anything, you know, if there's any old front end that comes online later, or there's a backend that lags for some reason, or whatever, I mean, that's technically not possible.

Let's say in the future, we had, you know, distributed setup, and it was possible, it was a backend that was lagging, because that all those new versions have this migration thing set up, the first thing they do be like, Oh, I'm receiving a version one, but I'm actually on version two, let's run that through the migration first. And so now you always have consistent, like that's consistently being executed in the latest version, regardless of kind of what's at play in that synchronicity.

So I think that that is a really difficult problem to solve outside of the context of a type safe, pure, immutable and exhaustive language. And I think that's what makes this thing so nice. And I'm like evergreen in Elm, I think is super delightful. It just sheds all these delightful properties. And so yeah, we leverage that as much as possible as we can.

I don't think you would need the type safe parts, but it definitely helps.

Yeah, it definitely you could with discipline get the same effect. But yeah, you could.

And it makes the contract more explicit. I find that like thinking about migrations in this paradigm, like it, it's making me think of the data modeling in a different way, as you said, like more, more focused on the ideal data modeling, which often just a vanilla LMAP makes us do this too, I think, right? Just think about the data modeling in a, I don't know, less hacky way, like just what would be the ideal way to model this?

And then you just kind of do that and, and let everything flow from there. And like, for the, you know, strawberry ice cream deprecation, like I was talking about that strategy, you could use of, you know, leaving the users flavor, selected flavors as mango, and then having some separate data that tracks that it's actually, it's actually not mango, right?

And if you're doing, if you're doing a sort of migration in this more old school way, where you have these implicit contracts, then that that's not really any worse than modeling, modeling it explicitly.

But if you, if you have the ability to have all these pieces, connects together very explicitly, I would tend to think of it differently, I would tend to think of it as I want my data to reflect exactly what it is. So I would tend to want to say, you know, maybe it's like a variant of flavors, or maybe it's like a wrapper around that.

So you have, you either have like a selected flavor, or you have a legacy selection, in which case, shipments can't proceed or whatever. But I would want to model that state very explicitly, because otherwise, you might end up with a code path where you aren't considering that case.

And what ends up happening all the time with sort of supporting these, these migrations, and these conceptual changes to the domain, in a code base is you don't consider how a change affects the entire system.

You end up with all these sort of conditionals scattered around where you could easily forget something. But as with any Elm custom type, you have, you know, when some when an Elm custom type changes, it forces you to consider the impact of that change everywhere, even if it's trivial, it forces you to explicitly, you know, recognize that it's changed.

And so instead of just scattering some conditionals around and saying, Oh, if it's this weird, extra Boolean that I need to check for that it's this weird case, like that ends up being really unpleasant to maintain and a huge source of bugs and also a huge source of like, you know, the more veteran programmers on that team who know this code base are like, Oh, like, talk to talk to this one programmer, you know, they know all the ins and outs of this code base,

they know all the conditionals you need to be sure to check for, they know all the strange Booleans in the system that you need to check for anytime you do something. But if you model it as a custom type where you're saying exactly what it is, like a user doesn't necessarily have a flavor, they may have some legacy thing. And why does that happen? And maybe, maybe you get to a point where you can drop that at a certain point, maybe you, you have, you are able to confirm that everybody has migrated off of that.

And then you can migrate off of that and, and reflect that. But it really allows you to think in terms of your ideal domain modeling instead of hacking something together and throwing some conditionals in there.

Yeah, absolutely. So this is this is slightly a tangent away from evergreen. And I kind of see this as more a benefit of Lambdaera itself, or at least a benefit of the idea of like, what if we didn't have this disconnection between the way that we choose to store our data and the way that we choose to model our data? Right. And so like, if you're already familiar with Elm, and you have experienced like the delight of Elm's type system in, in the large, right, I reckon in most,

cases, you can model, like your view of the world really nicely, right, especially like with custom types. And so there's this, I think, when you're in a more traditional code base, I think, at least the way that I'm thinking of more traditional code base is like, I might model those invariants with a new custom type on the front end.

But usually I start to, like, especially for something like this, right, where we go, okay, we're discontinuing this product, right? The marketing team has said, like, could we, you know, could we do this, like, it would make our lives easier. Could you do this stuff in the UI, but you know, like, if it's going to take more than a day, and it's going to impact our backlog, like, we just don't, like, just don't bother, we'll just manually send some emails, or we'll do something, right?

Like, because engineering time, I think, becomes scarce and precious in organizations, usually. So, you know, if you sit down, you're like, okay, well, yeah, you know, we could try and manage this for you. Okay, front end, easy, that's fine. We'll just put this new state and we'll do that. But, okay, so we're going to change the profile system in the backend, we're gonna have to add this new field, we're gonna have to do that migration.

But this other system uses that way to make sure, you know, like, and you're thinking about all these layers and the different systems that you're gonna have to coordinate. Whereas in Lambdaera, you can kind of go, okay, let's add this field to the thing. And bam, we forced all these failures across the, you know, we can see immediately what the impact is, like, we're driving, what is our to do list in terms of implementing this feature. And we can see straight away, like, oh, yeah, you know what, we forgot, this is used in this module, and it causes heaps of, you know what, guys, this isn't going to be worth it.

Or we can say, oh, I got two type errors, like, no, I reckon we can do this, you know, and you're not thinking about all these extra steps about how that change or set of changes is going to be translated into these kind of primitives, like into this primitive obsession that we talk about, or like, you know, Boolean blindness, or, you know, just this generally this idea where we have to dumb down to types, which we kind of, I mean, until we are graced with this much anticipated, you know,

anticipated future release of Evan's work and whatever's happening in the database side, right, but at the moment, there isn't a really great way to put custom types natively into Postgres, for example, right, so on the project, so I work on those elements, the front end, your custom types become something entirely different, there's a lot of glue around modeling that, and modeling those changes. And so I think a lot of the time, doing silly little things like that, it's kind of like, ah, too hard basket, let's not bother trying to, let's not mess with the stack.

Whereas I think in the Lambda situation, it's like, yeah, let's mess with the stack. Let's actively mess with it, because, you know, later on, the compiler is going to be like, all right, cool. I saw you messed with XYZ, can you please fix that? Tell me what you want to do with the migration. So I think that, yeah, that's a cool part. I'm excited for that on my own projects, because that's what I want. You know, when I'm dealing with my hobby projects, or I'm picking up a project I haven't touched for months. And I'm like, I want to add all this and change this and do that.

I want to have that joy of not now worrying like, oh, is my modeling correct? Do I need to fix how I've set my stuff up in Postgres? Am I going to have SQL queries somewhere that are now wrong? Yeah, I really like what that gives you from whatever green gives you, whatever green ideology kind of gives you in that kind of full stack Elm context. So yeah, and Lambdaera pitch rant at this point.

No, it's amazing. And Jeroen mentioned the idea of testing before, like that also strikes me as something that, you know, I mean, for a team that really, really wants to robustly manage their migrations and data integrity and all these sorts of things, just having evergreen migrations is huge. And, like, you know, makes the whole process so much more explicit.

And you could probably write some sort of tests manually for that. But I could imagine some sort of automated things around even like testing the UI after a migration with, you know, Lambdaera program tests or, you know, testing the hot swapping. There are so many things you could imagine conceptually when you just have these pieces fitting together in this way. So it's really intriguing, really exciting stuff.

Yeah, definitely. So that kind of thing, like being able to test migrations is not something that's easily done today in Lambdaera. Like if you're in Lambdaera live, like the live development environment locally, and you've changed all your types, like you'll be working on an app with the latest version. Right. But there's not a super easy way. You can with some effort do it manually. And I've helped people try and figure that out. Like the pieces are there, but it's not ergonomic.

But yeah, I think that would be really, really cool in future for you to be like, hey, I want to pull down my production model and run it through the migration. And then I want to play with that result locally and see what that looks like. So that would be one cool improvement.

Another one that, and so this is kind of like a full disclosure on a downside of this approach. So something that's kind of come up recently, there's been more, increasingly more teams using Lambdaera. And so when you have a team of people using or trying to develop an app together, we end up with this problem of, you know, it's kind of fairly common if you're working on a backlog and you have a few features and these teams are like kind of in an agency setting.

Right. So they're working on behalf of a customer. So they go, okay, well, you know, I've got a pull request for this particular feature. Can I deploy like a preview version of this app so we can kind of show the customer we can do some QA, right. Or some kind of review so they can take a look at this.

Now this has caused an interesting problem with Evergreen or some confusion, because if we think about the Evergreen assumption, right, the assumption is that you've got a straight linear change version one to two to three to four. And the idea is that when you're doing a check, what you're checking against is the production app.

The being the operative word there as in the singular the, because if you have multiple production apps, suddenly everything starts to not make sense. You know, if you've changed your types, you've changed it relative to which production app.

So we kind of had this issue where teams would go to create, they would just manually create another app. They would manually call it, you know, my app dash preview one, some feature. And then first things first, they would try to deploy their Lambda app that was only previously going to production.

Let's say it's at version seven. So they tried to deploy to this new app and the app goes, well, hold on, you're deploying version one, but I'm seeing like version seven snapshots, like what's going on? Right? So the compulsion there, I think the natural thing is to be like, oh, that's weird.

I don't know what to do about this. I'm just going to delete Evergreen folder. And now it goes, okay, great, cool. I can deploy version one for you. Right? So now you've got this code base that's deployed to version seven and to version one in two different apps.

Now, a Lambdaera doesn't have a preview app concept right now. So as far as Evergreen is concerned, there's two production apps. Right? So let's say throughout the course of your pull request, you change types.

And whenever you do, and you're checking against your preview deployment, Evergreen is going to be like, okay, well, this is your production app. Looks like your types have changed. You need to write a migration.

Right? There is a way in Lambdaera currently to be like, do you ignore the migration? Or like, I'm happy for you to just drop it on the floor and reset my backend model to init.

So they might do that because they're like, oh, this isn't important. You know, my state isn't important on this preview app. And so they do that a few times.

And then if they're unlucky, what's happened a couple of times was you get everything's good and they're like, okay, great. Let's merge. And now you merge that in and you've clobbered kind of your Evergreen history.

And now you're on your main app, the one that's already at version seven, you're trying to deploy and it's looking at maybe like an app version three.

And so in that context, it goes, okay, cool. You've got app version three, app version seven's improved. And it tries to figure out what's going on and it doesn't really make sense.

And then we end up in a really confusing position. So the way that I'm thinking to fix this, and maybe there's multiple ways to address it, but at least the Lambdaera way that I think we're going to do the first version is going to be,

okay, let's have the concept of preview apps. And in a preview app, like it's a first class concept, your main production application has this thing called preview apps.

So maybe somewhat similar to kind of like what people might be familiar with, like Netlify, or I think Vercel might do the same thing.

And so the idea is that when you deploy to a preview app, Lambdaera will be like, ah, okay, this is a preview app. We're going to completely ignore anything to do with migrations.

The assumption is that we're always deploying a version one, and we're always going to re-initialize.

And actually, we might be cheeky and be like, we'll try restore the existing state into the new version.

But if it happens that you've changed your types, we're just going to reset it. If the decoder fails, and it's a strict decoder, it's slightly variant from Elm's decoder.

So if all the bytes aren't consumed perfectly, then it will fail. So it has to be a perfect decoding from bytes into the value.

Then we've got this separate chain. So that would restore back to Lambdaera's assumptions, which is if you have an app, there is only one production, and there is only one evergreen story.

There is only one chain history of changes into production. And then we've got this separate mechanism, which is preview apps, no migrations, no nothing there.

So the idea is that you do your pull request, you go through all the changes, evergreen doesn't bother you at all.

And then once it's approved, you merge that into main. And now when you're trying to deploy main, now you've got that.

That's when that check comes in and goes, oh, okay, you're trying to deploy. Let's consistently look at what's in production now.

What have you been up to locally? And what's changed? Let's get you to do that migration.

So yeah, that's coming soon. TM, trademark.

Yeah, what I would have imagined to be maybe slightly easier, or well, the previous thing sounds awesome.

But like, if you had a v7 in production, and you're trying to deploy and it's a v10, because you tried to do some migrations in the meantime.

I feel like maybe accepting unseen versions would have been easier.

So there's another scenario that I can see, which could be problematic when your team grows, when you're working with teams,

is that, like, I'm adding a new feature, or I'm changing a feature and the requires to write a migration, maybe complex, maybe not.

And two other people on my team do the same thing. And then we all leave on vacation, and someone else tries to deploy and has to write those migrations,

because we didn't in the meantime, right? But I could imagine like, well, I'm done with my work, let me write a migration that will make a v8,

which we're never gonna ship. But we'll merge it anyway. And then someone else does the same thing v9 and v10. And we ship that.

I just say this, like, not just, I'm just throwing this out, but I'm sure that there's some problems that you have in mind.

Like, nope, that's not gonna work.

Yeah. So the first thing that would happen, remember that the snapshot only happens when you deploy. Right?

So let's say we have three pull requests, and every single pull, let's, so the worst case scenario is this.

Worst case scenario is you have three separate pull requests, all three team members have all changed the backend model.

All three team members independently on their branch have done a Lambda check against production, they've generated migrations, and they've implemented them.

Right? And then now they start merging. So first person merges, they get in first. Hunky-dory, no problems.

Second person now is probably gonna have GitHub conflicts on their pull request. Right?

Let's say even worst case version, let's say it's a clean merge for some reason. Right?

So they don't have any conflicts. Right?

You mean that both would have generated a v8 migration and that would...

But it would have been identical except for the differences. So let's say they changed very different parts of the model. Right?

All of the snapshots were almost identical except for these deep changes. Right?

So maybe the changes ended up in different snapshot files. Right?

So maybe Git's like clever and it's like, ah, yeah, you're merging the same thing except for this.

And I can figure out the merge. I'll merge it for you.

And let's say the third person does the same. Right?

What's gonna happen now is because there's only one deploy that's possible,

whoever gets to the deploy is gonna have to go through that Lambda check process.

And potentially, if Git has tried to be too clever, you're gonna get type errors. Right?

Because something as part of those mergers might not have fully carefully covered things.

So there, if you think of like, you know, like what kind of burden are they stuck with?

You know, all three of you have conveniently gone on holiday.

So it's the worst, worst, worst case. Like what's the absolute worst that happens?

The absolute worst that happens is, let's say it's me. I'm stuck with it.

I go, ah, you know, Dillon and Jeroen have left me with this.

So I go, you know what? I'm gonna delete the migration.

I'm gonna do a Lambda check again. Lambda always does the type snapshots. Right?

So say you did type snapshots, you didn't deploy, you committed them,

and then you changed more types and you did a check again.

Lambda is gonna replace those version snapshots. Right?

Because you haven't deployed yet. So if you haven't deployed version five, if that's the next version,

but you've been changing stuff, it's just gonna keep replacing the snapshots until you deployed.

Once you've deployed, it's gonna be like, okay, well fine. Next one's version six. Right?

So if I delete the migration file, I do a Lambda check.

I'm gonna get now consistent snapshots with everybody's changes together.

If their changes together, the mergers didn't type check, like if there was an Elm compiler error,

I wouldn't even be able to generate the snapshots. I would just get an Elm compiler error first.

So let's say it's like worst, worst, worst, worst case.

The code is broken. It's been merge broken. So A, I fix all the Elm types. Great.

Now Elm's compiling. B, I run Lambda check. It redoes the snapshots.

Cool. C, Lambda has now done the snapshots and it sees the migration's not there.

So it goes, cool, I'm gonna generate the whole migration file for you. Right?

With the placeholders for the bits that I can't migrate.

And so now my job is to go, okay, let me see if I can go to those individual pull requests

and slice out the individual specific migration implementations that everyone's already done.

If I can, and they fit in, and then it type checks, then I go, great.

I've gone through pretty much a mechanical process just following the types

and getting stuff done in.

If instead it was, hey, two people have actually changed stuff in the same thing,

like somebody's both added and removed variants on the same custom type

on two different PRs all at the same time, that's a really great stopping point

for me to look at this and be like, this is nuts.

We need them to come back from holiday and explain what should happen,

or I need to go to a product person, or I need to figure out what's actually going on here.

And I think that's cool because if that happened without this setup,

there wouldn't necessarily be an indicator there that something's gone wrong.

You might merge these migrations together, or actually even worse,

if I think of Rails, I have a lot of experience with Rails as a counterpoint.

In Rails, each developer does their own migrations as a separate file.

So you wouldn't necessarily even be aware that someone else was adding and removing stuff

to the same model because that would all be in different migrations.

There'd be no natural mechanism to see a conflict.

So you could end up in a situation where you deploy these non-commutative migrations

that end you up in a weird schema state, but that nothing catches.

Whereas at least in the Elm Lambdaera world, there'd be warning signs there.

There's things that in the worst case would make you be like, huh, what's going on?

So I think that's pretty cool. That's a decent outcome, I think,

even though there might be some pain.

So to summarize, your recommendation is for everyone to run Lambdaera check

and write a migration, and then those migrations get merged somehow by someone at some point?

No. So my recommendation would be that you do the migration part separately

on the main branch if you've got lots of people editing the same stuff.

Or that would be a point that you communicate together as a team.

It brings that kind of idea of continuous deployment back a step.

So it's not as freeform as some companies may practice,

like, oh, everybody can deploy and we deploy all the time,

we just don't think about it kind of thing.

So it makes that a little bit more centralized.

But I think what you get as a result is you get a much greater ability

to model your business logic directly.

And it means that you don't have to think as carefully about the consistency

or the invariance that you're holding in that migration.

So it trades off, I suppose, some of that kind of maybe,

I don't know if you'd call it decentralized deployment model

to something that's a bit more centralized, but that gives you back

a bunch of guarantees in return, if that makes sense.

So I remember that at one point, someone told me that Arm Review

was kind of slow with their projects.

If I recall correctly, that was James Carlson,

who's a fervent user of Landera.

And I checked it out and was like, yeah, this is a bit slow.

And I figured out why.

I think I know why.

There was an evergreen folder with over 600 versions,

meaning over six, you don't have a migration for every version,

but a few hundreds migrations and snapshots.

And there was code that remained in the project

and that Arm Review had to run to go through,

which is now a bit faster, so I'm not getting those issues anymore.

But yeah, I was wondering, when should you remove those migrations?

Should you? And can you remove v1?

Is there a way to tell, oh, well, no one is using v1 anymore

because Landera knows which client applications are running?

Do you have the knowledge or not at all?

Yeah, so the reason, maybe I'll answer your question backwards.

So here's why you wouldn't want to remove all the migrations.

You wouldn't want to remove all the migrations

if you wanted to preserve a full stack,

full time history, time traveling debugger.

And this is a feature that doesn't exist yet.

But if you wanted to use this feature when it does exist,

you can imagine a slider and if you can slide back like 17 versions

to a point in time backup

or like a log stream restoration in Landera, you could do that.

Because it could go through the migration chain

to get you to the right data source.

So that's why you wouldn't maybe want to remove those.

But saying that...

Can you migrate backwards?

No, but it would be being able to get to the exact state

that you were in for any given version

at any point in time in history.

So if you had the old migration chains,

you can make sure that if the last snapshot was in version 10

and you were trying to get to it, well, this wouldn't happen

because we take snapshots anyway. So it's a bit of a moot point.

But this idea that you could slide a piece of value through,

like the scenario that somebody told me

where they had a customer that came back like a year later

and they started seeing events from a year before.

That person could get a hot reload a year later.

Like their app could go from version 10,

in Jim's case, to version like 1500

and it could slide that value like all the way up all the versions

and if they were halfway through filling in a form, in theory,

the form would upgrade and all their stuff would still be there.

That's far-fetched.

In theory, that would be possible.

From a practical point of view, if you weren't concerned about that,

there's no really good reason to keep more than a few versions.

The only reason you want to keep a few versions is if you wanted to roll back your data.

To say you've done a migration

and you've done the wrong thing in the migration.

You set a dict.empty or a set.empty somewhere where you were being lazy

for the moment because you were like, I'll do the migration later

and you did it and suddenly you destroyed all your users.

If you wanted to go back in that case or if you had a customer

that had been like, oh my god, it's Wednesday but we just found out on Tuesday

that Mario went in and deleted 600 blog posts,

can you please roll back to our state on Sunday?

That happened to be four versions ago.

Keeping those migrations around lets us go, yeah, sure, we can load up that old version

and bam, it'll upgrade through the last four migration functions

into your current app.

We wouldn't have to roll your whole app back, you can have all the new features

we can just roll the data back in a safe way.

I guess you could use Git to restore those migrations again.

Yeah, so you've got it.

Which is a good feature of Git, right?

Yes, you've got it exactly. From a technical perspective, there's no reason

given that the snapshots would have always had to exist when you deploy,

there's no reason you couldn't clean that up or that Lambda Era

couldn't clean that up for you.

The actual reason that everything is there is I decided early on that I wanted

to keep everything really explicit and visible

because in my head that would demystify it.

People could go and actually look at the Elm code and be like,

there's no actual magic here, it's literally just the Elm code

and it's literally just there.

The fact that it's kind of there and you can see it,

yes, we're doing code generation but we're not hiding it away somewhere

and it's not going to break in really weird ways.

If it breaks, it should break with Elm type errors pointing to files

that you can go and look at and be like, oh yeah, that looks wrong or right

or whatever it is.

It doesn't seem to happen often to my absolute bewilderment

but I've always got this terror that I've implemented something wrong

and someone's going to do a migration

and it's just going to horrendously generate the wrong stuff.

So I kind of want that also to be visible so that the user can see

and if I've done the wrong thing, they could fix it.

You could fix the type snapshot yourself or you could modify stuff.

Thankfully, there's some gen issues in the latest migration generation

but so far there haven't been many or maybe only one or two a long time ago

issues with the type snapshotting.

So far the type snapshots extract correctly it seems.

Knock on wood, obviously.

So yeah, it's more a social reality thing.

I think it's a bit of a shame that Elm is not a big part of the Elm community

so yeah, it's more a social reason

for them being there than a technical necessity, so to speak.

So yeah, we might make that more magical in future

and then Elm review won't have problems, side effects as it were.

So the Evergreen migration auto generation

which we haven't really explicitly talked about

but I understand that was a big pain point

that was addressed by this latest release

which is v1.1.

Is there much to say about that

besides that it does most of the tedious work for you?

That's kind of the headline of it.

Yeah, that's the headline.

So the thing that people would run into that I think people would find confusing

is like, say you had a custom type.

Let's say we had the Ice Cream custom type

but with lots and lots and lots of flavors.

Let's say we had 200 flavors.

And say you've changed a field somewhere else

and you have to write migrations.

So like, not sadly, but as a trade-off

of Elm's current equality model, right?

You couldn't just take that old custom type

and cram it into the new one, right?

Like Elm would be like, well, these are different.

They're in different files, right?

They're different namespaces.

These are different values,

even though they're structurally the same.

So in prior to version 1.1,

the migration would only generate the placeholders.

Like, so the top level function types and names,

and it'd be like, okay,

here's the six migration functions I need,

but you have to go to all the work

of implementing what's inside them,

including writing a massive function,

migrate ice cream flavor,

case old of every single old variant

matches to every single new variant.

The only difference being they're in different namespaces,

but otherwise it's like the same text

over and over and over and over, right?

So I think this was frustrating as a user experience

because it was like, if I'd only changed one field,

now I'm writing migrations for all fields

and all custom types everywhere,

and I have to do this every time.

So it wasn't the end of the world.

Some people found like, okay, once I've done it once,

I can pretty much kind of copy paste

a lot of my migration implementation,

but I wasn't happy with it.

If you've done it 600 times.

Yeah, Jim got really, really, really good

at doing these migrations, clearly.

But for newcomers as well, it was really confusing, right?

Like it made them confused, extra confused.

Cause it's like, I'm like, oh yeah,

you generate migrations for your change types

and also for these types that haven't changed at all.

And it's like, once you get through it

and once you think about it, you're like, oh yeah, okay.

I can understand now if I understand Elm

why this is necessary, but yeah,

it was getting you to have to think about something else.

So yeah, long story short now, Evergreen,

where those types haven't changed,

it does a pretty good job at basically generating

a bunch of that for you.

And so it tries to, as deeply as possible,

I mentioned it zips effectively these two types.

It starts at the top and keeps going through them.

So if it's a record, it tries to pair the record fields

by name and so on and so forth.

And then yeah, anything that's been added,

it gives you like a little notice to be like,

hey, this variant has been added.

It's just a reminder in case you wanted some old variants

to map to this new variant that doesn't exist yet.

And also, hey, this variant has been removed, right?

Like what do you wanna do with this old value?

Cause it has nowhere to go.

So yeah, that now tries to be a lot more kind of automatic.

So yeah, the feedback so far is pretty good.

And Jim's happy at least.

He's my, I think he suffered the pain point the most

of anybody categorically.

So he's told me he's enjoying it.

And he says migrations only take him like,

you know, a minute or two now to sort out.

So yeah, that was the call.

Yeah.

That's great.

So one thing I've been curious about,

so this new release also ships with the Elm PKG's JS spec.

And I've been curious like how,

so from what I understand before this,

with a Lambda app, you couldn't just add a.js file

and ship that and arbitrarily add ports

and JavaScript behavior on the page, right?

So I was really curious to understand like,

how does that design decision and how does Elm PKG JS

fit into the concept of evergreen migrations

with the front end and the guarantees you're trying to give

in a front end Lambda application

or the front end part of a Lambda application?

Yeah, absolutely.

That's a great question.

So in short, there's a few features in Lambda

that have actually been there for a few versions.

And I've been kind of trialing it out with some customers

who've ran into certain kind of limitations

and they needed solutions for.

And so what I announced in the last version,

I think was this idea of labs.

So Lambda Labs is like a set of features

that are in Lambda that you can use in production,

but they're marked labs

because it's kind of like buyer beware, right?

It's like, there's a reason this isn't recommended yet

or isn't part of the mainline platform.

So Elm PKG JS, which kind of, yeah,

kind of started from the spec,

which I was hoping maybe it would take off,

but it hasn't yet, but maybe there's still time.

But the idea was to be like,

I'd noticed this problem where a lot of the JavaScript

that people wanted to use is this.

And I think as an Elm community,

we've talked about this problem a few times

where we've got like,

there's certain Elm packages that are like,

hey, this Elm package requires some ports

and some JavaScript set up.

Here's a bunch of long-winded instructions

of varying consistency between packages

of how to do that.

And it feels like,

I don't think it's a massive issue,

but it's kind of like, it's just a bit,

it's a bit painful.

I was always like, oh, how do I do this?

And you paste this and where do I paste it?

And should I put that on this file?

And what bundler do I use?

I was kind of thinking about that experience

with Lambda being like,

what would be a nicer way to do this?

And with Evergreen in mind, right?

And also this restriction where,

we don't want this on the backend at the moment.

So in the front end, it was like, okay,

a great example and a package that kind of got native,

quote unquote, a support for Lambda era

is Martin's Elm audio package, right?

So he added a specific,

like Lambda era front end with audio.

So it's a function where you put your Lambda era app

in that particular wrapper.

And then he has an app wrapper

that depends on certain ports

and add some extra functionality to support

like loading audio and playing audio

and managing like the various,

the state bits of that, right?

So as a user, you can kind of be like,

yeah, I have a normal app

and then I wrap it in this Lambda era front end with audio

and then I get like some extra bits, right?

So I can manage audio and that requires some JavaScript.

So the idea was to say, okay,

well, there should be some way,

like what effectively does this slimline JavaScript need?

Effectively, it needs a way to hook into init, right?

So when the Elm app is being initialized,

we want to get an instance of that Elm app

so that we can bind our subscriptions,

like our port, our inbound and our outbound ports.

So Elm package.js was being like, okay,

how could we, what would it look like

to have like a really delightful standard

for shipping a bit of extra JavaScript

and some ports with an Elm app

in a way where ideally it was type safe, right?

Like it was very clear what ports are there,

what things go in, what things go out,

how should they be used

and for it to be able to check this, right?

And there was some bigger grand ideas about like,

you know, should we do like community verification

that the JavaScript isn't gonna launch a blockchain client,

you know, or something like that, you know,

like maybe, you know, introduce like some safety stuff

to be like, you know, if you do like Elm package.js

install some package,

it's gonna do an Elm install of the package

and it's gonna pull the JavaScript down

and it's gonna set everything up in a consistent way.

And maybe that would make it really nice, you know,

for the use cases.

So there's, you know,

there's like a copy to clipboard example

in the Elm package.js spec

and a few other examples of like, you know,

what would it look like to have these little bits,

you know, little port bindings to web APIs

usually that aren't available natively in Elm.

So yeah, the second concern there was when I was,

so there's a proof of concept implementation of this spec.

The spec has got nothing to do with Lamdera,

but Lamdera implements that on package.js spec,

at least in its first version, as far as I'm aware,

it's the only consumer or implementer of the spec,

which was also written by me, so maybe that's why.

But I haven't pushed it too hard, I guess.

And so yeah, in the Lamdera implementation,

we only have this init,

but in the spec I was also considering like an upgrade,

right?

So what happens when a front end is upgrading,

maybe rather than init being re-invoked

and you having to carefully think about,

well, what happens if init gets invoked multiple times?

You know, yes, you want to rebind your ports to the new app,

but maybe you don't wanna re-initialize

like the audio context, right?

Because the user's browser hasn't reloaded.

So there was an idea of like,

well, could we just do all that in init

and say to people,

you have to think about init as being kind of like

item potent, I guess, you know,

like it can be run multiple times

and you have to do what's sensible when that happens,

or should we explicitly have,

okay, this is an init thing, and then here's an upgrade.

So that in upgrade, you could just be like,

okay, you know, I know I don't have to do any of the init

stuff, it's already there,

I can only do the code that needs to happen for upgrade,

which is probably rebinding ports.

So that's kind of an open question.

And that's why on package JS implementation,

Lambda is still in labs,

because that's not fully handled.

So there is a way,

currently the process is you contact me,

but there is a way to opt out of

the hot reload stuff in production.

And some people have chosen to do that on their apps

where they're like, you know,

I don't have that many users,

or I don't worry about that,

but I've got like some, you know,

complicated JavaScript setup

that I haven't figured out how to handle

this evergreen concept.

And so they might opt out of the live reload.

And then instead what happens is that

when the app deploys and then your front end version happens

it forces a hard refresh.

So it forces a full page reload and reinitialization,

you know, which will lose some front end state for people,

but gets back the, you know, things aren't,

you don't end up in that state where, you know,

things are broken or you're sending old versions

and you don't have the new app version.

So we still get the consistency at the loss

of some front end state.

Does that make sense?

Right, yeah, so how does that fit in with like the guarantee?

So like what would happen if it was just a free for all,

you can run JavaScript on the front end of a Lambda app,

like, or what do you gain by the design decision

to not allow that?

What's the motivation behind that?

Yeah, so there is no constraint actually,

like you can do anything in that JavaScript

on the front end.

What I suppose, like the guarantee that gets made

is that that JavaScript will only run

in the context of an app being initialized,

which isn't a huge guarantee,

but it means that you, your code doesn't have to worry

about the initialization setup

and Lambdaera can continue to control,

like there's a harness, right?

Like that does the evergreen magic and some other stuff

and does special bindings

to make the front end back end stuff work, right?

So Lambdaera platform has a bunch of harness things

that it does for you to make everything seamless

and everything work.

So yeah, the thing that we get from a Lambdaera perspective

is that we let the user kind of plug into that

into a sensible way, but yeah, 100%,

like a user can throw exceptions in that JavaScript code,

the user could crash the app,

like that all the normal JavaScript stuff

comes back into play.

So the user has to be careful, but yeah,

I suppose the only other guarantee they get

is that that in it will be called again

when the upgrade happens.

So it gives them a mechanism if they wanted to try

to make the evergreen philosophy work

with their JavaScript stuff.

If the JavaScript simple enough,

you don't really have to do anything,

but yeah, if you had something a bit more complex,

like an audio context, then yeah,

you could be like, okay, cool,

I know and it's gonna get called again.

I'll put in some guards to check

whether I'm doing the first initialization

or maybe I'm already on my second one.

Mm-hmm, got it, so conceptually,

like if you really wanted to model

what happens when you invoke a port,

I guess you could just say, if an exception happens,

we wanna model that explicitly and you could,

but I mean, since an outgoing port doesn't get anything back,

couldn't you just like let a port call happen

and then say, if it crashes,

we just catch the exception and log something,

something like that.

But I guess having the Elm package JS spec

gives you a box to make those,

to organize things and to make those safety guarantees

within a port, is that sort of the motivation?

Yeah, so the Elm package JS spec does kind of philosophize

about what would be a good overall mechanism there,

which could include those safety guards,

but the Lambda implementation currently,

it just goes to the, at least,

it just kind of goes to the concept of being like,

okay, you put your JavaScript for your individual port

use cases in this folder in this way,

and then anything that's in that folder,

I'm looking for an init in each file

and I'm gonna run it for you, and that's as far as it goes.

So none of the spec stuff about like the types

and the safety of that,

like you still have to implement that yourself.

So it's not like a full automatic implementation

of the spec, but I would like to have that eventually,

like it would be nice if there,

like if that was a feature where you,

if you did like a Lambda install of a package

that did have JavaScript,

that Lambda would be like,

hey, this package is JavaScript,

would you like me to set this up

and create the packageports.elm file

and put all the types in there for you?

And I'm like, then it's just ready to go.

I think that would be really nice.

And then it's just a question of,

would people find that interesting as a standalone tool?

And would the community find that interesting

as a way to be like, hey, this is how we bundle

small bits of JavaScript,

utility JavaScript with our packages.

It's definitely an anti-goal for it to be like,

this is how you drag in 17 NPM dependencies into your project

like it's absolutely not for that use case.

There's an, I mean, there's an example

of how you would use Elm package.js,

but at this stage, you'd have to bundle

some of that stuff yourself, right?

Like you'd have to pre-package things.

Like it's not really for that.

The idea was how do we like in a nice way

get like this ancillary JavaScript for Elm packages.

Yeah.

Well, amazing stuff, Mario.

If people wanna learn more about the latest release,

more about Evergreen, more about Lemdira,

what should they look at?

Yeah, absolutely.

So lemdira.com, probably the easiest starting point

and easiest to remember.

With a B, right?

Oh, you're just ruining all of my naming.

It's definitely not with a B, lemdira.com.

But yeah, that'll get you through some of the pitch

and pretty much straight through to the documentation.

So you can take a look at that.

Yeah, there's some example apps there

that are really quite small and contained.

So I'd recommend people take a look at those.

Evergreen, I wouldn't, I mean, I would say

it probably makes more sense to look at Evergreen

when you need to look at Evergreen.

If people are interested, you can read the Evergreen docs,

but I think it probably makes more sense in practice

when you're actually trying to get from one model to another

in a specific use case for your specific app,

but then going through that process,

I think you can be like, oh yeah, that makes sense.

I've done these changes and this migration

that I want out of that.

Reading it as a high level, I'm not sure how well

that lands.

But yeah, anyway, all the docs are there.

And yeah, as always, we've got a Discord

full of lots of lovely and helpful people.

So if you want to ask any questions or ponder anything,

you're always very welcome in there.

Yeah, that's pretty much it.

Wonderful, thanks so much for coming on, Mario.

Yeah, thank you.

No, it's been my absolute pleasure.

Thanks for having me as always.

And you're in, until next time.

Until next time.

Music