elm-pages 2.0

elm-pages 2.0

elm-pages v2 is here! We discuss the latest features, including pulling in data with the new DataSource API.

PublishedAugust 2, 2021

Episode#36

Introducing elm-pages v2!
Pre-rendered pages
elm-pages has a build step
Parse, Don't validate
Frontmatter
DataSource.File
DataSource.fail
DataSource.Glob
Distill API DataSource.distillCodec
elm-graphql DataSource example from incrementalelm.com
The 2.0 docs are at elm-pages.com
#elm-pages channel on the Elm Slack
Incremental Elm source code
Elm Radio site source code

Transcript

Hello Jeroen. Hello Dillon. Today after 35 episodes we talk about Elm Pages again.

After 35 episodes and actually after our first episode was about Elm Pages 1.0 and about a year

ago actually a little bit over a year ago now and the last I checked episode 1 is actually our most

downloaded episode. Oh that's not true. Oh no it changed yeah I think you might be right yeah I

think you did some good marketing recently. Maybe I must have. Let's double check. Oh yes so Evan's

open source episode that we recorded with Evan Ciplicki is number one but number two getting

started with Elm Pages our first episode. Yep and Elm UI is one download below. That's true. So it's

a it's very frail very fragile number two spot. That's true you can you can place your vote by

listening to one of those two episodes. But if you're listening to this episode well it kind of

counts towards Elm Pages so if you want to support Elm UI then you should stop listening. Yeah that's

true that's true although you still downloaded it so we're looking at download counts. Oh that's

true yeah you're you can't do anything anymore. Yeah sorry. Just share Elm UI more yeah or share

Elm Pages more. Share this episode. So I listened I actually listened back to the original Elm Pages

episode a little while back to just sort of like look at you know take stock of where things have

gone and compared to what's changed with 2.0 and actually our 1.0 episode not to try to gamify the

system of getting more downloads there or anything but it's still relevant. We talked a lot about

just kind of jamstack concepts and and why that's an interesting fit with Elm but the architecture

with 2.0 has changed dramatically so I think we'll get into some of the details there which I think

are very interesting. All right maybe let's start with just saying what Elm Pages is because some

people may not be aware they may they might be rushing towards the Elm Pages one episode but

let's let's redefine it here. So Dillon what is Elm Pages? Okay so at its core Elm Pages is a

platform for pulling in data and pre rendering it to a set of static pages that you can deploy with

this CDN like Netlify for example. So what I mean by pulling in data is there's this concept in Elm

Pages 1.0 it was called static HTTP data meaning that you could pull in data from APIs and have it

there when your page pre rendered. When you built your application? When you build your application

and it outputs for your blog posts it outputs an HTML page for each of those etc etc for all the

different pages so you could pull in data from for example from the elmradio.com website. What that

does it's an Elm Pages application and it actually uses the Simplecast API which is our sort of

hosting provider for for the audio and we use API calls from Simplecast and fetch metadata about

where the mp3s are hosted and actually some some other metadata about the size of the files and

stuff that the RSS feed needs. Generate an RSS feed all using pure Elm code generate the the pages

for each episode and it's using API data and that's available at the pre rendered HTML and

it's also available when your Elm application hydrates so it loads the HTML which has been

pre rendered if you have something like Twitter or Slack basically scraping to get those nice social

previews they're going to be able to get that because it's pre rendered so the core idea is

that like to get those nice social previews and stuff sometimes you need data from different

sources and Elm pages gives you the ability to get that data before init even runs in your Elm

application. So for pre rendering... Just to clarify so you said that it generates a lot of HTML

files and RSS files and I'm guessing kind of whatever you know that you wish for? Exactly

it's just it it doesn't specifically generate anything except what you define including RSS

feeds and all of that that's you know some Elm code that you write to output a string to a file

path and when you run your build it generates that. All right but so you generate HTML files

but you also generate JS files so you it's not just static files and static contents it is also

like just regular Elm applications that you can interact with just the way you're used to with Elm.

Exactly right yeah you have a full fledged Elm application once the page loads that starts by

you know loading the pre rendered HTML and that's available for social media previews that need to

scrape that but then you get a full Elm application and in fact it it takes over and does single page

app navigations from there when you when you load a new page so it's not fetching HTML to go to a

new page it's running as a continuous Elm application. Yeah so it's downloading the the

HTML file the renders it and then when you say it hydrates it replaces all that by a native Elm

application. Exactly and when you transition between pages you can even have a shared model

exactly like Elm SPA and that design was inspired by Elm SPA's design there and that shared model

will persist as you do page changes because it's a continuous single page application from from

there and so for this sort of so there's a different life cycle for Elm pages where there's

a build step normally Elm pages normally Elm applications don't have a build step but Elm

pages gives you a build step so you get this pre in it so when you when it pre renders a page it

calls init for that page but it does not call update so it renders the initial in knitted

version of any given page. And that's where we'll be rendered in or sent to that HTML file that will

be yes pre rendered okay. Exactly right but before init even happens Elm pages is going to resolve

your data sources so the data sources could be in Elm pages 2.0 it can be not only an HTTP data

source so you could do like for example for the listing page for for the Elm radio website it goes

and fetches an API request to get all the listing of all episodes and then for each of those it

makes a follow up HTTP request so you can do data source dot and then and you can do follow up data

source requests from that initial data that you got so it opens up all sorts of possibilities and

that that will be resolved by the Elm pages platform before init is called and your page is

pre rendered at your build step or in the dev server. So all of the data is fetched and then

condensed somehow into one big giant data structure or multiple I don't know yet we'll go into that

later and that will be potentially available to the pages init function. That's right you can pull

it in you can use it in init update and you also have that data in your view as well as in your in

your head function which is where you can kind of put your SEO meta tags to say the Twitter card

previews or JSON LD structured data you know you can pull in all sorts of metadata but you have

access to the resolved data sources when you render those. So what is the benefit of having

that build step? Right so one of my one of my favorite benefits from from having like a pre

rendered Elm site is that so we've we've talked about the idea of parse don't validate a lot which

is like you know one of my one of my favorite kind of design principles that I think about often and

with Elm pages because there's a build step so you can like parse don't validate with a subsection

of your data in your Elm app you can parse don't validate for a model for a page so you can say

hey when I go to this page I need I need to have this data but if something's wrong with the data

and I do you know I do this parsing then I can parse it into something that could potentially

fail right that's sort of what parse means so that's a really powerful pattern but you have

all these states in your application you try to sort of pull them to the top so you don't have

shotgun surgery checking for things repeatedly and having you know not knowing if you have a

good state all over your code. So that's when you're actually doing that logic you have the

data that you need and you don't need to do any branching anymore. Exactly but you do need to

branch at least at the top level because you can't guarantee that things will be in a good state so

you need to check it at some point at runtime but when you have a build step you can actually parse

don't validate and turn that those two tracks of the success track and the fail track you can turn

the fail track into a build failure so you can parse it into either a build failure which will

give you an a nice error message when you run Elm pages build or if you're in the Elm pages dev

server when you're on that page it'll even hot reload as you change your code and as you change

you know files that it might depend on like markdown files that it might depend on it will

tell you that there's a problem so for it like let's take an example so we haven't mentioned yet

but data sources can in Elm pages 2.0 can include a file data source so you could read like a

markdown file so if you wanted to ensure that you have a title you know you could you could

either extract the title by parsing the markdown and checking that it has an h1 tag or you could

get the title from like front matter like a little yaml thing between the triple dashes

that's called front matter at the top of markdown files so you can you can access those using using

the data source dot file module in Elm pages 2.0 and you can fail if so so you can do data source

dot fail or you can pass in a decoder to this this front matter data source and if the decoder

fails or if you were to call data source dot fail or decode dot fail or whatever at any point that

turns into a build failure so you don't have data sources don't give you a result that may have an

error they give you a build error and you run that build and if the build succeeded that's the

okay version if it didn't succeed that's the error version and you get that in your dev server or in

your build step so that I think is huge because you know we love guarantees and we do there's no

way for Elm to guarantee that all your data is valid and that all your HTTP requests succeed and

that the server is available and well behaved and gives you a good response or even you know a a

REST API may return HTML for some reason and you don't know until you do it but since Elm pages

has a build step you actually do it so you do know before your site goes live so you can parse

don't validate at build time not at runtime with some of these bits of data that you pull in during

the build step yeah so with the example of the the block with a title and description if one

markdown file doesn't have a title yes you will get a build failure if you exactly if you've

requested that one would be required yes so you can you can just write your markdown file if it

doesn't have a title because you forgot to write one you have a build failure and in your code

which does all the rendering which does all the logic you can just go ahead and say okay I have

a title exactly and otherwise you will have an Elm compiler error yeah exactly so you get the

nice benefits of parse don't validate turning it into the nice data type that you want that

represents that everything went well but you don't even have to worry about the failure case and this

is actually like it's a very nice workflow as well when so like it sort of changed my workflow where

if I depend on some API data you know like if if I'm pulling in a listing of items from like a

content management system CMS or you know whatever it may be my process has changed where I'll start

by doing an HTTP data source and then I'll do decode fail and what that's going to do so let's

take an example let's say we're building a blog but the blog is hosted with a with a content

management system so we're using an API to get a listing of all blog posts so so we want to create

a blog index page so I would start by making the request to that API and then I would just do decode

dot fail and what's going to happen then is when you're looking at that blog index page in your

browser with the Elm pages dev server running it's it's going to tell you hey I got this HTTP

data but the decoder failed and here's the JSON data if you think about what you would have to do

to get that workflow in a in a vanilla Elm application there are a lot of steps involved

so you would have to you would have to say in init here's an HTTP request to perform here's a

message to to call back to you know when you receive the HTTP response in the update function

you have a clause to handle that message you have something in your model that tells you that you've

got that response oh and by the way it's gonna need to you probably want to wrap that with a

remote data type because it could be loading or could be an error and then well you're gonna need

to handle all of those different loading and error and success states in your view if there's an error

state you're gonna have to turn that error into a nice printable string and don't forget to like

wrap it in a pre tag so you can like debug it in real time and now you can go do that process if

you didn't miss any steps along the way there easy peasy we're used to it I guess but it's a lot of

work so that's I've just become addicted to this process now with Elm pages of this lifecycle where

I just say so so the way a data source is wired in is you have these page modules so you would have

something called like source page blog dot elm and that would map to the slash blog route in your app

and that's going to have a top level value so just like you have init update subscriptions you're

going to have data and that data is just going to be a data source and so for the data source we

would do you know data source dot http dot get we'd write our data would be the content the the result

of the data source data would be representing the data source of what you're going to fetch

okay so that's what that's what data is and then in your view init update if you have those you can

you can do it just with them with it with a view you don't need an update function but you you get

that data so whatever type your data is there's a so you're going to have a type alias data often

it'll be like a record could be a custom type if you wanted to but you're going to have some type

called data with a capital d and you're going to have a top level value called data with a lower

case d and that lowercase d data is where you define your your data source of what you're going

to fetch and the and the top the capital d data is the data type you're going to get back so of course

if you you know if you're adding more bits of data that you're expecting to receive that data type is

going to change you add fields to the record or whatever it may be but there's no message there's

no update clause there's no case statement checking whether it's loading or error or whatever so that

workflow feels so nice to just say you know data equals data source dot http dot get give it your

url and uh and then in your view function you're just like oh here's the here's the data i asked

for it's it's just there or if it's not there the dev server or the build command are telling you

i tried to make this http request and something went wrong yeah so earlier you said that whatever

is returned by the init will be rendered i guess by yes by being called by the view function

my mental model was like okay so the init function returns a temporary model until something has been

loaded because you still need to to do all the fetch commands handle the update and in the view

render an intermediate state and then the render state state but you would pre render the intermediate

state but with elm pages you would actually render the completed view for the completed data

so you render more than what you would get with a normal application exactly because the elm pages

platform gives you this core concept of a data source that it's a declarative way to describe

the data your page depends on statically now um you still have an it an update which can perform

commands and if you wanted to get real time data you can do that in those places with regular http

but the elm pages platform takes care of all the details of resolving that before your init

or view function gets called for the first time that's just part of the platform and so so yeah

you don't uh i think of it as just declaratively describing data rather than like imperatively

saying go and do this here's the intermediary model state along the way as you get up this data

you get you know or you know it's it's a lot more like just defining a task and resolving it and you

can pull in all the data you need and once that's all resolved you just have that data available

so at the beginning you mentioned that elm pages will build a set of html files and javascript

files and whatever and those are already have all the content that they need as we've seen now and

you deploy this to a cdn so that means that if your um cms is down the server is down because

of an outage or something then you can't build because all those htp requests will fail but if

at some point you have failed you have built and the cms is down but the yes the pages application

is released and deployed yes then you don't need to depend on cms and everything is smooth even

though the cms is down exactly that's right and so like i think one of the things people people

sometimes wonder about you know data sources and that sort of thing i mean it um you know we've

kind of talked about how it's convenient but then what about like you know wait a minute so you're

fetching this data at the build step but then i have i have to make sure i get that data before

i load a page because it's hydrating an elm application right so that seems a little odd

and and that's true you do need to um you do need to get that data so like if if um you go to the

uh the root page it gives you the pre rendered html everything loads up it hydrates into an elm

single page application and then you click to the blog page right and now all of that data that you

gathered to to get the listing of all of your blog posts from your cms where does that come from

because it's a single page elm application so is it hitting your cms or what is it doing as you said

it's not hitting the cms which means that even if the cms were to go down that page of the listing

of blog posts would still be there yeah i guess you could still request from the cms new data but

you would have to do that manually yourself you don't have to that's right yeah exactly if you

wanted to for some reason um with the cms in in most cases you're probably going to be using data

sources to get that data but but absolutely in some cases you might want real time data to check

if a live stream is ongoing on twitch right now or to get the number of likes on a blog post or

something like that so the cms gives you that data from the in the data source the data source

gets that data and that data ends up in a json file and so all of the data that that uh blog

index page depended on ends up in this json file and elm pages what it does is it fetches that

json file for you before it goes to that page so so it gets and if you hover over a link you can

look in your network tab and you can see when you hover over a link that it's going to go and fetch

the data that it needs because it usually takes between one to 200 milliseconds to um between the

time your your cursor enters the region of a link and the time you click it because you usually

don't click it at the far edge you click it towards the middle and humans are slow even if

you try to click it fast you're probably not going to do it faster than 100 milliseconds and it turns

out that's actually like enough lead time to in most cases make it feel pretty instantaneous to

have that data available so that when when um the user clicks on the page it's ready and it does the

single page transition quickly so do you have a json file for the whole application or for each

page it's split by page yeah exactly yeah it's split out by each page so it lazy loads as needed

for a page when you go to that page it gets the data it needs is that the only data that is being

fetched no more js no more css it it is and i mean i would love to to have like code splitting but

uh there are some reasons that i can't really do that because the way the elm compiler works it

would um i saw this as a as a bone as a good side i wasn't like oh it should split it up i was like

it would be cool jason it would be cool i would love to be able to do the um to do the code

splitting feature but there are some limitations that i can't really hack it but i would love if

that feature were something i could make use of in the future but for now i mean the bundle sizes are

are pretty reasonable so that's okay but yeah it only fetches that json file and then now the reason

i think it might seem strange at first right because like wait a minute so it's still making

these htp requests right so what's the point besides this sort of convenience and you know

having this data available at pre render which we talked about i think there's an additional

benefit which is kind of like you were hinting at if your cms is down your site is still up

but what if the htp request fails to get that json file that it needs for your data sources

for that page because that one is also hosted somewhere that can go down it's hosted somewhere

that could go down but guess what so is the rest of your application so the uptime characteristic

characteristics they're going to be tightly linked together so if you're the odds are if

you're seeing a page you're going to be able to get that data and also you're hosting it on a cdn

whatever that stands for content delivery network something like that cloud delivery yeah maybe

content delivery network so the idea of a cdn is that you have dumb hosting where it's just giving

you files it's not going through the complication of talking to a server and that server is handling

multiple requests and it has your application logic and it has to go read something from a

database it's just here's a file here you go and in fact since it's so simple it's just reading

files why don't we distribute those files across a lot of networks across the world so we can serve

it to you from the closest network so if you're in japan we're not going to serve it to you from the

east coast of america and vice versa we can serve it up from the nearest data centers because it's

just literally giving you a file that's on the disk so it can do that extremely efficiently that's

like serving from the edge so anyway there are benefits to that not to mention that you can do

data source dot and then so you can get the listing of all of these things and then you can go get

follow up htp requests for each of those and that data is all kind of aggregated into this json file

so even though you wrote and then and did all these follow on requests when it builds it it's

all just there so it just goes and grabs it it doesn't need to wait for that waterfall of requests

so um so i think that's pretty cool also um the uh so elm pages you will probably notice if you

if you play around with elm pages that there's this module called optimized decoder so ilyas did some

incredible work on that uh yeah ilyas van pier is he dutch i think he's from belgium oh cool okay

or is dutch and uh well i'll leave the correct pronunciation to you but uh but yeah so i don't

remember we told this story the last time uh we talked about on pages but yeah i i um i saw

ilyas's json decode exploration package and noticed that it it can warn you if you have unused json

data so if you if you have some json in a in a response and you don't consume it then it it tells

you and gives you a warning and i'm like that's interesting because i've been thinking it would

be really cool to keep track of the json that's used and not and strip out the unused json data

and i i was just i just messaged ilyas and said i wonder if that's possible and then literally like

that same day he's like oh i played around with it and you totally can and here it is

nice which i thought was incredible but yeah so that's the backbone of the optimized decoder

api so what it does is it um when you're running it at at build time it's going to go to the extra

effort which uh is a little bit more expensive than running a vanilla json decoder it's going

to go to the extra effort of keeping track of which json data in in the json blob that you've

consumed using a decoder so if you do decode decode dot field name now it knows that that

field has been consumed if you do you know decode dot index one it knows that array index was consumed

and it actually will null out any data that it didn't need to consume for example so it's it's

pretty sophisticated although it's also pretty simple which is why elm is incredible because it

has this purity and so you can make these kinds of optimizations so that's uh it's doing that

work in the build step in the in the runtime when the user is running it in their browser

it just runs a vanilla json decoder for the performance so you don't incur the cost of running

that optimized decoder but in the build step it figures out which json fields you used and strips

off any unused fields which is pretty nifty so that that comes for free it strips out the

unused fields from the json that you download when you hover a link when you go to a new page right

um from the json in a data source so which ends up being that json so um and it will actually um

uh it will actually merge together if so if you made um if you had a data source doing an

http request to the same url two different places and you consumed one field from one of those data

sources and another field from the other data source and you did data source dot map two and

got those two pieces of data together it's actually going to handle that correctly and

just merge it into a single data source with the consumed data you which you're welcome because it

was a lot of work no i mean i did it because i couldn't couldn't help but but build these features

because i i get really excited about them but um but yeah it it just it just works as far as you're

concerned but there are all these optimizations happening under the hood yeah so there's one

thing that we haven't touched on upon is that data sources are agnostic right yes to where they came

from yeah um you can have data sources for files data sources for http but what you get is data

source of data and where it came from or how you got it or how it was transformed in the meantime

doesn't matter so you can combine things and if you are if you have a blog where you have a lot

of markdown files and you are migrating them one by one to a cms you could you could have one data

source for all the files one data source for the cms and then combine them maybe having one tick

priority over another maybe and but then the rest of the transformation is uh we don't care where it

came from we can keep that data like okay this data source came from http this one came from files

we can keep that data because we can map we can end them all those data but we don't have to care

and that is pretty nifty yeah i i agree that's that's my favorite part i mean that that's my

favorite part of elm pages to me that's that's the heart of elm pages is is the the data source

concept and the engine of resolving them and optimizing them but like you say it's if you have

a data source of list of blog posts that's that's all it is right when you're using that all you care

about is that you have a list of blog posts you don't care where it came from elm pages doesn't

care where it came from it'll take care of resolving and getting the data that as you

describe where you need to fetch it from from uh you know grabbing it from files and from cmss

you can even do data source dot succeed and just get some data in there as uh as our listeners

probably know we we like using succeed because succeed it's a key to success it's the key to

success t shirts are on the way really no not really i lied not yet but they could be

so yeah that's that's the beauty of it yeah so data sources some come from files i come from

http yes but once it's run yeah it actually come from json files that's right that's true so

exactly that's right yeah so even like so when you get uh front matter using a data source what's

actually doing is it's so what what elm pages is really doing under the hood is it's sort of like

this this runtime for for an elm build step and that runtime is using node js so it's using node

js to read files right elm can't read files and it reads those files if you say you want to read

a file and you want to read its front matter what it's doing is it's um it's actually uh parsing

that front matter and that yaml into json and then it gives that to you so you can do a json

decoder so you don't have to worry about doing yaml parsing or any of that you just get json

and in fact you can use an optimized json decoder because if there's some yaml in the front matter

that you don't use then it's just going to get stripped away if it's an unused fail field uh and

so this is one of the really big changes in elm pages 2.0 also is that really um elm pages

there's less to it now it's all just sort of data sources that's all it is that's how you read files

that's how you get a listing of files that's how you um define routes that's that's what you do for

everything and and so if you understand data sources well then you understand elm page as well

that's the one concept that you really need to invest in understanding to understand elm pages

yeah what i'm imagining is that a lot of the code that you wrote before and that you put in in it

update message and stuff is now put into the data sources so like you you could be a data source

expert now yeah totally yeah that's right so like with with elm pages 1.0 there were like a couple

of additional concepts and it's it's pretty cool because i actually like removed a lot of code

going you know migrating from from the 1.0 code base to the to the 2.0 internals a lot of code

went away you mean from projects that use the elm pages or inside elm pages uh well both but uh but

i'm speaking about the internals which i think is really cool because you can do more with it but

the internals are actually less complex because it's just using this concept of a data source to

do more things so does that mean that it's easier for contributors to to help out uh probably so so

so one thing that is sort of on my mind for data sources is the idea so so we've got we've got

different types of data sources we've got htp data sources which we've talked about you you

give an htp you know you describe an htp request to perform and a decoder to use and you've got

that data you can do uh file read data source so you can do data source dot file and you can

do the data source dot file dot only front matter and give it a decoder to get the title and

description from a blog post for example you can do data source dot file dot body without front

matter and it will give you everything but those things between the triple dashes at the top of the

file so you you can you can read in these files and uh there's also a data source dot glob api

which allows you to depend on data that is just describing which files are on the file system so

if you wanted to go to a folder and say hey i want i want a page for every uh or you know do an do an

index page of blog posts and instead of being hosted on a cms it's local markdown files in your

blog folder and so you could do uh data source dot glob of um you know of effectively blogging

slash star dot md and then uh that data source is going to be you know a list of strings of those

um of those files but so basically what what a data source is is it's a way of having you know

having this data available for the sort of pre init phase that it's it's there on initial render

for the pre rendered html it's there for the head as we've talked about but it's really just like

as you said it's just json data and you can map it all together and you've got got that data when

you go to a new page one of the things i've been thinking about is a sort of um having a mechanism

for defining custom essentially like a javascript port where you have some json data and if you

think about what a port is in elm you send json out and you receive json back through ports that's

what ports are in elm that's actually not that different than what a data source is a data source

under the hood is sending json to describe hey here's um here's this file i want to read or

here's this glob pattern and then it sends back data and under the hood it is using ports with

the elm pages runtime but you don't have to worry about that step don't tell anyone and uh so well

what if you could write custom ports and say hey when when this message comes in just like you

write a javascript port send this data back so maybe you want to say here's a an npm package

that i have installed and it you know knows information it's like a database of time zones

that conveniently there were a million of them to choose from an npm and i wanted that or i want to

run a shell command that generates this svg image and so you can take that that json is the input

and the output is you know conceptually you're deterministically um giving the same data for a

given input right just like an http request to an api even if that's technically not true you have

to conceptually think of it as like declaratively representing a single resource so you can say

hey you know you're deterministically representing a single resource so anyway that's that's something

on my mind is a sort of using that same data source mechanism as a way to send back json data

from these sort of javascript ports so that you could do things like running shell commands or

whatever custom things you needed to do to get that data yeah yeah the the issue with ports is

to to get a result always just like with http you're right expected to get a result at some

points yeah but because you have a build step you could potentially say okay if i haven't got

anything after x seconds you you make a build failure exactly and you can create an abstraction

too where you say like hey here's um you know you're gonna get a single port that port gives

you some json data and you respond to that json data however you define so you you have a tag or

whatever you could use lmts interrupt and um and and return you know have this contract with yourself

that if i get this type of data i'm gonna do this type of thing and so you could basically have just

like an async function that gives back data and it's going to be run at the build step and once

your lmap is out there being used by a user it's not doing any of that because it's already resolved

it in the content json so we talked about deploying to cdn because what you build gets deployed to

your cdn can you briefly go about how that works and how complex that is sure yeah so you essentially

so lm pages 2.0 uh there is an init command so you could just do you know npx lm pages init if uh if

lm pages is uh if 2.0 is published by the time this is released then it would just be npx lm pages

init if it's not then i believe in you dylan you can do it i believe in me too yeah i've got four

weeks you listen and know whether he succeeded or not that's true yeah uh so but yeah well i'll i'll

put the instructions regardless in the show notes so then you've got a skeleton lm app lm pages

application and uh you know if you're deploying it to netlify you can npm install the netlify

cli and then netlify init to get a netlify.toml configuration file that's going to have a command

for the command i think it npm installs for you lm pages uh apps are going to require an npm

install because because lm pages is an npm dependency and then um and then you do lm you

know lm pages build and it generates the files to a dist folder um in your netlify config you can tell

it that the folder to publish is your dist folder and then you push that up to netlify and it's gonna

run lm pages build and publish the dist folder and now you've got an lm pages app okay i imagine you

still need to sign up for netlify maybe pay the free tier is quite generous so if you're building

a hobby project chances are you you won't have to pay but there are lots of good hosting providers

out there and netlify i've been very very happy with yeah that's what i use also for my plugin and

i don't think i've paid anything except the domain name obviously all right so lm pages 2.0 is

simpler than lpages one for the people who are using it already uh will the migration be difficult

or do you have any tips on how to do that yeah that's a good question so well so there are a few

concepts that maybe we should um introduce first that we haven't talked about yet so with lm pages

2.0 they're um so very similar if you've used lm spa then these patterns are going to feel very

familiar um i got a lot of inspiration from ryan's approach to this so there is a uh source slash page

folder page with a capital p so if you have a page dot blog dot elm you know page dot blog module

like like we described before it's a file based router so page dot blog corresponds to slash blog

page dot index corresponds to the root page slash and you can do a dynamic uh route segments and so

you could do um page dot blog dot slug underscore and that would correspond to slash blog slash

sometimes they write it as colon slug meaning that it's a dynamic segment that could it's a

placeholder could be anything so slash blog slash hello so the way that that that works is first of

all um you can use elm pages add so you could say elm pages add blog with a capital b and that's

going to add source page slash blog dot elm you could do elm pages add blog dot slug underscore

and that's going to give you the uh the route corresponding to slash blog slash colon slug

so that's that's the first thing to know between elm pages 1.0 and 2.0 is that whole setup is

different that there's a file based router and so in 1.0 the routes were uh based on files in your

content folder usually they were um markdown files and uh so if you had uh content slash blog slash

hello dot md then you would get a corresponding route blog slash hello yeah okay that's what i

have in my blog as well and that is not going to be the case in elm pages 2.0 right so you can

build that functionality using the building blocks with 2.0 or you can do anything else you could uh

have your blog post coming from a cms with with uh 1.0 you couldn't um there wasn't a way to

have a data source determine which static routes you were going to have so in version one yeah you

had static htp yes to get data from external resources and you had the file based routing

right but it was marked down file based routing so it was a one to one mapping between a a url

and a file whereas in elm pages 2.0 it's a one to one mapping between a route and a module

and when i say route i mean blog slash dynamic slug is a route not blog slash hello that's a url

it's a page so in 1.0 one file in your content folder markdown file usually was one url in 2.0

it's a module page dot blog dot slug underscore and that's that route which actually gives you

multiple pages and so if you wanted to translate that to elm pages 2.0 so in elm pages 2.0 you have

these page modules very similar to uh these page modules in elm spa so if you're defining a pre

rendered route which is what we've been talking about this whole time and spoiler uh that's not

the only thing it's it's sort of in an alpha phase but um i i do have some sort of alpha phase

functionality for doing sort of serverless and on demand on demand routes which we can get into

that's a whole nother can of worms all right but let's just focus on pre rendered routes for now

which is the ones that are uh the html is pre rendered at build time and so um so to do that

you've got so you've got a top level page which is just like your your main in a in a regular elm

application but it's the page for each page module so each page module exposes this top level lower

case p page value so if you do page dot pre render you give it head which lets you um you know do the

seo data that we talked about you give it data that we talked about and that data is going to

be the data source it resolves that is available before init and then you've got routes so routes

are the uh the routes that you're going to pre render so for a blog post where does that route

come from so now how would you how would you describe routes if only there was some declarative

way to describe some data you needed to gather to define your routes yeah if only we had a some kind

of data source yeah if you could like get some source of data declaratively yeah so it uses the

data source api so a route is a data source of list of route params so um now route params are

the route params are completely based on your page module's name because it's a file based

routing based on the name of the module so so for our example of page dot blog dot slug underscore

the route params are a record with lowercase slug equals string with no underscore so

those dynamic segments are translated into into fields in your route params record if there are

no dynamic route segments then the route params are empty like for page dot blog so the routes

you just do a data source of of those route of a list of route params so for example

uh we could say routes equals data source dot succeed and then a record slug equals hello

and now when you go to you know when you run the dev server elm pages dev and go to go to the

browser open it up go to slash blog slash hello it's going to be rendering your page if you go to

slash blog slash getting started with elm pages it's going to give you a 404 page and it's actually

uh i i put a lot of love into that 404 page to to actually like give you feedback saying hey this

matched this route but it didn't match any of the routes for this type of page the pre rendered

routes and that's why i'm showing you a 404 uh and then it's going to list out the routes which

we only defined hello as a route yeah only the dev server right only the dev server but yeah

i've seen a screenshot and it looks pretty cool again about trying to give the best air feedback

to the user because there's so many things that can go wrong like what data sources are you

fetching how is it mapped what is the resulting data source yes and you don't you know as a user

you you don't trust the tool if the tool doesn't give you feedback you're not going to trust it

it doesn't matter there's no way to earn a user's trust if you don't give them feedback you can't

just say trust me i'm doing the right thing because if you're not giving them feedback

feedback creates trust and if if the uh if the dev server is always right and has no bugs in it

and is very consistent and reloads anytime configuration changes and pages change or whatever

it actually doesn't matter if it's not giving feedback you're still not going to trust it so

yeah i i i really like that as a user i like using it and getting that feedback this is how you know

that dylan actually cares about you user like what he just said that was his his heart that's

just spreading his love it's a love letter that was that was my love letter to the user um yeah

that's how you create routes now if you wanted to create routes to match the v1 behavior of elm

pages what you would do is um so v1 there was the content folder so you you would create a glob

pattern to um to match the content folder so so now the way the glob works you do you can do i

don't want to get too bogged down by it but you can look at the api docs for it and i think the

the docs are pretty helpful but you can do glob dot match on a literal like content slash then you

like content slash then you can do glob dot capture to pull a piece in so it's sort of this

um it's very similar to the elm parser api um the parser api has you know pipe equals and pipe dot

and those are those correspond to glob dot match or a glob dot capture is the pipe equals that

eats the thing and glob dot match is equivalent to pipe dot which ignores ignores the thing yeah

so anyway you you create a glob pattern you can extract the information you want from it so if

you just want to get the slug you can if you want to get the full path you can get that but so you

get that glob pattern and you if you were to take uh if you were to just capture the slug part so

that means for content slash you're just going to match that for the you're going to do a wild card

that you capture the star part and then the dot md part you're just going to match which means

you match against it but you don't get that data what that means is you've got the slug yeah so

you're building a glob pattern like just the way you would type it in a terminal like uh content

slash star star or star slash no content slash star dot md and so you're building a glob and

you're also saying this this here is what i'm actually caring about i care about the star part

maybe i care about the content part maybe i don't um and then then you have a you transform that

to a data source exactly then you yeah then you say to data source that's right you can do glob

dot to data source and that kind of finalizes the glob gives you the data source and yeah and again

you know what you're describing there that's that's parse don't validate so you get just the

data you want rather than just saying here's my glob give me the list of matching files which you

just built up but then you have to strip off the stuff afterwards again that you already sort of

you had it in this nice neat format where you just could have said i want this data well you

can just do that so then you can uh you can map each of those so you could do like data source dot

map and then parentheses list dot map rep prams and that would turn each of those into the rep prams

because the rep prams is a record you know it's a record constructor that takes a single string

that's gonna that's gonna be your slug so now you've just defined a glob data source that's

gonna correspond to all of the files in the content folder now you hinted at the double star

and you can do a recursive wildcard and so if you if you really wanted to just exactly match

the elm pages v1 behavior you could use that to create a glob pattern that matches every nested

file in the content folder and and that's going to be all of your routes and you could actually use

the um uh so elm pages the file based router we kind of hinted at this last time with ryan that

uh elm pages has this sort of uh special feature of the splat routing so if you wanted to if you

wanted to truly have exactly the behavior that elm pages one had for its for its routes where

the content folder is directly mirrored in the urls in your application that's what you would do

you would create a splat route so you can do elm pages add all uppercase splat splat is in all

uppercase the other dynamic route segments are capital not all uppercase so all uppercase splat

and you could do double underscore which is going to be an optional splat which means it's also going

to match the top level root route okay what does splats mean what does it come from there are

different terms for this sometimes people um sometimes people use the term like rest params

like in a you know in some programming languages you can have a variable number of arguments var

args and then you you get those as an array uh netlify's redirect routes use the term splat to

say whatever comes after this just capture it give it give it to me in a list so there are different

terms that are used for it but splat is just like you know uh similar to like the um like you know

ecma script javascript calls it spread sometimes spread operator dot dot dot uh it's basically just

like give me everything that comes after this yeah and i'll i'll handle it somehow yeah exactly

i'll i'll slurp up all those values so that's um that's what splat does is for the um so blog dot

slug underscore is gonna match exactly one segment after blog so if you do blog slash hello slash

world it's not going to match that but if you did a splat route then it's going to match it's it's

going going to match anything so you can you can actually do you could do like page dot blog dot

all uppercase splat underscore and that would match blog slash hello world it wouldn't match

slash blog because we did a single underscore so it's a required splat the double underscore

is an optional so it's the same with um like for example in the elm pages 2.0 docs i have a route

that is um docs it's page dot docs dot section underscore underscore the reason i do that is

because if you go to if you go to slash docs slash what is elm pages then it shows you the what is

elm pages docs section but if you go to slash docs i actually also want that to show you what is elm

pages so i want the same page module to be responsible for rendering and showing the

metadata and everything so i want to use the same page module for both of those routes whereas in

the case of like page dot blog that's a totally separate page listing blogs compared to page dot

blog dot slug underscore so i want those to be separate page modules so that so that's the

optional double underscore at the end of a segment okay so i imagine that if if somehow um two routes

or two modules cover the same route like yeah uh then you have a build failure well it's actually

gonna just uh it deterministically will match routes and you can shadow routes and the more

specific one will take precedence so like for example if you had you you could have splat routes

and you could also have a blog route so if you have like a top level splat that's gonna match

slash about slash you know 2021 conference event slash whatever you could handle all of those with

splat however you're getting wherever those pages and their data lives but then you could also

specifically have a um a blog page page dot blog and that that takes precedence because that is a

specific route that you defined so um yeah so you so the route that you that it parses into and by

the way there's a there's a route module that's generated so so elm pages just like lmspa you you

get a generated route module that has this route custom type and you can use that to like uh

programmatically link to pages so the specificity is deterministic and it doesn't depend on which

routes you've defined so like for example if you had a splat route and you said for that splat

route you're going to handle a route of blog but then you also have a blog route page dot blog

well the the blog route is just always going to match and it doesn't depend on whether on whatever

else any other routes have so the the the routes that you'll parse into is deterministic based on

the specificity of routes yeah but i probably should specifically if you have two matching

routes i should give a build error for that specific case but but in terms of um like i think

this is just a helpful concept to keep in mind that the uh what what a route matches is not based

on uh the specific routes you have it's based on the specificity of the page modules so you know

what a route will decode to regardless of which routes are pre rendered or exist in your app if

that makes sense distilled is not yet in on pages right it is oh it's it's fully stable so uh i've

seen you talk about the concept of distilled uh data can you explain what that is yeah so in

frameworks like nextjs and a lot of these javascript based static site frameworks or jmstack

frameworks the way that you get data to the page is you would say like in nextjs you'd say get

static props and then you have an async function you make http requests you read files you glob

you whatever all these things that data sources do again at build time yeah that's right so with

nextjs it it's going to do some fancy sort of static uh transformation of the your code and it

actually splits apart the parts of the code that it's going to run on the server and that it's

going to run on the client and get static props is going to be run on the server or whether that's

a build server or a serverless function or whatever it may be and so um so get static props

you get all this data and then you you give it a serializable json object and that's the data that

gets sent to the page so if you wanted to split out data you could do whatever complex work you

wanted to to do http requests and read files and then you could just say oh here's this prop the

number of blog posts is 5000 but you could just say blog post count colon 5000 now that's the

data that shows up on the client bundle right yeah it doesn't matter whatever other data came

before that with the data source api and elm pages it's actually keeping track of what data

you depended on and just pulling in that data for all your data sources for that page right

yeah and it puts it into the json file right exactly strips down a bit but it strips it down

exactly so if you are depending on some json fields and not others it's going to take care of

only having the data you depend on in the bundle which is great but sometimes if you want to count

count count the number of posts you don't want all of that data so there are many many cases

where where you might want this you know i i've been um building like a digital garden for myself

that uses back references and so the back references track every single uh every single

post that refers back to the current post and so you know that can now that what that means

is that it's like a it's like a glob data source to list out all the posts and then it's a follow

up you know data source dot and then for each of those uh do a data source dot file dot and then

read the file if i were to just let that data uh show up in the bundle then it would be pulling in

50 plus pages of markdown files or however large the the number of files grows to which is not good

and it's not not needed because all i need is i need to know for each page does it refer back to

this page or actually i don't even need to know that i just need a list of pages that refer back

so it could be three three strings a list of three strings rather than 50 markdown files with all of

their contents so with the distil api that's that's what that's for and it's the distil api

lets you work in more of this way that that like next js get static props works where you just say

here's the data make sure this data is there for the page to render with and so distill so i made

like a couple of distill helpers distill codec and distill serialize codec and those are for uh

mini bills codec package and the siri and martin stewart's serialize package elm serialize and you

know the the reason they're codecs is because the way to still works is you need to teach the data

source how to turn that into some json that it's going to include in the json for that page so you

need to encode it you need to teach it how to encode it to put it into the json file and you

need to teach it how to decode it to take it out of the json file it turns out codecs are really

good for that although actually it doesn't really matter because it's always going to encode it and

run the decoder so you're actually never getting the data before it went through the encode decode

cycle so if you did it in a non reversible codec fashion it actually doesn't matter because you're

you're not going to see the data without being encoded and decoded whether it's in the dev server

or the build or whatever that's probably what you want in most cases so um but you can just use the

low level distill if you want to provide a function to encode the json and a function to decode to

json for whatever reason but one really cool thing about distill is that so for one thing uh

you know if you wanted to get that list of back references you could do you know distill codec or

distill serialized code so it's you know data source dot distill codec or data source dot distill

serialized codec and then you give it a unique key so we could say like back references in this case

and then that's just a string and then we give it a codec since the the data source that we arrived

at in the chain up until this point was a list of string it was data source list of string then we

just need a codec of type list of string so we could just say codec dot list codec dot string

or serialized dot list serialized dot string depending on which package you want to use there

and that's it now it is now it's going to strip down just that data to a list of strings so if

you look in your json for that page that's what you'll see also uh any computation that came

before that is not performed in the client side in the browser it's only done in the build step

which is also cool so if you wanted to compute some fibonacci number and then distill that into

an int you could do that and it wouldn't actually run the fibonacci code in the browser so that

opens up some use cases and it's a good thing to have in your toolkit so if you if you're noticing

large json files that that's what you'd want to reach for and i'm thinking about ways to to give

some some nice feedback to tell you if your json files are getting large just to as a feedback

mechanism so i might do that in the dev server server and or the build command that'd be cool

yeah so um i've asked this question in during the first episode also uh but when would it make sense

for someone to choose elm pages over a regular elm project i think it's worth you're addressing

you're addressing this question i don't know if your answer changed well certainly uh the the

things that you can do with elm pages 2.0 are a lot more flexible and it's it's just generally

become more mature but i mean if you're not if you have no use in pre rendering pages then you

know why would you pre render pages if you if you're not using data sources then why would you

use elm pages you know that that's sort of my thinking on it is like if you're like oh wow data

sources that's great now i can have this data there and um i can think about this this data

being there for this page without having to go through the whole update message cycle and all

of that and it's declarative data and that that abstraction works really well for this use case

then elm pages might be a good fit if if it's like i don't really need this data source thing

then what then why are you using elm pages that's that's sort of my rule of thumb yeah if so if you

don't have any data to pre render yeah you use a regular elm application yeah or elm spa might be

a great fit yeah so um yeah so you know that that's gonna tend to be like content sites and

now i i really do believe that um content sites deserve good social previews that's just something

i i believe like to me it's super anti climactic if if i put some love into like writing a blog

post or something and then share it on twitter and it's like incremental elm.com slash blog slash

whatever can you please stop pointing fingers at me please i want the love i put into the

posts to be reflected in like a polished looking social preview and same for a marketing page or

a restaurant you know restaurant menu you know site or a film festival site or whatever it might

be like these sort of um content based sites i i think deserve nice previews and also like i think

you can leverage like elm pages at its core is giving you this tool for for pulling in this data

into nice typed elm data that you can use on your pages and like i just think elm is a very good

tool for that like elm has an incredible type system it's so nice like transforming data and

passing data around and parsing data right that's like that's what's amazing about elm like any like

even somebody who's not sold on elm is like yeah i really like doing that sort of thing with elm

you know that you hear that sometimes like well that's sort of what what data sources are really

good at is letting you use data that way like there's no graph ql abstraction because we've

had something much better than a local graph ql database for getting files and stuff we've got elm

and elm's type system and elm's feedback mechanisms so i'd much rather use that now that said i i

absolutely use graph ql and elm graph ql to like fetch data from cmss like uh like sanity or

contentful that expose graph ql apis because why not do you have a graph ql data source yeah uh

well um so uh elm graph ql uses like vanilla json decoders not optimized decoders and you can use

that with um with um with elm pages um you just have to use these unoptimized requests but they

are optimized in a different way right exactly right yes exactly because graph ql already sort

of does this data splitting by returning exactly the data you requested so it would it would be

unnecessary for that so yeah okay yeah so that works quite nicely so i i can i'll share a little

link to like a snippet that i have for just performing and decoding that using a data source

but it works works very nicely but why but why would you use an abstraction like that i think

that you know frameworks like gatsby js for example use this graph ql abstraction because it's

a typed way of getting data but elm i couldn't ask for a better way of dealing with types data

than elm so yeah one of the principles of elm pages is to let elm shine and so anyway if you

if you find that declaratively describing the data that your page depends on and transforming

that and having it available on page load is a useful abstraction then chances are elm pages

will probably be a pretty good fit yeah i think just marketing sites in general if you if you

want to reuse like the same uh ui elements uh yeah can be a good fit even if you don't have

any content because at least we'll get um uh pre rendering and therefore better seo metrics yeah

so that's useful even if you don't have any data sources to to fetch from you can yeah do uh data

source dot succeed to have to have static data sources right yeah or just uh hard code the data

into your views or whatever it might be yeah absolutely that's that's true that's true right

but if you don't need pre rendering or data sources then probably not what you need yeah

but yeah i remember that we talked at one of my previous company about using elm and one of the

concerns was seo so elm pages somehow solves that yeah yeah like you don't need seo you know you uh

if it's like internal user authenticated pages no yeah yeah then you don't need it it's not going to

help you out there so seo is you know i mean one one rule of thumb that people sometimes use for

jamstack sites being a good candidate for jamstack is do do people see the same content when they go

to the page and you can sort of enhance it with some user specific content if somebody's logged in

but they're seeing an ecommerce page and they can see their shopping cart items but otherwise it's

the same exact page for everybody and has the same inventory for everybody or whatever it may be

and and those things really need to have seo and you want to have good first paint times as well

okay i think it's time to to close so elm pages is a project that takes data sources of any sort

and builds static html files rss feeds css files javascript files and therefore elm applications

that you deploy to a cdn and if at some if at any point you have something that wasn't as expected

you get a build error and during the elm application you still have elm for all the

nice things that it has is that a okay summary i'd say so yeah that's a great summary now there

are several things we didn't get to touch on today so we might need to do an elm pages 2.0 part 2

episode uh just to bump up our uh percentage of downloads for elm pages related episodes a little

bit but um take that elm ui uh we didn't talk about sort of uh the the api routes and you know

file generation part of it we didn't talk about serverless and on demand which is more alpha so

maybe when that stabilizes we can revisit that but yeah that uh that's really the heart of elm

pages i think we covered it pretty well and you you put a lot of effort and love into the dev

experience also we didn't talk too much about it but from what i can see without having tried it

too much it really looks like you put a lot of love into that so that's thank you yeah yeah no

that's that's true i um i'm really pleased with the architecture in fact like the whole exploration

of elm pages 2.0 began with trying to um uh sev forno who's been like really awesome just like

discussing all these ideas of directions elm pages can go with me um was sort of saying like hey i

wonder if elm pages could do serverless rendering i think that's where the elm pages 2.0 exploration

started because that ended up being a giant nerd snipe which uh i i welcome it was incredible and

uh i ended up building that and saying like oh you actually can and actually now that i have this

mechanism for give me a single page and i will render that page instead of doing like full

builds and rendering everything well hey that's really cool for a dev server because we can pre

render pages in the dev server and we can do everything in this sort of just in time fashion

rather than uh so you know in in the style of tools like vt and snowpack just very rapidly

doing the minimum amount of work to give you hot reloads and stuff like that and so anyway yeah the

the the architecture um i'm very pleased with and and i've tried to make the performance really

noticeably fast and it's it's feeling really nice to work with all right so how do people get started

with the elm pages uh check out the docs there's a fresh doc site for 2.0 and join the elm pages

channel on slack i'm always happy to answer questions there also i'll link to some uh sites

using elm pages 2.0 and some code bases uh i haven't upgraded uh elm radio to 2.0 yet but i

will soon but incremental elm and the elm pages doc site are in 2.0 so there's those are some

code bases you can take a look at too what is the link for the elm pages documentation well assuming

that it's published uh by the time this episode goes live uh that would be elm pages elm pages

dot com slash docs all right well all right i still have plenty of questions like i know

well i think we're out of time yeah yeah i think uh well let's let's revisit this one uh

when we've got some new stuff to talk about here that sounds good

all right well until until next time until next time