The section I took out of the build

I have collected records for a long time, and I picked it up again recently because I felt like buying vinyl once more. I keep the list on Discogs, a music database where each user can register the records they own, and I keep it up to date because it is useful to me: I check it when I am about to buy something. It existed before this part of the site, and it would carry on existing without it.

At some point I felt like showing it here, and that is how Music appeared, one of the entries in the menu: the grid with the cover, the artist, the title, the format and the year of each record, and a page per record with its tracks. For now there is only vinyl, but the section does not have to be limited to it.

This closes the series about how this site was built. The three parts before it describe what the build produces; this one, the section I ended up taking out of it.

Showing that list here turned out to be a different problem from publishing an article, and the reason lies in how the rest of the site is built.

This whole site is built once and served as files that are already written. This section does exactly the opposite, and here is why.

In A blog that publishes itself I described how 11ty writes the HTML of every page during the build and how GitHub Pages does nothing but hand it over. That works because the content is mine: an article does not change unless someone edits it. The collection does not meet that condition —it updates on its own, every time I buy a record and note it down— and that is where the approach breaks.

The data is not mine

When I wrote the plan for this section there were 224 records. There are quite a few more now, and by the time you read this there will probably be more still. Not one of them needed me to deploy the site in order to appear.

That is the whole argument. A static site can ask an API for data at two moments, and choosing between them is not a technical question.

At build time. The runner makes the call, once per deploy, and the visitor gets the HTML with the records already written into it. It is served fast, it works without JavaScript and it holds up even if Discogs goes down. But a new record does not appear until I deploy again.

In the browser. Every visitor makes the call when they open the page, which arrives empty and is filled in by JavaScript. A new record shows up on the next visit, without me touching anything.

I started with the build, and that is how the section was published. I moved to the browser because of that last difference: with the records baked into the build, every purchase would have required a deploy for the site to tell the truth. A static site generator freezes the content at the moment of the build, and that stops being an advantage as soon as the content is edited by someone who is not you.

The deploy got cheaper along the way, and by a lot: taking the records out of the build meant no longer downloading one cover per record on every publish, and the build went from almost three minutes to under a second.

I keep the collection data on Discogs because that is where it makes sense to keep it: there is a community database with an entry for every edition, and I am not going to duplicate it in my repository.

I ruled out the browser on a false argument

This is the uncomfortable part, and the reason this article exists.

The first version asked for the data during the build, from the file that made the requests to the Discogs API and left the records ready for the templates. Inside that file I had written a comment myself explaining why the collection could not be requested from the browser: it would be 224 requests per visitor, one per record, and they would end up failing against the Discogs API rate limit.

It was not true. The collection endpoint returns, alongside each record, a basic_information object with the artist, the title, the year, the format and the cover. Everything the grid draws, in other words. At a hundred records per page that is three requests, whether the runner makes them or a visitor does. The 224 would only be needed by asking for the full entry of every record, which never happens.

The limit did not work the way I thought either: Discogs counts 25 requests per minute per source IP, so every visitor brings their own allowance and spends 3 of 25. There is no shared counter for everyone to use up.

And the proof that my comment was false sat a few lines below it: that basic_information went past it on every build.

I never checked it because I had written it, it sounded confident, and it was in the place where you expect to find the explanation of what the code does. Every time I went past it I read it as a fact, when it was an assumption I had made in the first version.

Note to self: if a comment justifies a decision, I write the evidence into the comment as well. That one gave the conclusion —it cannot be done from the browser— and kept the reason to itself, and doubting a conclusion with no reason attached means redoing the whole argument.

What to check before calling an API from the browser

Calling someone else's API from the client depends on things the API decides, not you. I checked these one by one against Discogs, with curl:

curl -s -D - -H "Origin: https://jestemrique.github.io" \
  "https://api.discogs.com/users/Nrique/collection/folders/0/releases?per_page=100&page=1"

• CORS is open. The response carries access-control-allow-origin: *. Without that header the browser refuses to hand the response to my JavaScript and there is nothing to be done from my side: it is not a permission you ask for, it is one that is granted.
• No credentials. The collection is public and answers 200 with no token, and so does each record's entry.
• The User-Agent is mandatory for clients that are not browsers. From the browser there is nothing to do, and that works in my favour: JavaScript is not allowed to touch that header, and the browser sends its own.
• Discogs sends no caching header at all: no Cache-Control, no Expires, no ETag. Faced with a response that carries no instructions, the browser applies heuristic caching, which means it decides for itself how long to treat it as good. Since the only thing this section promises is being up to date, both fetch calls go with cache: "no-store".
• There are no webhooks. I looked for them because that would have been the elegant alternative: Discogs tells me, I deploy. It is not implemented. The only automatic way to find out from the build would have been to ask at intervals.

Caching is the easiest of the five to overlook, because nothing breaks: without no-store the section would carry on working and would sometimes show records from a few hours ago, which is exactly the problem I set out to solve.

What it costs

It is not free. Asking for the data from the browser has its costs: some I saw coming and some I ran into along the way. These are the ones I know about.

Without JavaScript the page is empty. That leaves out crawlers that do not run it, and the previews you get when the link is shared, which read the HTML as it arrives. There is a wait on every load, short but real. And Discogs becomes a single point of failure for the section: if its API does not answer, there is nothing there.

There is also a limit you do not see until you hit it. A static site has no secrets: whatever the browser needs, the visitor can read. It did not matter here because my collection is public, but the day a token were needed, that token could only live in the build. Putting it in the client is publishing it. This architecture and an API with credentials do not fit together.

If the collection ever stops being public, this section has to be rebuilt, and I can only think of two ways out. One is going back to the build, keeping the token as a GitHub Actions secret —which is where the deploy key already lives— and accepting once again that a new record waits for the next publish. The other is putting something in between that holds the token and talks to Discogs for me, leaving the site static except for that one point, in exchange for one more piece to maintain. I have not decided because I do not need to yet.

Between the API and the page there is always a translation layer

That layer did not change when the section moved to the browser: it is the same code, in a different file. And it is where almost all the work of the section is, because the data that arrives does not have the shape you would want to draw.

Names come with a number after them. Discogs returns Jeanette (6) or Lolita (5), because inside its database that number tells apart artists with the same name. Outside Discogs it looks like a typo, so I strip it.

Literal alphabetical order is no use. Sorting by the first letter piles up a lot of records under the T of The Cure and the L of La Casa Azul. I sort ignoring the article, the way a record shop does, but I show the full name.

And accents do not sort themselves either. JavaScript's default order sends accented letters behind the Z, so Ángel would come after Zappa. The fix is comparing by language with localeCompare.

The format arrives nested and in the plural. One record can have several formats at once —a box of LPs is Vinyl and Box Set— so one of them has to be picked: if there is vinyl, it is a vinyl.

And sometimes the year is missing. Some records do not carry it. I draw nothing: no dash and no "unknown", which take up room to say there is no data.

The missing year is not something I can fix. It is not a field of mine: it belongs to the edition's entry in the community database, not to my copy of the record. You work with the data you get.

The text does not live in the JavaScript either. It travels in a <script type="application/json"> written by the template, and that is where the formats arrive already translated, from a 11ty data file. That way the module is identical for both versions of the site. It is the same idea that holds up part 3: the key Discogs stores is neutral, and each language hangs its own label on it.

One page for every record

Every record has its own page, and none of them exists as a file.

The arithmetic decided it. Generating those pages in the build is one call per record, and at 25 requests per minute that is several minutes of deploy which also grow with every record I buy. Asking for it when the page opens is one call, and only for the record someone is looking at.

What I did want were addresses that look like addresses:

/es/musica/disco/10000-maniacs-blind-mans-zoo-591270/
/en/music/record/10000-maniacs-blind-mans-zoo-591270/

That path does not exist. GitHub Pages serves 404.html for any path it cannot find, so that file does two jobs: it is the site's real 404 —which it did not have before— and it recognises record paths and forwards them to the page that knows how to draw them. Afterwards the JavaScript puts the proper address back in the bar.

The cost is that the response carries a 404 status even though the record page displays perfectly. A visitor does not notice; a crawler does.

And yes, this is another one of the things I have left to resolve. It works, but it is a trick: the server says that page does not exist while the page is showing a record. Since JavaScript was already drawing that page and it was not crawlable, in practice I did not lose anything I had. And if I ever generate the record pages in the build, these addresses do not change: they start existing and return 200.

The number at the end is not decoration either. Out of the 224 there were when I measured, 221 slugs were unique: Annabel Lamb — Once Bitten appears twice, the two 1983 editions, and La Pandilla three times. Five records would be left without an address of their own. There is a second reason that cannot be measured: titles are edited by the community, so someone else's correction would silently break any link already shared. The slug is text for whoever reads it; what identifies the record is the number.

Coming back from a record page to the collection, the browser runs the module again, and it used to ask for the records a second time. The collection is now kept in sessionStorage: what is already there gets drawn so the way back is immediate, and meanwhile the fresh data is requested, redrawing only if it has changed. It is sessionStorage and not localStorage on purpose, because it is cleared when the tab closes: keeping my collection forever on a visitor's disk would be holding on to something nobody gave me.

The window width lies

A record page is two columns —cover on one side, data on the other— and there is no media query deciding them.

When I went to write the threshold, I measured the space that page actually has at different window widths:

At 768 there is less room than at 600. That is not a measurement error: from there on the navigation sidebar appears, which on small screens is a bottom bar and steals no width. A min-width: 768px would have put two columns exactly where they fit worst.

The window is not the container. What counts here is the space left for this page after everything else, and that is what container queries are for:

@container (min-width: 34rem) {
  /* two columns */
}

Underneath there was another problem I took longer to see, because it was not about width. The titles are written by Discogs, and some run to 104 characters; the site's type scale gave the record title 48.8px, the size of an article headline. El Sonido Efervescente de La Casa Azul took eight lines and 439px, and left the text column 231px taller than the cover. Dropping the size to 1.75rem makes it four lines, and the mismatch between columns went from a range of 351px to one of 88.

The things that broke along the way

A @container in rem does not break, it switches off

The record page was in a single column for several weeks and nobody noticed.

The threshold did not move; the width underneath it did. A change from the redesign —it did not touch a single line of this page— left the usable width at 624px, while the threshold at the time, 38rem, works out to 646. The page was twenty-two pixels short of its own rule.

A threshold that is not met gives no error and no warning. The rule simply does not apply and the design stays in its previous state, which is a valid state too: one column looks perfectly fine, it just is not what I had decided. I found it by looking at the page, not by a test. It is now at 34rem, which leaves 46px of slack, and it is still in rem on purpose, because someone who enlarges the browser's text needs the record page to fall back to one column sooner than someone who does not.

The index opened a hole and the search box is what covers it

The collection has an alphabetical index: you press a letter and only that one stays. The sections that do not match are hidden with the hidden attribute, which takes them out of the accessibility tree, as it should.

It also takes them out of the browser's find bar. With a letter open, Ctrl+F only finds things inside that letter, and anyone who has not noticed there is a filter on will conclude the record is not there.

That is why the search box on the page always searches the whole collection, even with a letter open, and typing in it undoes the index filter. If it respected the selected letter it would leave the same hole open, only with a different interface.

Pico calculates instead of inheriting

Twice, in the same piece of work.

The search field stayed 51px tall and would not come down by touching the line height. Pico does not leave the height to the content: it calculates it with a formula that adds the line height, the vertical padding and the border. Changing the line height moved the text inside the box without moving the box.

And the record grid had bullets on it from the start without them showing. Pico declares list-style: square on the li, not on the ul, so removing it from the ul does nothing however much priority you give it. In part 2 I described how cascade layers settle who wins when two rules point at the same element. Layers are no use here, because the problem is not who wins: it is that the two rules point at different elements. The bullets were being drawn in the gap between cells and passed for dirt in the background until a new frame gave them away.

The first click on a letter jumped to the bottom of the page

A link to an anchor does two things, in this order: it jumps to the target and then it announces that the fragment of the address has changed. So the browser jumped with the whole page expanded, and only then did my code shrink it to a single letter. The target stopped existing halfway through the jump.

Measured at 1280, it went past position 12,214 of a 22,272-pixel page and ended at 1,041, which is the bottom of the page once filtered. It only showed on the first click, because from then on you were already at the bottom. The fix is doing things in the opposite order: intercept the link, filter, and set the scroll position last.

What I learned

The question I started this section with was how to get my collection into the build. It was the wrong question, and it took me a whole rewrite to see that the right answer was not to put it there at all.

A static site generator does one very specific thing: it takes the content at an instant, writes the result and leaves it still. For an article that is exactly what I want, because the instant I write it and the instant it is published are the same for all practical purposes. For data that changes without me touching the repository, freezing it means the site tells the truth on the day of the deploy and lies a little more every day after that.

The other thing I take away is not about architecture. For a long time the code held the reason the collection could not be requested from the browser, written by me, and it was false. I did not check it earlier because it was inside the file doing the work, which is where you expect to find the truth. I deployed the first version of this site convinced that a 200 meant it worked, and the mistake is the same one: a comment is not a check either.