EN NO
← Front page lab

Section taxonomy

Written 28 October 2026, from a question Mads asked while reviewing the lab variants: do we actually have the section in the data for each item?

No. We do not. The section an item appears in is computed at build time, inside themes/emfsignal/layouts/index.html, from two fields that were never meant to carry it: source_type and categories. 41 Hugo filter expressions do the work. Nothing in the bundle YAML, the story frontmatter, or factory.db records where a story lives.

This page proposes the fields that would record it, and lists every value each one would take. It belongs under the lab because the lab decides part of it: if the winning front page turns nine stacked grids into filter chips, the shape of section changes with it.

Status, 30 October 2026

Mads reviewed this page on 29 October and ruled on the three decisions. What he settled is marked decided below; what is still open is marked open. The build has not started. He wants to continue on it this weekend.

Done and live.

Decided by Mads.

Status, 1 November 2026: the first two columns exist

The build started here, and it started the way the ruling above requires: with columns in the database.

Where they live. Both are columns on the items table in factory/factory.db, alongside status, title and short_id. Not a separate table, not a JSON blob.

Column Values Means
funding_opportunity 1, 0, NULL a reader could apply for money or places
happening meeting, conference, NULL what kind of gathering the item announces

Does every item carry them? Every one of the 14,286 rows has both columns, because a column belongs to the table, not to a row. 14,285 of them hold NULL for funding_opportunity and 14,285 hold NULL for happening. NULL means "not asked, not answered" โ€” it is not the same as 0, which would mean "we looked, and this is not a funding opportunity". That distinction is why funding_opportunity is nullable rather than a plain boolean defaulting to false: we can tell an unclassified item from a classified negative one.

Neither column is a cache. Nothing else in the system holds the fact, so neither can be recomputed from anything, which is why update_item_fields refuses them. Writes go through factory_db_lib.set_classification, which validates the vocabulary and appends a classified event before updating the column, the same shape as set_status. A raw UPDATE is still the wrong way in, and the nightly rebuild-status only ever writes status, so it cannot revert a classification.

funding_opportunity is deliberately orthogonal to source_type. Mads' reason: "In that way we can control how we display such items in the future, irrespective of what source category is set to." A call for applications can arrive from an NGO, a journal or a government, and it is a funding opportunity in all three cases. One row is flagged so far, the BioEM Short Term Mission call.

happening answers the question this page has carried since 29 October. Mads: "regarding events/meetings/conferences i think the right call is to add a variable in database called events that can take values like meeting and conference." It is not called events because events is already the append-only log table in this schema, and not event_type because that is already a keyword argument of set_status meaning that log's event type. Mads chose happening on review: "Maybe for even better clarity we should use a totally different name, like happening."

It records what an item announces, not what it reports on. A conference report is not a conference. Mads, 1 November: "The six bioem 2026 daily report should btw not be classified as event/happening. they are reports from an event/happening, not an announcement of the happening." So exactly one row carries a value today: icems-biomed-romania-2026, whose source is titled "ICEMS-BIOMED 2026 Conference Announcement" and whose summary says the conference "will be held May 7-9".

Next on these two. The value lives in the DB, goes from there into the bundle YAML at s10, and renders on the page โ€” Mads' sequence. First display: a label in the story meta line, between the source category and the AI-use note.

Backfilling is safe, and here is the evidence. Both columns are nullable with a default of NULL, so adding them was a plain ALTER TABLE ADD COLUMN, not the 12-step table rebuild that widening a CHECK requires. All six places in tools/ that run SELECT * FROM items set row_factory = sqlite3.Row and read fields by name, so a new column is invisible to them, and the two export paths (factory-db-serialize.py, export-explorer.py) build their dicts field by field, so the columns do not leak into items.json or the explorer until something is written to put them there. Nothing reads them yet, so nothing can break on them.

Open.

Correction to this page. The per-value counts in section 3 and section 4 below were produced by a regex over the frontmatter and are not reliable. Treat them as indicative until they are regenerated with a YAML parser. The source_type counts in section 6 have been regenerated and are correct as of 30 October.

Why it matters

The current derivation is subtractive. Research is literally defined as everything Research minus therapeutic minus screens:

$researchStories = $allResearchStories
    | complement $therapeuticInResearch
    | complement $screensInResearch

So a section is defined by what it is not. Add one category to a piece and it silently changes home. That is the same root cause as the auto-note that fired on the wrong section, and the research category that shadowed a reclassification.

It also loses items. When this page was written, six published stories had no front-page home at all, because their source_type had no filter anywhere in the template. All six are fixed as of 30 October:

Story Was Now Reaches the front page via
reuters-nflpa-emf-monitoring News Press the Press grid
korea-japan-rf-tumor-replication-study News Press the Press grid
daily-express-havana-syndrome-norway News Press the Havana cluster card
super-bowl-lx-emf-controversy-roundup News Original the Super Bowl cluster card
ssm-sweden-emf-report-2025 Government Policy the Policy grid
icems-biomed-romania-2026 Meeting Meeting the conferences grid, via the conference category

Note what the last two rows show. ssm-sweden was fixed by correcting the field. icems was fixed by adding a category, because Meeting still has no filter of its own: six BioEM stories were only ever visible because they happened to carry conference too. That is the subtractive derivation doing exactly what this page is about. A story's home depended on a category nobody chose for that purpose.

Three decisions needed

1. Where should the three unhandled source_type values map? Proposed: News โ†’ press, Meeting โ†’ events, Government โ†’ policy.

Partly decided. News and Government are gone: the four News stories are Press and the one Government story is Policy, both corrected at source by Mads on 29 October. Meeting is open. The proposal to fold the seven Meeting stories into NGO is rejected:

"They are apple and pears, ngo is used for ngo/advocacy so that piece doesn't fit that category." (Mads, 30 October)

He is right, and the mistake is worth recording: NGO answers who published this, and a conference programme has no publisher of that kind. Mapping Meeting to NGO would have bought a front-page home by making the attribution false. All seven now reach the page through conference instead, so the question can be answered properly rather than quickly.

2. Merge the five slug splits? Same meaning, two spellings, and in each case the template only matches one of them:

Merge Status
eht โ†’ eht-news done, fcc-public-comment-rf-remand-2026
events โ†’ conference done, icems-biomed-romania-2026 only
power-lines โ†’ powerlines done, bouwman-catshark-behaviour-offshore-wind-emf-2026
voice โ†’ voices done, three stories
49ers โ†’ superbowl-emf dropped

Two exceptions, both his:

ehs-camping-week-netherlands-2026-second-edition keeps events. A camping week is not a conference. It is source_type: NGO carrying [ngo, events], so it already had a front-page home and the second category costs it nothing. Only icems needed the move, because it was Meeting with [events] and had no home at all. The merge was never about the spelling; it was about one story.

The two Cowan pieces keep 49ers. "Cowans pieces should stay in cluster card for cowan not something else btw." Both are PeoplesVoice already carrying cowan-commentary, which is what drives the Cowan cluster, so they group where he wants. Merging would have added them to a cluster that is not theirs.

3. Delete the nine category values that just restate source_type? Decided: no, not now. The counts below are from the original regex pass and are unreliable, which is its own argument for not acting on them yet. research [104], policy [72], ngo [46], press [26], industry [13], voices [7], commentary [6], local-engagement [6], voice [2]. Across 524 published stories, 113 carry both the research category and source_type: Research, 61 carry the source_type without the category, and 12 carry the category while the source_type says something else. 174 items use one or the other and only 113 agree.

"It is hard to see all implications, but we might clean up and improve categories later. Do we use these for any functionality right now? I think not?" (Mads, 29 October)

Mostly not, and that is the useful answer. Categories drive the Super Bowl, Havana, screens and old-news complements on the front page, the conferences and funding grids, the voice collection cards, and the automatic section notes. Everything else is inert. His intended use, browsing stories by category, does not exist yet.

The automatic notes are the part that must change regardless. A standardised section note should fire on source_type alone, never on a category. Today some fire on categories, which is how a category nobody chose for that purpose can change what a story says about itself.

The model

Three kinds of field, not one hierarchy. Mads' correction, 28 October: only section and subsection are hierarchical. Topic and tags are free. The promotion flags are orthogonal to all of it.

1. section โ€” hierarchical, exactly one

Four, not seven. Top Stories Now, Old News Revisited and EMF Signal in Other Media look like master sections on the page but are not sections: they are flag-driven surfaces. See below.

2. subsection โ€” hierarchical, exactly one

Under scientific-discourse:

Under governance-citizens:

Under scientists-corner:

Under extra:

3. topic โ€” free, zero or one

A collection with its own page at /topics/<slug>/. Independent of section: the Havana card renders inside Press today, but that is a layout choice, not a property of the data.

4. tags โ€” free, zero or more

What is left of categories once the section drivers, the topic collections and the source_type duplicates are taken out. 45 values.

5. surfaces โ€” orthogonal flags, zero or more

Promotion, not placement. An item keeps its one home and additionally appears here.

old_news is a category today, checked by a hand-rolled loop at index.html:1123. It behaves exactly like top_story and should be promoted out of the category bag for the same reason.

Open question for Mads: two separate booleans, or one surfaces list? The list scales if more promotion shelves appear, which the lab variants kept wanting.

6. source_type โ€” retained, demoted to one job

Who published this. Today it also decides page placement, fires the automatic section notes, and resolves voice portraits. Under this model it keeps only attribution, which is what makes placement safe to change.

Regenerated 30 October with a YAML parser, over 524 published English stories. News [4] and Government [1] are gone; Original is new.

Original is EMF Signal's own writing, added 29 October on Mads' approval. It has no external source, so it is the one value that exempts a piece from the quote rule, and the exemption is declared at s7 and checked against the bundle at s9 rather than being a silent skip.

Mainstream is gone, as of 1 November. It was never a kind of source at all. It answered a different question: not what kind of thing published this but how prominent is the outlet. A mainstream newspaper report and a mainstream TV segment are both mainstream and have different source types, and forcing both facts through one column means one of them is always lost. Mads ruled: "I think mainstream should be an own independent variable (true/false)."

So the three pieces that carried it are now source_type: Press with mainstream: true:

I argued the Nobel Prize piece should not be Press, on the grounds that nobelprize.org is an institution publishing on its own site. Mads overruled it with a fact I had not checked: "nobelprize item can actually be classified as press also, since it is co-produced with bbc." Worth recording, because the general rule still holds โ€” an institution's own site is not press โ€” and this piece is an exception to it rather than a counter-example.

Counts on this page are of the published archive, which is why it read three while the corpus held four. The fourth was content/rejected/disabled-world-ehs-day-nearfield-2026.md, in a section whose _index.md cascades _build.render: false, so neither it nor its Norwegian twin was ever built to a page. It was rejected on 4 July 2026 as a duplicate of 9th annual World EHS Day marks global awareness event for electrosensitivity, which is published and classified source_type: NGO. On 1 November Mads chose to recode it to match: source_type: NGO, no mainstream flag. Disabled World republished an awareness piece written by Safe Tech International and Europeans for Safe Connections, and the published twin already records that authorship.

One unpublished bundle also carried the value, asianet-wireless-earbuds-brain-cancer-who-2026, with no story in content/ at all. It is now source_type: Press, so that a value nobody uses cannot re-enter the corpus the day someone drafts it.

Mainstream is therefore gone from the data and, on Mads' instruction the same day, from the vocabulary too: "Yes, strip mainstream also from the validators." It is no longer an accepted source_type in tools/verify-category.py or in tools/generate-verification-report.py, no longer has a colour in tools/publish-draft.py, and no longer appears in the bundle template. A draft that declares it now fails verification rather than publishing quietly. Retiring a value from the corpus and retiring it from the vocabulary are two separate jobs, and only the second one stops the value coming back.

source_type does not apply to calendar entries, as of 1 November. All 34 calendar files carried it, and on inspection they had grown their own dialect: two said Advocacy, which has never been a value a story could take. The obvious fix was to recode them to the story vocabulary, and that is what was done first. Mads then found the better question:

"The source category for such happening/event/calendar listings is non-sensical in a way (they don't make sense for happening/events/calendar listings) since they are not emfsignal news pieces and the purpose of source_type is to enable us to classify items and give the readers some context that helps them judge an emfsignal news piece that is based on a source."

A calendar entry is a pointer, not a piece. It has no source it is based on, so there is nothing for the field to attribute. The value was also invisible: an event page renders source and source_url, and has never rendered source_type anywhere. Its single reader on the whole site was a colour class on the retired old front page.

What a calendar entry is lives in calendar_type instead, which /calendar/ and the front-page mini-calendar actually read, for the dot colour and the row label: event [11], advocacy [5], submission [2], legal [1]. That field already existed. Seventeen entries had been relying on the template's default rather than declaring it, so it is now written into every file, and verify-category.py --all-events asserts the absence of source_type and the presence of a valid calendar_type.

A calendar entry is two files. content/no/events/ is a parallel directory of hand-translated entries, and the whole removal above was done, built and diffed on the English half before the Norwegian half was noticed, still carrying the field. Nothing on the English site would ever have shown it. The check now walks both halves and fails if a pair disagrees on calendar_type. It also reports the four English entries that have no Norwegian twin at all, and so appear on only one of the two calendars: childhood-leukemia-workshop-munich-2026, dennjiha-online-meeting-june-2026, ehs-camping-week-netherlands-2026-second-edition and world-ehs-day-2026.

The near miss worth recording: the first instinct was to add a value like event_listing to source_type instead. That would have been Mainstream again, five hours later, a value answering a different question parked inside a field that answers this one. When a value will not fit a field, check whether the field is wrong before widening the vocabulary.

7. Database-owned fields โ€” the facts that are not tags

Five columns on items that are not part of the tag taxonomy at all, because they are facts about the piece rather than shelves to put it on. Mads set the direction on 1 November: "The value lives in DB. From DB it goes into yaml and thus we can use it live at page." The bundle never carries them. s10 reads the columns and writes them into the generated frontmatter, so the database is the only place they are edited, every change appends a classified event the way a status change does, and repackaging a draft always re-reads the truth rather than preserving a stale copy.

Column Type Meaning
funding_opportunity 0/1 A call for money or places that a reader could apply to
happening meeting or conference What kind of gathering this piece announces
mainstream 0/1 The outlet is mainstream press
short_title text, 80 max Tabloid-length headline for the same piece
short_title_no text, 80 max The same, in Norwegian

All five default to NULL and are NULL on almost every item.

happening is about announcements only. Our own report from a conference is not a conference, which Mads made explicit when I over-classified the six BioEM 2026 daily reports: "they are reports from an event/happening, not an announcement of the happening."

funding_opportunity is deliberately independent of source_type for the same reason mainstream now is. A funding call can arrive from an NGO, a journal or a government, and it should display as a funding opportunity in all three cases.

short_title is the newest and the least settled. Mads asked for it on 1 November so the front page can one day run short headlines the way a newspaper does. For now it is shown above the real title in drafts only, so we can look at a few before deciding whether to use it in production and whether to backfill the archive. Two exist so far:

English Norwegian
MAN-0038 BioEM 2027 goes to Gdaล„sk BioEM 2027 til Gdaล„sk
MAN-0036 Our 2026, in numbers Vรฅrt 2026, i tall

The bilingual question this section asked on 1 November is settled: Mads' answer was "Short title misses Norwegian translation." So it is two columns, not one string, and not one string machine-translated at render time. A short title is editorial writing โ€” it is the headline rewritten to fit a width โ€” and a translation of it is a second piece of editorial writing. Either may be set without the other; where short_title_no is missing the Norwegian draft falls back to the English one and the label says "mangler norsk oversettelse", so the gap is visible during review rather than silent.

That answer also settles the backfill question, which is why it was worth asking first: a backfill now has to produce two strings per item, and the Norwegian one cannot be generated.

Not classifiers

Deliberately outside the taxonomy.

What building it looks like

Revised 30 October after Mads' ruling: "regarding one source of truth, that should be db. And everything else uses values from db." The original version of this section proposed a derived index, with the YAML bundle as master and the database following. That is the wrong way round and is dropped.

First, a finding that makes the case. Neither source_type nor categories is a column in factory.db. As first written on 30 October the items table had fourteen columns โ€” id, url, source_key, source_id, title, source_date, first_seen, display_date, display_date_from, status, status_reason, status_updated_at, created_at, short_id โ€” and nothing else; the five in section 7 above were added after, and none of them is a classifier. So on 1 November, when Mads asked "Are any of the values that are used to collect collections in db?", the answer was still no, for every one of them: source_type, categories, voice_portrait and country_code all live only in the bundle and the frontmatter it generates. Nothing that decides where a story appears, or whose collection it belongs to, is queryable. Both fields exist only as incidental text inside append-only JSON: source_type appears in one stage_runs.input, 17 stage_runs.output and 42 events.data blobs, and nowhere queryable. So when Mads asked whether the four source_type corrections had been written to the database, the honest answer was that there was nothing to write them to. The classification that decides where every story on the site appears has never been in the operational source of truth at all.

That is the gap this build closes.

  1. Columns first. source_type, section, subsection, topic on items, with tags and surfaces in their own child tables. Written through a DB-first helper, never by raw UPDATE, so the nightly rebuild does not revert them, the way items.status already works through fdb.set_status.

    items.source_type landed on 3 November 2026, the first piece of this build. One nullable TEXT column, no CHECK: the vocabulary moves, so it is enforced in factory_db_lib.set_classification against factory_db_lib.SOURCE_TYPES, which is now its only definition. It had been spelled out by hand in story-creator.py, verify-category.py and generate-verification-report.py, which is why retiring Mainstream on 1 November took a three-file edit; all three now import it. All 538 published stories were backfilled from their frontmatter in tools/repairs/migrate-items-source-type-2026-11-03.py, one classified event each, and the resulting distribution matches a yaml.safe_load census of content/stories/ value for value. Slugs resolve to items by exact artifacts.path, preferring the published row in the five same-paper-two-item cases; a LIKE '%slug%' resolves 44 of them to two items and must not be used.

    Still to come in this step: section, subsection and topic, which wait on step 2 because they are derived from the mapping, and the tags/surfaces child tables, whose shape is Mads' open call.

  2. config/sections.yaml โ€” one mapping table, replacing the 41 inline template expressions. The rules become a file you can read, and the migration that populates the new columns runs from it.

  3. s10 writes the computed values to the database, then exports them to the frontmatter. The bundle stays the editorial input; the database becomes the classification record; the frontmatter becomes a build artifact of both.

  4. Templates read .Params.section. No where/complement chains.

  5. section_override: and subsection_override: in the bundle, for when editorial judgement disagrees with the mapping. Same shape as summary_length_override_reason: visible, not achieved by category-fiddling.

  6. Backfill every published item, 538 as of 3 November, of which source_type is already done. It is a pure function, so it is cheap, and it is the point at which the archive's classification becomes queryable for the first time.

  7. Move the automatic section notes onto source_type only. A note must never fire off a category.

The payoff is larger than the front page. Every lab variant currently re-derives the section rules itself, so seven variants are seven copies of the same logic, and /data-explorer/ re-derives them a ninth time. After this they all read one field, and the question which section is this story in becomes a query instead of a build.

One value, many languages

Mads asked the right question about step 1 on 1 November: "how do we solve the multiple language thing on this, since i guess the Norwegian and English item use different words for the values? One variable with one set of values, or how do we do this best? Eventually we likely expand to new languages."

One variable, one set of values. The column stores a language-neutral machine key and nothing else, and every language is a label applied at render time:

Stored once Rendered
source_type = "NGO" Organisation / Organisasjon
source_type = "PeoplesVoice" People's Voices / Folkets stemmer
country_code = "PL" Poland / Polen

The test for which shape a field needs is whether a second language means translating or writing:

So the short_title_no precedent is right for what it is and would be wrong here. A column per language for an enum would mean 535 items each carrying the same twelve strings twice, and a fourth language would mean a migration over every row to add information that was already known.

Country names were the proof. country_code was already the canonical language-neutral value in the frontmatter, and the bug Mads found on 1 November โ€” "fix the country names to norwegian country names in norwegian pieces" โ€” came from printing an authored English country_name sitting beside it. data/country_names.yaml is now one record per country with a field per language, read through partials/country-name.html, and the Norwegian pages say Polen. That file is the shape this section is arguing for, minus the database.

Where the country of a story actually lives today, in order:

  1. country_code: "PL" in the bundle YAML. This is the value. Nothing else is.
  2. s10 copies it into the story's frontmatter, so the built page carries the code, not a name.
  3. data/country_names.yaml is the lookup table, 88 records keyed by that code, each with an en and a "no" field.
  4. partials/country-name.html does the lookup and owns the fallback order.

It is not in the database. items has no country column, so the code cannot be queried, counted or corrected centrally; the only way to find every story about Poland is to walk the built pages. That is the same gap as source_type, and the fix is the same one: the column holds the code, the lookup table holds the names, and the table is a translation file rather than a second copy of the data. Nothing about the table changes when the column arrives. The bundle stops being where the value is authored, and starts reading it from the database like everything else.

Nothing here is indexed. · back to the variants · live front page