bdeshiandClaude Opus 5
b313b729d9
One idea at a time: definition, necessity, priority, layer, recommendation. Every row is now shipped, scheduled, parked with a trigger, or dropped with a reason — so both tracking files are gone rather than left as a parallel backlog. Roughly a third already shipped. A quarter needed only a theme fragment or CSS, including several the list assumed were features: arbitrary `theme.*` frontmatter attributes already reach templates through Extra (verified), native lazy loading already ships and is deliberately absent on lead figures, and in-page timelines are a theme-defined container. roadmap.md gains an "Order of work" — nine bodies of work in sequence, because the sections this review added had accumulated without one. Item 0 is a decision, not work: core sits at ~2965 of 3000 and logging alone wants the remainder, so discover.go moves out *and* the ceiling rises with an ADR. feed.go and web/extras.go cannot move; they are resolver cases, and three parked items now wait on that same seam decision. Two ADRs came out of it. ADR-0083: the dependency rule described a practice it forbade. "Stdlib first, always" and "usually 40 lines of stdlib" would, read literally, have argued against all five modules here — every one is a format or algorithm somebody else specified, and their allowlist comments say so. Surfaced when this agent proposed hand-rolling Reed–Solomon for QR encoding. The test is now whether you can verify it, not whether you can write it, with compute-versus-hold as the counterweight. conventions.md and allowed-deps.txt contradicted it and were realigned. ADR-0084: the Effect primitive said every Effect is "idempotent and re-runnable from scratch" — true of derivatives, false of anything outbound, since a sent message cannot be recalled. Split into artifact-producing and outbound, the latter idempotent only against a delivery ledger, at-least-once with a dedupe key, and ledgers in a -state directory distinct from the disposable cache. Asked for as forward-looking design so webmentions need no redesign; recorded as a shape rather than built, the way extensions.md records the Extension struct. Also fixed a STATUS claiming Effects were "not buildable yet" when the derivative pass has inhabited them since ADR-0042. Findings that were defects rather than ideas: the Atom feed emits no <author>, which RFC 4287 requires. HTML comments in content are published verbatim — verified — so anything commented out is already public, and a check warning is scheduled rather than the engine deleting authored bytes. `Page` has no Date, so a theme cannot mark up dt-published or show an article's own date. A frontmatter naming rule after this agent proposed `archive` alongside the accepted `archived`: no near-homograph keys, and name the benefit rather than the vendor. reference/microformats-and-indieweb.md is new and is the reference asked for — microformats2 properties and IndieWeb rel values mapped to where a theme puts them, with spec URLs, marked scheduled. rel="me" is the highest-value lowest-cost item in that space and needs no endpoint at all. Four things this agent got wrong and the human caught: asserting "the engine never fetches" from a sentence scoped to the content repo, twice; inventing a data-sovereignty argument for an item he had filed as "a cool geeky thing", when the raw Markdown in his git already is the sovereignty; the archive naming; and claiming microformats could be fully implemented today. 15 files, +556/-385. No code changed. Nothing was scheduled that this agent could not name a consumer for. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
45 lines
2.4 KiB
Markdown
45 lines
2.4 KiB
Markdown
# Bengali morphology for search
|
|
|
|
Status: parked — original work, not a sub-task
|
|
Raised: 2026-08-02
|
|
|
|
Exact-form search is scheduled without this and **documents its own absence**. This file exists so that
|
|
"add search" can never be read as having solved it.
|
|
|
|
## What exact-form search already does
|
|
|
|
Whitespace tokenising plus the NFC normalisation that is already unconditional (ADR-0015) finds the word you
|
|
typed, in English or Bengali, if it appears in that form. That is genuinely useful and it is not
|
|
English-only — Bengali separates words with spaces, so this is **not** the CJK segmentation problem.
|
|
|
|
## What is actually missing
|
|
|
|
**Morphology.** বই, বইয়ের, বইগুলো are one lemma carrying case and plural suffixes, and exact-form matching
|
|
treats them as three unrelated terms. A reader searching for the base form finds none of the inflected uses.
|
|
English gets away with naive matching far better than Bengali does, which is why this is not a polish item.
|
|
|
|
Concretely absent:
|
|
|
|
- **No mature Go stemmer exists.** Snowball has no Bengali algorithm, so there is nothing to import — this
|
|
is the case ADR-0083's "take the module" rule cannot help with, because there is no module.
|
|
- **Suffix stripping needs a real rule set**, not a hand-written list guessed at from a few examples. Getting
|
|
it wrong produces confidently wrong results, which is worse than exact-form matching that is honest about
|
|
what it does.
|
|
- **Transliteration search** — typing "boi" to find বই — is a separate feature again, and arguably wanted
|
|
more than stemming by anyone typing on a phone keyboard.
|
|
- **Orthographic variation** beyond NFC: the same word written with different but valid conjunct forms.
|
|
|
|
## Why it is treated as research
|
|
|
|
`harness/roadmap.md` singles this out: *"Bengali tokenisation for search is the genuinely novel problem here
|
|
— solve it well and deliberately, not as a side effect of adding search."* `ideas/exploration.md` repeats it
|
|
as the one item nobody else will solve for you.
|
|
|
|
The failure mode being avoided is specific: a stemmer bolted on during a search feature, tuned until the
|
|
author's own test queries pass, then quietly wrong for every word he did not think to try.
|
|
|
|
## Trigger
|
|
|
|
Deciding to do the linguistic work — deliberately, as its own piece, with a way to evaluate results beyond
|
|
"my queries look right". Until then exact-form search ships and says what it does not do.
|