Discovery & indexing

This page documents how Waidrn makes its public pages findable. It is written for operators and contributors: it describes public behaviour, and reading it grants no special access. Authors and readers do not need it, though it explains why the archive is built the way it is.

Principles

• Server-rendered and crawlable. Every public page is plain HTML that a crawler can read without JavaScript. Discovery never depends on scripts, redirects or an app shell.
• One canonical host. Absolute URLs and canonical links are built from the configured canonical host, never from the request's Host header. Tracking parameters never appear in canonical URLs.
• Public only. Discovery surfaces include only content that is meant to be public. Drafts, private works, signed-in-only works, unlisted projects and suspended or deleted accounts never enter the sitemap, feeds or social cards.
• Privacy by construction. Discovery uses public counts only. Daily aggregates are counts with no personal data, no IP addresses and no identifiers. Social cards for private or missing entities return a plain 404 and never leak a title.
• Own words. Every title, description, feed and card follows the repository's copy policy: Waidrn's own vocabulary, no comparisons, no named platforms.

What is indexable

• The archive front pages and the first page of each public listing.
• Every public, published work and its chapters.
• Every public thread on the boards, and its story reader and threadmark index when the thread is a serial.
• Every public project proposal and its public receipt.
• Fandom landing pages for fandoms that have at least one public published work.
• Creator profiles for accounts that have published public work or public projects.
• Public status, statistics and trending pages, and the trust and documentation pages.
• Public feeds: works, fandom, boards, projects, work chapters and comments.

What is deliberately not indexable

• Search result pages. They are marked noindex, follow; crawling them would create an infinite set of near-duplicate pages.
• Deep pagination. Listing pages and their later pages are reachable and followable but are not promoted to crawlers; the stable URLs are the entity pages.
• Accounts and transactions. robots.txt disallows /admin, /settings, /notifications, /wallet, /studio, /auth, /search, /api, /login, /register, /logout, /report and /gift. None of these appear in the sitemap.
• Non-public content. Drafts, private works, signed-in-only works, unlisted projects, deleted content and suspended accounts are excluded from every discovery query.
• Tombstones. A pruned or deleted item keeps a public tombstone for accountability, but it is not promoted in the sitemap or feeds.

Canonical URLs

• One canonical host, configured at deploy time; the apex HTTPS host is canonical.
• Entity pages use stable, human-readable URLs: works at /works/{id}/{slug}, threads at /threads/{id}, projects at /projects/{id}/{slug}, fandoms at /stories/fandom/{slug} and creators at /creators/{username}.
• Every indexable page carries a self-referencing canonical link with no tracking parameters.
• The canonical host redirects permanently to the other host if a second host exists; both must never serve the same content. This is an operator setting, not something the application guesses.

Sitemaps

/sitemap.xml lists the home page, the main public listings, and every public work, thread, project, fandom and creator, with a last-modified date. It is generated from the database at request time, so new public content appears without a rebuild. Non-public content is filtered out before the sitemap is written. If the archive ever grows past the practical URL limit for one sitemap, the sitemap index will be split into per-kind child sitemaps.

Feeds

RSS 2.0 feeds let readers follow the archive without an account:

• /feed.xml and /feeds/works.xml — newest works across the site;
• /feeds/fandom/{slug}.xml — newest works in one fandom;
• /feeds/boards.xml and /feeds/board/{slug}.xml — board activity;
• /feeds/projects.xml — project market activity;
• /feeds/work/{id}.xml — new chapters and comments on one work.

Feeds use absolute links, carry the author (where relevant), cap at 50 recent items and are cached briefly. Every relevant page advertises its feed with an alternate link so readers and feed applications can find it.

Metadata and structured data

• Every indexable page gets a unique title and description derived from its own data — a work's title and summary, a board and thread activity, a project's state and category, a fandom's live counts, a creator's bio.
• Standard link-preview metadata (Open Graph) is generated per entity, plus a large-card preview image from /og/{kind}/{key}.png: a 1024×537 PNG rendered in-process, deterministic, cacheable forever, and a 404 for anything private.
• Structured data (JSON-LD) describes the entity on the page: the site and its search action; works as creative works with author, dates, word count, language, tags and series membership; threads as discussion postings with author and reply counts; projects as creative works with requester and category; fandom pages as collections with item counts; profiles as profile pages; and breadcrumbs on listing and detail pages. User-provided titles are escaped so they can never break out of the structured-data element.
• Articles carry published and modified timestamps; pages advertise their feed with an alternate link.

Fandom pages and the tag directory

Fandom landing pages gather everything about one fandom in a single indexable page: live counts of works, chapters and words; newest and recently updated works; the most-used tags in that fandom; and the fandom's feed. They exist only for fandoms with public work. Accurate fandom and freeform tags therefore do real discovery work — the tag directory and the moderation tag tools exist partly to keep them tidy.

Public numbers

/stats publishes aggregate counts (works, chapters, words, creators, posts, funded WAID, simulated-economy status) and /trending lists what is active recently. These are public counters with no personal data, and they give press, new readers and returning members a checkable picture of the archive.

Validation and guardrails

• scripts/validate-seo.sh checks the running site: robots policy, sitemap shape and privacy, feed content types and item shape, per-page metadata and canonical correctness, unique descriptions, structured-data parsing, noindex behaviour on search, and the social-card image format.
• The repository's copy policy applies to every title, description, feed and card. Discovery copy uses Waidrn's own vocabulary and no comparison language.
• There are no third-party scripts, analytics, external fonts or CDNs on discovery pages; the social cards are rendered by Waidrn itself from an embedded font.

Operator-gated steps

These are prepared and documented, but they need the operator's accounts and consent, so they are not automatic:

1. Verify the live host serves /robots.txt and /sitemap.xml over the canonical HTTPS host and that the canonical host redirects as configured.
2. Submit the sitemap to whichever search engines the operator chooses, once, from the operator's own account — no third-party script is added to the site for this.
3. Keep canonical-host discipline: one host serves the site; the other redirects permanently. The configured canonical host must match.
4. Monitor: watch coverage errors and 5xx rates in the operator's chosen search console and re-run the validator after major content changes or releases.
5. Scale the sitemap into child sitemaps when the archive approaches the single-sitemap URL limit.

How authors help discovery

• Write a real title and a specific summary; both become the page title and description.
• Tag fandom, characters, relationships and freeform tags accurately — tags are how readers slice the archive, and the fandom pages are built from them.
• Publish chapters normally; each published chapter is a stable page.
• Keep series and work metadata accurate so structured data describes the work correctly.
• No keyword stuffing, no duplicate filler summaries: search quality follows writing quality.

The rest of the trust pages are listed on /docs. The live economy and its invariant are on /economy.