TNIO AI bot

The TNIO Librarian was a Discord bot I built for the TNIO community, The New Imperial Order. The community keeps its rules, rosters, character records and storylines in a Google Drive archive of Docs and Sheets, and the bot answered members’ questions from that archive. It chose a source to suit each question, drafted an answer in an in-universe voice, cited the source titles, and refused to send answers that failed its own verification.

It ran on ai-bravo-02, a container on grey-server. The guest was already stopped when I archived the project on 2026-07-25, and I deleted it on 2026-08-09. The records don’t say why I shelved the bot.

Item	Value
Guest	ai-bravo-02, CT 105 on grey-server, Ubuntu
Size	6 vCPU, 23.81 GiB memory, 8 GiB swap, a 100 GiB root volume on ssd-lvm1
GPU	Seven NVIDIA device nodes passed into the unprivileged container
Services	lore-discord-bot.service, lore-search-http.service, lore-rag-sync.timer and ollama.service, all as user services
Corpus	45 Docs and Sheets in the archive’s active manifest
Tests	Policy and corpus regression suites, and a 214-case evaluation suite
Archived	2026-07-25
Deleted	2026-08-09

What it answered

The bot sorted every message into one of three kinds:

• Archive questions went through retrieval and source verification.
• Ambiguous factual questions got a clarifying question back.
• Casual messages got an in-character reply without pulling unrelated archive records into it.

The archive covered the guild rules, character and faction progression, the military, intelligence, Sith, droid, starship, beast, dice and combat rules, the current offices and rosters, character and storyline records, and ledgers and registries kept in Sheets.

The bot didn’t treat every record as equal. A question about who currently holds an office preferred the organizational references. Progression questions preferred progression records, ship and droid questions their codices, faction-slot questions the guild rules, and story questions the narrative records. Rosters stayed valid for membership questions, but they could never override a direct policy source.

Its verifier rejected an answer when:

• a policy answer relied only on a roster or tracking row;
• a direct rule source had been retrieved but a weaker source won;
• the answer said the archive had nothing on a question that the retrieved text did answer;
• raw Sheet labels such as Column A or Row leaked into the reply;
• stale conversation context changed the topic;
• the cited source didn’t support the claim.

Members could react to a reply with a no-vote, for a wrong answer, or an abstain, for a partly right or uncertain one. A script turned those reactions into a review queue, which I used to tell source-selection errors from source-reading errors, vague answers, ambiguous questions, voice problems and genuine gaps in the archive.

The retrieval design

Discord question
  -> intent and topic routing
  -> archive retrieval
  -> source-authority ranking
  -> answer draft and verification
  -> Discord answer with source titles
  -> no-vote or abstain review queue

The Discord bot, written in Node, posted each question to a small HTTP service on the same guest. Behind that sat a cache and then the agent itself, in lore_agent.py, which worked in three stages:

1. Retrieve. It gathered candidates three ways: semantic search over the archive’s text chunks, structured records built from the Sheets, and calls to a small tool server the agent calls for retrieval, including literal term sweeps across the corpus.
2. Sift. A language model judged the candidate pool and kept the relevant ones.
3. Answer. A final language-model call wrote the reply from the evidence that survived, and the verifier checked it.

Ollama ran on the guest, on the loopback interface only, to make the embeddings for semantic search, with the qwen3-embedding:4b model. The vectors lived in a Chroma index on the guest. The language model that judged and drafted answers ran as a separate inference service and isn’t covered here.

A systemd user timer kept the corpus current. Every six minutes it ran the Drive sync through the Google Workspace command-line client, rebuilt the local corpus and source map when files changed, and restarted the search service only when the corpus version changed. A Drive change monitor compared the active manifest with the live archive, and a status file recorded each sync’s result.

The May 2026 fixes and audits

Most of the bot’s accuracy work happened between 2026-05-11 and 2026-05-13, each round recorded as a dated report.

Date	Report	What changed
2026-05-11	Retrieval, cache and routing fixes	Bounded the sift fallback, gated the answer cache, made sync failures visible and restored Drive access
2026-05-12	Accuracy policy	Policy and eligibility questions now prefer guides, codices and progression records over rosters
2026-05-12	Corpus authority audit	Compared all 45 archive files with the bot’s manifest and added a source-authority map, policy cards and aliases
2026-05-12	Deep corpus coverage audit	Processed the whole corpus, mapped 138 topics and 34 overlap risks, and rebuilt the source map
2026-05-13	Evaluation and verification upgrade	Added the final verifier, structured Sheet records, a 214-case evaluation suite, the feedback queue and Drive monitoring

Fixes, 2026-05-11

Members had flagged 18 bad answers. Reading them against 409 recent requests and the service logs turned up four root causes:

1. The Drive sync had stopped silently. Its OAuth token had expired or been revoked, so the index froze, and only the service status showed it.
2. The sift step failed open. Whenever the judging call errored or was skipped, the code passed all 22 candidates, at up to 1,800 characters each, straight to the answer step. That overflowed the model’s context, which caused more errors on the next request, and the answer cited junk. I bounded the fallback to 16 candidates of 1,200 characters.
3. The cache was poisoned. Only the lowest-confidence answers were kept out of it, so a wrong answer marked high or medium confidence was served again for the full cache lifetime, in about 3 ms. I added a gate on what the cache would accept.
4. Sync failures were invisible. The wrapper script swallowed the sync’s error output; it now records a sync status.

Later rounds in the same report merged candidates by a quota per source, so one common word couldn’t fill every slot from half the corpus, ranked candidates by the kind of match, routed questions about leadership to the organizational references, and rebuilt the source map from the archive’s own contents. The leadership routing was needed because semantic search alone ranked unrelated documents above the canonical office record whenever their wording overlapped.

Accuracy policy, 2026-05-12

The answers were in the archive; the bot was choosing the wrong kind of source for new phrasings. The clearest case was a faction’s joining requirement: the faction guide stated the minimum rank, but a broad search for the faction’s name pulled in its roster rows, and the fallback ranking could put those above the guide.

I added detection for policy and requirement questions, so rank, eligibility, ownership, permission and progression questions are handled differently from roster, profile or lore-recap questions, and gave guides, codices, faction rules and progression records authority over rosters, registries and tracking sheets for them. I normalized the faction’s terms, quietened the automatic term sweep for broad one-word terms, added a targeted retry when the first pass found no authoritative evidence, and blocked roster-only policy answers, raw table dumps and “not found” answers when direct evidence was present. Six new regression tests passed, and I bumped the answer cache’s key so old wrong answers couldn’t survive.

One problem stayed open: a correct answer about current office holders still took about 55 seconds.

Corpus authority audit, 2026-05-12

The bot’s manifest held all 45 active files, 3,666 chunks and 5,061 records from the archive folder, so missing coverage wasn’t the issue; source selection was. From the archive I generated a corpus inventory, a source-authority map, policy cards, an entity-alias map and a set of evaluation questions, and loaded them into the bot. The agent now routed by question class across policy, rosters and offices, story, profiles, ownership, progression, abilities, combat, engineering, factions and casual messages, and retried with targeted calls when the first pass missed an authoritative source. Two of the codices refused local text export, so for those two I used the copies the bot had already synced.

Deep corpus coverage audit, 2026-05-12

A second pass processed every file in the manifest: 1,397,014 characters of exported text, 1,780 Sheet rows and 7,370 non-empty cells. It mapped 138 topics, found 34 places where sources overlapped closely enough to confuse retrieval, and generated 97 evaluation questions. The rebuilt source map, 171,233 bytes, recorded for each source the topics it is the primary authority for, the topics it can support, and the topics it should stay out of, with its important sections and a content hash. The overlaps clustered in abilities, the asset registry, beast policy and rosters, character progression, combat rules, crafting, crystals, current offices and droid policy.

Evaluation and verification, 2026-05-13

The last round added the final answer verifier, a check for a weaker source competing with a better one, stricter policy handling, guards against “not found” answers and leaked table fragments, and tighter use of conversation context so a fresh question didn’t inherit an old topic. Sheet rows now carried their headers, row fields, sheet title, row number and primary label, so the bot read a row as a record instead of pasting table internals into Discord.

The new evaluation suite had 214 cases covering all 45 files, checking required facts, required and forbidden sources, forbidden phrases, leaked Sheet fields and the verifier’s final state. The feedback queue started with 19 items, 16 no-votes and 3 abstains. All 26 top-priority cases passed against the live answer endpoint, after the run exposed and I fixed five more routing gaps.

Known limits

The product overview listed its own limits:

• The archive’s wording and Drive’s text extraction set the ceiling on answer quality.
• Conflicting records needed authority ranking, and might still need a decision from the community’s leadership.
• A passing evaluation covered the recorded cases, not every future phrasing.
• When the bot and the archive disagreed, the archive was right.

How it was shelved

By 2026-07-25 the guest was already stopped. I disabled its autostart, so a reboot of grey-server couldn’t bring an archived guest back, moved the 50-file project tree and its walkthrough into Archive/, removed the guest from the Ansible inventory and the SSH identity targets, and scheduled the deletion for 2026-08-15.

I deleted it early, on 2026-08-09. The destroy command was guarded on three conditions: the guest had to be stopped, autostart had to be off, and its root volume line had to match exactly. Proxmox removed the container and its 100 GiB volume, ssd-lvm1 allocation fell from 15.72 to 13.05 percent, and the cluster stayed quorate on five votes. A sweep on 2026-08-20 removed the old guest’s task logs from Grey while keeping those of the current VM 105, ubuntu-dev, which reuses the ID.

The archive keeps four generations of source, primary, remote, experimental and legacy, along with the tests, configuration snapshots, the product overview and the dated reports.