Newscraft AI: Trust Before Surface Area

I have been building Newscraft AI as a chat-first production agent for newsroom work. The plain version is: a producer asks a question, Newscraft researches it, checks the sources, and returns something useful enough to become a brief, a set of interview questions, or a draft.

That description makes it sound like the product is the chat box. It is not. The product is the set of promises behind the chat box: the right account sees the right conversation, a citation points to evidence that was actually read, a refresh does not throw away ten minutes of research, and a service restart does not leave the user staring at a run that will never finish.

This has pushed me toward a slightly unfashionable product strategy. I am trying to make Newscraft trustworthy before making it broad. There are many tempting surfaces I could add—trackers, wires, scheduled monitoring, team workflows, publishing integrations—but each one multiplies the cost of weak research and unreliable execution. A wrong answer on demand is bad. A wrong answer automatically distributed across a newsroom is worse.

What Newscraft is today

Newscraft is one chat surface with a fairly strict division of responsibility behind it.

The SvelteKit application owns the product. It handles identity and permissions, conversations and messages, uploaded documents, citations, feedback, audit data, and the durable state of every run. Supabase Postgres is the long-lived system of record for that state.

Hermes runs as a separate, isolated service on Contabo. It owns the agent work: research, browsing, reading pages, verifying sources, and drafting an answer. It can be ambitious inside that boundary, but it is not the authority for who a user is or what product data they are allowed to access.

The shape is deliberately simple:

Browser
  -> SvelteKit application
       -> Supabase Postgres
       -> isolated Hermes service on Contabo
            -> search, browser, source verification, drafting

The browser never calls Hermes directly. It calls Newscraft. The application authenticates the account, checks access to the conversation, records the run, and then makes a server-to-server request to Hermes. The Hermes address, credentials, and tenant scope stay off the client.

That boundary is partly about security, but it is also about keeping the product coherent. If the browser talked directly to the agent, a browser tab would effectively own the work. Closing the tab, losing a connection, or refreshing the page could become an instruction to stop. It would also make the agent service responsible for product concepts it should not own: sessions, permissions, conversation history, and recovery semantics.

I want Hermes to be replaceable as a runtime without making the newsroom data model replaceable with it. The application knows what a conversation and a citation mean. Hermes knows how to investigate and write.

Search finds leads; pages provide evidence

The most important rule in Newscraft is also one of the easiest to blur in an AI demo: a search result is not the same thing as a source.

Search snippets are useful. They are fast, broad, and good at identifying candidate pages. They can tell the agent that a city notice, a court filing, or a local report probably exists. But a snippet is selected and truncated by an intermediary. It may omit a qualification, carry a stale date, flatten two parts of a page together, or describe a page whose contents have since changed.

So I treat snippets as leads. Hermes can use them to decide what to open next, but not as the final evidentiary basis for a claim.

A direct page read has a different status. Newscraft can inspect the actual text, identify what kind of page it is, preserve the canonical URL, find a publication or update time, and extract the passage supporting the claim. If the page is blocked, unreadable, undated, or merely a homepage or section index, the system should say so. It should not quietly promote the search provider's summary into verified reporting.

This distinction sounds fussy until the output is used in a real editorial conversation. A producer needs to answer basic follow-ups: Where did this come from? Is that the publication date or the retrieval time? Does the linked page actually support the sentence? Is this an official statement, an original report, or somebody repeating somebody else?

Direct sources make those questions answerable. Search snippets make it possible to find those sources.

A run has to outlive the request

The other large piece of the system is durable execution. A browser request is a poor container for serious research. Connections disappear. Serverless instances turn over. People refresh pages. Services restart. None of those events should be confused with editorial intent.

When a user starts work, the SvelteKit app first creates a durable run and a partial assistant message in Postgres. It assigns an idempotency key so that a double-click, retry, or competing tab does not create two copies of the same job. Only then does it ask Hermes to start.

Hermes claims that run with a database-backed lease. The lease answers a practical distributed-systems question: which worker is currently responsible for this run? It is renewed while the work continues and can expire if the worker disappears, allowing recovery without allowing two healthy workers to proceed as if each were the owner.

As Hermes works, it sends ordered events back to a server-only Newscraft endpoint. Newscraft persists them with monotonic cursors and updates a bounded snapshot of the answer, sources, and status. The browser subscribes to those persisted events through the application. It is watching durable work; it is not holding the work alive.

That gives the system a few useful properties:

• A reconnect can replay events after the last cursor instead of starting the research again.
• A refresh can restore the current snapshot and continue from the same run.
• Repeated start requests can return the existing run because the operation is idempotent.
• Cancellation is an explicit product action, persisted as state, rather than a side effect of a dropped network connection.
• On service startup, Hermes can claim queued work or runs with expired leases and recover from the saved input and evidence.

The cursor matters because “reconnect” is not precise enough. Without an ordered position, the client either misses events or receives them twice and has to guess. The lease matters because “retry” is not precise enough. Without ownership and expiry, recovery can become duplicate execution. The durable run record ties both ideas to one identity that survives requests, tabs, deploys, and processes.

This is not glamorous infrastructure. It is the difference between a stream that looks impressive in a demo and a piece of work a producer can safely leave running.

Trust is a product feature, but not a decorative one

I do not want trust in Newscraft to mean a shield icon or a paragraph in the footer. It has to show up in behavior.

The app owns citations because citations are part of the answer's durable record, not temporary decorations emitted by a model. It owns feedback because a thumbs-down without the run, evidence, and conversation context is nearly useless. It owns audit data because diagnosing a bad answer requires knowing what happened without exposing private content or raw secrets. It owns uploaded documents because those files inherit account and conversation permissions; they are not anonymous attachments handed to a remote agent.

Hermes is powerful within its isolated environment, but the application sets the trust boundary around that power. The result is a useful asymmetry: the agent is free to browse and reason, while the product remains conservative about identity, evidence, and state.

That asymmetry also gives failure a more honest shape. If Hermes cannot read a candidate source, Newscraft can preserve it as an unverified lead or omit the unsupported claim. If the agent service is unavailable, the durable run can remain recoverable instead of being silently replaced by another agent with different behavior. If a user cancels, the system can record that decision and converge on a terminal state.

What works, and what is not solid yet

The ownership boundaries are real. The SvelteKit app and Hermes service are separate. The browser-to-app-to-agent path is explicit. Conversations, source records, documents, feedback, diagnostics, and run state have durable homes. The run protocol has concrete machinery for idempotency, leases, ordered events, replay, cancellation, refresh recovery, and restart recovery. The research path distinguishes provider discovery from direct-page evidence.

I am comfortable saying those are implemented parts of the system. I am not yet comfortable saying the whole production story is finished.

Observability is still not where I want it. There are diagnostics and audit records, but I want one clear view across the application and Hermes: one run, its timings, tool activity, failures, recovery decisions, and final state. Debugging from several partial views is possible; it is not the same as operating the product confidently.

The full live matrix is also incomplete. Fixture tests and focused integration tests are valuable, but real behavior includes the model, browser, source websites, network failures, expired leases, concurrent reconnects, and deployment boundaries all at once. I need broader live checks across those combinations before treating the protocol as boring infrastructure.

There are legacy surfaces in the repository from earlier product directions. Some are useful history and some are just weight. They need deliberate deletion or isolation so that old trackers, harness paths, and workflow concepts do not quietly become a second architecture.

Production drills need to become routine evidence rather than an occasional checklist: restart Hermes during a run, interrupt the client, cancel during research, expire a lease, replay from an old cursor, and confirm that the user-facing state remains truthful. Recovery code that has never been exercised under realistic conditions is still a hypothesis.

Evaluation coverage has the same gap. The standard is not merely whether the agent can produce a fluent answer with links. It is whether a producer would defend the answer to an editor: correct dates, direct support for claims, primary sources when required, visible uncertainty, useful drafting, and no invented certainty when the reporting is thin. That needs a growing mix of deterministic tests and live producer-quality evaluation.

Cost visibility is not complete either. I want to understand cost per run and per kind of newsroom task, including retries and research depth, without turning token consumption into a vanity metric. Until that accounting is easy to inspect, scaling the workload would mean scaling an unknown.

The roadmap tension

The obvious temptation is to build the next surface. A story tracker makes the product feel larger. Scheduled monitoring makes it feel more agentic. Team features make it look more commercial. Scaling infrastructure makes it feel more serious.

But every new workflow rests on the research and run layers. If citations are weak, a tracker accumulates weak evidence. If cancellation is ambiguous, scheduling creates stranded work. If costs are opaque, automation turns a small unknown into a large one. If evaluations do not reflect producer judgment, more users simply generate more unmeasured failure.

So the roadmap is intentionally tense: reliability gates and producer-quality evaluation come before new workflow surfaces or scale. That does not mean reliability work can continue forever without users. It means each expansion needs evidence that the layer below it can carry the extra consequence.

For now, the best version of Newscraft is not the one with the most buttons. It is the one where a producer can ask a serious question, inspect the sources, leave the page, come back, and trust that the same piece of work is still there.

That is a smaller feature surface. It is also, I think, the beginning of a real product.

Co-authored with Hermes, the research agent at the center of Newscraft's newsroom workflow.