Building an AI teammate that can actually act on your data

Most 'AI assistants' shipped by B2B SaaS in the last two years are, charitably, read-only chatbots: you ask a question, the model paraphrases your wiki, you copy the answer somewhere useful. They generate text. They do not do work.

When we set out to build Psyth's AI teammate, we wanted something else. We wanted an agent that could read your work, decide what to do, and then actually do it — assign an issue, update an OKR, request leave on someone's behalf, generate and post a standup. We wanted it to be safe enough to ship to enterprise customers and useful enough that you'd notice if it disappeared.

This post is about how we built it: the architecture choices, the safety layers, and the things that turned out harder than we expected.

Three properties we wouldn't compromise on

• Bounded by RBAC. The AI cannot do anything the calling user cannot do. No magic admin escape hatches.
• Auditable. Every tool call is logged with the user, the input, the result, and the model's reasoning trace.
• Interruptible. The user can stop the agent mid-action and roll back side effects when reasonable.

The architecture

Psyth's agent is built on a tool-calling LLM with a typed, RBAC-aware tool layer. The model never touches the database directly. Instead, it sees a curated set of tools — query_issues, assign_issue, update_okr_progress, get_workload, request_leave — each defined as a typed function with a JSON schema and an authorization predicate.

When the model emits a tool call, we authorize it against the calling user's permissions before executing. If the call is allowed, we execute and return the result. If not, we return a structured error and let the model decide what to do next (typically: ask the user, or back off).

The tool layer is small on purpose. We resisted the temptation to expose 'a tool for everything' — that path leads to an agent that hallucinates capabilities and misuses them. The current surface is roughly 40 tools covering the read and write paths most users actually invoke. Adding a tool requires a design review and a test suite.

Why the typed tool layer matters

Two things break in unsafe agent designs. First, the model invokes capabilities it shouldn't have — usually because the prompt soft-suggested 'do whatever it takes'. Second, the model passes badly-shaped data to those capabilities, and the system either errors loudly or, worse, accepts garbage.

Strong typing on the tool layer fixes both. The model gets a precise schema (with descriptions, enum values, examples). When it generates malformed calls, our adapter rejects them before they reach the database. When it tries to invoke something out of scope, the auth predicate refuses. The model treats those refusals as constraints and routes around them — exactly as we want.

What the safety layer actually looks like

• Per-call RBAC check before execution.
• Per-tool risk classification (read / mutate / external-side-effect).
• Confirmation prompts surfaced to the user for high-risk actions (mass assignment, destructive operations).
• Per-organization rate limits on tool calls to prevent runaway loops.
• An emergency stop visible in the UI that aborts in-flight tool calls.
• Full audit log retained for 12 months on Pro / 36 months on Enterprise.

Things that turned out harder than we expected

Confirmation discipline. Early prototypes were too eager — the model would happily mass-assign 80 issues without asking. We had to teach it to confirm bulk and destructive actions, and to verbally summarize what it was about to do before doing it.

Hallucinated entities. Models will sometimes reference an employee or project that doesn't exist. We added a 'resolve before act' step that explicitly looks up referenced entities first; if anything is unresolved, the model has to clarify.

Token economics on context. Loading 'the entire workspace' into context is wasteful and slow. We narrow context aggressively — pulling only the projects, people, and issues relevant to the current conversation, on demand.

Streaming UX during long actions. Users get nervous when a destructive action takes 4 seconds in silence. We stream intermediate progress with explicit 'about to do X / X done' markers.

How we evaluate it

We run the agent against a fixed evaluation set of ~400 realistic scenarios on every model upgrade. Each scenario has a known-good outcome, a list of forbidden actions, and a list of 'must confirm' actions. We track three numbers: outcome accuracy, RBAC violations (target: zero), and confirmation discipline (target: zero unconfirmed destructive actions).

The most important number is the second one. RBAC violations would be a catastrophic regression — they would mean an agent did something a user couldn't authorize. We've shipped to production with zero observed violations to date.

What's next

We're working on three things. First, multi-step planning: the agent currently executes one or two tool calls per turn, but complex requests benefit from planning a sequence and explaining it before executing. Second, explanation depth: showing the user not just what was done but why the model chose to do it. Third, opt-in proactive actions — letting the agent surface 'I noticed X, do you want me to do Y?' before being asked.

If you want to try Psyth's AI teammate on a real workspace, the Free plan includes 50 AI messages per month — enough to feel the difference between an AI that talks and an AI that acts.