127 problems found in a Charlotte agency's AI employees, and the plan that fixes them
AI agent audit and orchestrator rebuild for a marketing agency
A Charlotte marketing agency ran AI employees inside Slack, built by another developer. They answered, drafted, and ran morning briefings, but nobody could say who was allowed to ask them for what, or what they cost. We audited the system, found 127 problems, and are rebuilding it around one orchestrator with approvals by role.
Industry
Marketing agency, Charlotte
Client
Name withheld
Engagement
Agent audit, then a phased rebuild of an existing platform
Status
Audit done October 2026. Rebuild in progress
Tech Stack
Next.js, TypeScript, Supabase (Postgres and pgvector), Trigger.dev, Slack, Nango, Claude via OpenRouter and Anthropic
What did the agency have before we came in?
About ten AI employees, each with its own Slack bot. Staff mentioned them or sent a DM, the way they would message a colleague.
Behind them sat an admin portal, scheduled jobs for end-of-day collection, morning briefings and nightly checks, and connections to Google Workspace, ad accounts, the CRM, and the project boards. Every outside action was meant to wait for a person. The base was sound. The problems were in how the parts connected.
How do you audit an AI agent system?
Two ways at once. We read the code, the database rules, and the hosting setup. Then we used the live system in Slack as different kinds of users and tried to make it do things it should not.
Every finding got a plain description, a proposed fix, a severity, and an estimate. That turned a vague feeling that the AI was unreliable into a list the agency could plan against.
What did the audit cover?
127 findings in five areas.
01.
Access and setup: 29 findings
Who owns the code, the hosting, the keys, and the admin logins. Who can sign in, and who can approve.
02.
Slack, routing, and approvals: 41 findings
How a message reaches an agent, what the agent may do, and what waits for a person.
03.
Clients and memory: 22 findings
Where knowledge lives, how it is found, and how one client's facts are kept from another.
04.
Proactive alerts: 10 findings
What the system notices on its own, and whether that ever reaches the right person.
05.
Cost and scheduling: 25 findings
What every run costs, which jobs run when, and what happens when one fails.
What did the audit find about approvals and access?
Anyone could make themselves an approver. A test account with an outside email, never confirmed, approved an email to a client, and it was sent. Staff could approve their own requests. A plain “remember this” became a rule for every agent: a fake “50% off for every client” note was saved, and the sales agent repeated it.
The agents shared by room, not by person. A member got a client’s retainer in a DM, and when asked in a public channel, the agent posted it there too. Anyone in Slack could ask the AI to read login codes from the shared mailbox. And code the AI wrote from Slack could merge into the live app with nobody reviewing it.
Why did the agents take the wrong action?
Actions ran before anyone could stop them. In testing, a calendar invite became a Slack message, and a thank-you email became a proposal PDF. Approval cards then asked people to review actions that had already run.
Chat history was kept per person per channel, not per thread, so topics leaked between threads. A tag anywhere but the start of a message could go unanswered. After a few actions in a DM, the agent stopped acting and posted an empty “please review”.
Where was the AI money going?
Every call carried the full instructions and every tool: about 28,000 tokens. One SEO skill added about 6,500 tokens whenever the word “seo” appeared. One long thread used about 14 million tokens. An extra paid AI call ran before every reply just to pick the model, adding up to 8 seconds.
Dashboard tiles were 68% of the AI bill, and two of them used AI just to count. A task set to run every 2 minutes was accepted, about 720 AI runs a day. And three screens showed three different monthly totals, so nobody knew the real cost.
Why did the alerts never reach anyone?
The system spotted the right things, but little reached a person. The morning briefing had been off for days. The best reports ran in dry-run mode. 66 suggestions sat waiting for approval in threads nobody watched, most over a week old.
Real alerts were buried under repeats and developer jargon. The overdue-task alert flagged finished tasks and tagged people Slack could not find.
What is the new design?
Slack message, then a master orchestrator, then memory if needed, then a department agent, then the skills and tools that agent picks, then approval if the action needs it, then the reply.
The orchestrator checks who is asking, which client the channel belongs to, and what that person may do. The channel identifies the client. It no longer decides which agent answers. Each agent loads only the skills it needs, from short descriptions, instead of every instruction on every call.
What does the rebuild plan cover?
Four phases, in this order, because each one depends on the one before it.
We kept the agents, skills, and tool connections that worked. Nothing was rebuilt for the sake of it.
01.
Orchestrator, permissions, and approvals
Every Slack message goes to one master orchestrator. It knows who is asking, which client the channel belongs to, and what that person may do. Then it picks the agent.
02.
Structured memory
One memory service. Operating docs and brand files in Google Drive, feedback and lessons in Obsidian, both searchable through RAG and tagged by client.
03.
Proactive work
Daytime checks instead of overnight only. Each alert goes to the person who can act on it, by DM, in plain language. Suggested memory changes wait for review.
04.
Ledger, cost, and scheduling
One record per request, one cost source by person, team, client, and agent, minimum intervals and daily caps on scheduled tasks, and alerts when jobs fail.
How will the agency know it works?
Tests for roles, approvals, and full Slack conversations are added as each part is built, so a fix cannot quietly break later. Every request becomes one record: who asked, which agent and tools ran, what it cost, and who approved it.
The agency gets a weekly progress update. This page will be updated when the rebuild is done.
Frequently asked questions
Common questions about AI employees and agent audits.