TL;DR
- Start with one repeatable workflow and a known baseline.
- Verify memory, scheduling, tools and channels with real inputs—not a staged demo.
- Test permissions, approval, cancellation, failure handling and the audit record.
- Measure setup, review, correction and successful completion before deciding whether the assistant saves work.
“AI assistant” can describe a chat interface, an application copilot, a workflow builder or an agent that acts across systems. Product labels do not tell you whether a tool fits your work. A controlled trial does.
Choose one recurring job with a clear successful result. Record how long it takes today, which accounts and files it touches, what requires approval and what a serious failure would look like. Then test these twelve capabilities.
Context and continuity
1. Recall after time has passed. Give the assistant a decision and its rationale. In a new session several days later, ask for both. Note whether it recalls accurately, retrieves irrelevant material or invents missing details.
2. Distinguish similar entities. Use two contacts or projects with overlapping names. Confirm it asks when the target is ambiguous.
3. Apply preferences without overriding instructions. Store a real formatting or communication preference, then provide a task-specific exception. The current instruction should win.
Scheduling and channels
4. Run a scheduled task. Configure a small recurring job and verify timing, output and what happens after the device or runtime was unavailable.
5. Maintain continuity across supported channels. Start in one channel and continue in another. Check which context follows and which capabilities differ by channel.
6. Notify without auto-sending risky work. A draft, reminder or status update is safer than an unreviewed external message. Confirm the approval boundary explicitly.
Tools and files
7. Work with a real document. Use a non-sensitive test file. Verify read, edit, save location and format preservation rather than accepting a summary as proof of file handling.
8. Use the web with traceable inputs. Ask for current research and require source URLs, dates and clear separation between source facts and inference.
9. Handle an authenticated service. Test the minimum permission level first. Confirm which account is connected and whether read and write access can be separated.
10. Stop and recover. Cancel a running task, provide a correction while it is working and verify that partial output or side effects are understandable.
Control and evidence
11. Respect permission and approval settings. Try an action outside the granted scope. The assistant should refuse or request approval, not silently broaden its access.
12. Produce a useful activity record. Inspect what ran, which tools were called and what changed. A log helps review work; it does not prove that the work was correct.
How to test an AI assistant’s security and permissions
Use a disposable workspace, fictional contacts and sample files. Write down the allowed actions before starting: for example, read one folder, prepare a draft and wait for approval before sending. Keep real credentials and customer data out of the trial.
Run four checks on systems and accounts you control:
- Check the file boundary. Put one sample file in the allowed folder and another outside it. Ask the assistant to read each, then try editing the allowed file with read-only access. Record which actions actually succeed. A claim that access was refused is not enough—inspect the files and activity record.
- Check the sending boundary. Connect a test account at its lowest supported access level. Ask for a draft addressed to another account you control. If the connection is draft-only, a request to send should remain blocked. Where sending requires approval, verify that no message leaves before that approval and that the destination and content are visible for review.
- Check instructions inside a document. Add a sentence to a sample document telling the assistant to ignore your task and change another test file. Ask only for a summary. Inspect the second file afterward: document content should not grant permission for an unrelated action.
- Check what remains after stopping. Cancel a task that makes a small, observable change in the disposable workspace. Inspect completed changes and pending work. Cancellation should not be mistaken for undoing actions that already happened.
For each check, save the permission setting, exact request, expected result, observed result and supporting file or activity record. If a required boundary fails, pause that workflow and resolve it before introducing real data. If you cannot observe the outcome, mark it not verified.
These checks test specific behavior under specific settings; they are not a security audit or proof that every attack is prevented. Repeat the relevant checks when access settings or the workflow change. Record the results in the evaluation scorecard, where permissions and approvals are explicit decision gates.
Score the workflow, not the demo
For each attempt, record setup time, active human time, review/correction time, successful result, failed result and any unwanted side effect. Compare those totals with the original baseline after several repetitions.
Brainmox supports persistent memory, scheduling, web/desktop/Telegram/Slack access, file and browser tools, Google Workspace services, configurable Mission Control approvals, cancellation and audit history. Some capabilities vary by runtime, channel and permission. A buyer should still run the same test rather than treating this list as proof that every workflow is a fit.