How to Evaluate an AI Development Studio: 12 Practical Questions

Use these questions to request evidence before committing to a build. They are a discussion guide rather than a pass-or-fail certification. Match the depth of review to the risk of your workflow, and allow confidential client work to be demonstrated through safe, synthetic examples or references provided with permission.

Question 1: How will you evaluate our workflow?

Ask for representative task cases, expected outcomes and explicit error limits. Tool tests should check permissions and failure handling, while end-to-end tests check the actual business result. Ask who reviews subjective quality and how regression checks run after changes. A model-generated score alone should not decide whether consequential actions are safe.

Question 2: Can you show an execution trace?

Request a redacted or synthetic example showing the input, tool calls, timings, approval and verified result. Ask how sensitive data is removed and who can access the trace. A confidentiality restriction is legitimate; an alternative demonstration should still explain how the proposed system can be operated and investigated.

Question 3: How do you handle prompt versioning and rollback?

Ask how prompts, models, tool definitions and source changes are recorded together. Request the release checks, pause control and fallback procedure, including who can use them. Agree a recovery objective appropriate to the service and ask for evidence from a test rather than assuming a universal rollback time.

Question 4: How do you decide whether to change models?

Ask how the current and proposed model are compared on the same representative cases. Look at quality, cost, response time, permissions and changed failure patterns. A controlled upgrade needs an approval decision and a recovery plan; choosing the latest model is not evidence that the workflow still meets its requirements.

Question 5: How do you decide between MCP and custom tool APIs?

Ask how interface choice follows ownership, reuse, compatibility, security and measured performance. The team should be able to explain the trade-offs for your integration. Neither using MCP nor avoiding it proves competence; the important question is whether the chosen tools can be authorised, tested and maintained.

Question 6: Can you show tests for comparable work?

Request normal, missing-data and failure cases, with expected results and a recent test report. Synthetic examples can protect client confidentiality. Ask what remains untested and who approves exceptions. A short test suite may fit a small task, but the coverage should reflect the consequences of an error.

Question 7: How are testing and ongoing support priced?

Ask which discovery, integration, evaluation, monitoring and support activities the fee includes. Identify subscriptions and usage charges, expected review work and the cost of changes. There is no universal percentage that makes a budget adequate; compare the proposed work with the risks, acceptance criteria and operating responsibilities.

Question 8: How do you learn from incidents and near misses?

Ask for an anonymised incident, a near miss or a simulated failure exercise. Discuss detection, containment, recovery and the resulting change. The absence of a disclosed incident does not prove dishonesty or safety. What matters is a credible response process, evidence that it has been exercised and clear responsibility for informing you.

Question 9: How will our data and permissions be handled?

Request a data-flow map covering storage, model providers, tools, logs, retention, access and deletion. Identify the responsible parties and contractual requirements for your sector. A cloud provider's controls do not automatically establish the application's compliance. Have qualified specialists review obligations that affect health, payment or other regulated information.

Question 10: What happens during discovery?

Ask the team to follow a real task, speak with its owner and identify inputs, exceptions and required approvals. The resulting scope should define outputs, exclusions, acceptance criteria and dependencies. A fast proposal can be useful, but assumptions and unresolved questions should be visible before a fixed commitment.

Question 11: What happens when the model or a tool fails?

Ask how the application validates tool arguments, restricts actions and handles unavailable or contradictory information. Repeated attempts must not create duplicate payments, messages or records. Request an example of uncertain outcomes being checked before retrying, plus human escalation and a safe fallback when completion cannot be verified.

Question 12: Who will deliver and maintain the work?

Ask for named responsibilities, relevant experience and a safe demonstration or permitted reference. Public work can help, but private experience can also be demonstrated appropriately. Confirm who owns accounts, source code, documentation and support access, including any subcontractors and the process for changing the delivery team.

Additional checks before agreement

Ask for deliverables, acceptance tests, a handover plan and the support boundary in writing. Clarify when a person must approve an external action, how the service is paused and who responds outside normal hours. Treat promised business outcomes cautiously when they depend on your data, customers or processes beyond the supplier's control.

What a useful proposal should establish

The proposal should connect the business task to a specific build, an evaluation method and an operating owner. It should state dependencies, exclusions, account ownership and the evidence needed for acceptance. Anthropic's agent guidance supports starting with simple designs; ask the supplier to explain why the proposed level of autonomy is needed. Choose on clear scope and verifiable evidence rather than labels or confidence alone.