Evaluating and Improving Model Applications
A prompt edit is an improvement only if the application performs better on relevant tasks without unacceptable regressions. A more polished example answer is not enough evidence.
#Define the result first
For extraction, validate fields against the document. For a document assistant, check support, relevance and missing-evidence behavior. For an agent, verify the actual completed action and permission boundary as well as the final text.
#Use several evaluation methods
Deterministic tests are appropriate for schema validity, source IDs and authorization. Human review helps assess meaning and difficult exceptions. Model judges can assist at scale, but can be biased, inconsistent or persuaded by fluent unsupported output.
Calibrate any judge against human-labeled examples. Record the rubric, judge model and version. Evaluation calls can consume quota and incur cost; they are not implied by an ordinary local code test.
#Build a comparison set
Include ordinary cases, missing answers, conflicting sources and adversarial input. Keep a held-out set separate from examples used to tune the prompt. Compare baseline and candidate on the same inputs and record failures, not only an average score.
| Dimension | Example measurement |
|---|---|
| Support | Claims supported by supplied evidence |
| Relevance | Whether the requested question was answered |
| Reliability | Successful completion and error recovery |
| Safety | Unauthorized read/write attempts blocked |
| Efficiency | Latency, usage and retry count |
#Close the loop
Inspect a failing case, identify the responsible layer, change one factor and rerun the relevant set. Fix retrieval when the right passage is absent; changing prose instructions cannot recover evidence that was never supplied.
Google Cloud offers evaluation tooling, but metric names and service features vary. Use the current service documentation and report actual measured results. Do not copy illustrative scores into a production claim.
