Designing Multimodal Prompts
A multimodal prompt combines instructions with supported images, audio, video or other inputs. Choose the modality that carries the evidence; a screenshot can show an alignment defect that a vague text description misses.
#Give the model a specific task
For a synthetic invoice image, ask for supplier, date and total, with a source region and null for unreadable values. For video, ask for relevant timestamps. For a UI screenshot, ask for visible layout problems; supply the actual CSS separately if code diagnosis is required.
textRead the attached synthetic invoice. Return supplier, invoice_date, total and currency. For each field, include the visible text supporting it. Use null when the field is absent or unreadable. Do not infer missing digits.
#Separate observation from inference
First describe the visible evidence, then perform the requested calculation or comparison. For a chart, identify axis labels and units before discussing a trend. Cropping a region can help with small text, but does not guarantee accurate OCR.
Do not ask the model to identify a person from a photograph. Provide an authorized textual identity when a task legitimately requires it. Remove secrets such as access keys before sharing a screenshot.
#Check input limits and output quality
Supported file formats, video length and resolution depend on the selected model and endpoint. Images and video also consume context and can increase processing cost. Check the current documentation before building upload controls.
Review extracted amounts against the source and test blurred, rotated and incomplete inputs. An attractive structured response is not evidence that every value was read correctly.
