Compare approaches on the same sample
Try different models, instructions or media engines before choosing settings for a larger run.
Choose the comparison flow
Settings that work on one document can fail on the next, and a cheaper model is sometimes as good as an expensive one. Comparing is how you find out before you run an action on hundreds of rows. Run two or more versions on the same small sample, look at where they disagree, and keep the one that's right most often.
Frisket has three comparison tools, for different kinds of work:
- Compare, on the Analyze tab, tries different models or instructions for an action on rows of a sheet, such as extracting fields from filings.
- OCR Compare, on the Media tab, tries OCR engines or settings on the same pages of a PDF or image.
- Transcribe Compare, on the Media tab, tries transcription engines on the same passage of a recording.
Keep the source and scope the same
A comparison only tells you something if the versions get the same input:
- Use the same rows, pages or passage for every version.
- Change one thing at a time. If you change the model and the instructions together, you won't know which made the difference.
- Pick a sample with your hard cases in it: faint scans, long names, missing values. Easy rows make every version look good.
When the versions disagree, check the source to see which is right. Look at a few rows where they agree, too. If you have an answer key, use it: it scores every row for you.
Hosted models charge per use, and every version you add runs on every row of the sample. Keep the sample small.