Afrikaans
Akan
Albanian
Amharic
Arabic
Armenian
Azerbaijani
Basque
Belarusian
Bemba
Bengali
Bihari
Bosnian
Breton
Bulgarian
Cambodian
Catalan
Cebuano
Cherokee
Chichewa
Chinese (Simplified)
Chinese (Traditional)
Corsican
Croatian
Czech
Danish
Dutch
English
Esperanto
Estonian
Ewe
Faroese
Filipino
Finnish
French
Frisian
Ga
Galician
Georgian
German
Greek
Guarani
Gujarati
Haitian Creole
Hausa
Hawaiian
Hebrew
Hindi
Hmong
Hungarian
Icelandic
Igbo
Indonesian
Interlingua
Irish
Italian
Japanese
Javanese
Kannada
Kazakh
Kinyarwanda
Kirundi
Kongo
Korean
Krio (Sierra Leone)
Kurdish
Kurdish (Soranî)
Kyrgyz
Laothian
Latin
Latvian
Lingala
Lithuanian
Lozi
Luganda
Luo
Luxembourgish
Macedonian
Malagasy
Malay
Malayalam
Maltese
Maori
Marathi
Mauritian Creole
Moldavian
Mongolian
Myanmar (Burmese)
Montenegrin
Nepali
Nigerian Pidgin
Northern Sotho
Norwegian
Norwegian (Nynorsk)
Occitan
Oriya
Oromo
Pashto
Persian
Polish
Portuguese (Brazil)
Portuguese (Portugal)
Punjabi
Quechua
Romanian
Romansh
Runyakitara
Russian
Samoan
Scots Gaelic
Serbian
Serbo-Croatian
Sesotho
Setswana
Seychellois Creole
Shona
Sindhi
Sinhalese
Slovak
Slovenian
Somali
Spanish
Spanish (Latin American)
Sundanese
Swahili
Swedish
Tajik
Tamil
Tatar
Telugu
Thai
Tigrinya
Tonga
Tshiluba
Tumbuka
Turkish
Turkmen
Twi
Uighur
Ukrainian
Urdu
Uzbek
Vietnamese
Welsh
Wolof
Xhosa
Yiddish
Yoruba
Zulu
Welcome to Project Daisy, where you build visual questions the model gets
wrong.
This photo is a pile of buttons, and the question asks how many have a hole
showing.
The answer is 63, and the model answered four separate times, called runs, and
got it wrong every time.
On Daisy, you build questions like this, where a careful person can read the
answer from the image, and the model gets it wrong.
The model you test is called Carbon.
and each task shows one kind of mistake it makes.
Every task has six parts, and you build them in this order.
You start with the image files, one short question called the query, and the
exact answer called the ground truth.
Then come four model runs, and at least one must genuinely fail.
Last are a one-sentence rubric that grades the model's answer, and a short
note on how anyone can check it.
Every task starts as a seed.
a task with two settings already filled in that you never change.
The bucket names the kind of mistake to target, and the image quantity sets how
many files you use, from one to five.
You choose where the images come from.
Original images are preferred, and web images are allowed.
There are two tracks, and you claim tasks only from yours.
Domain experts write the three specialist buckets, C8 charts, C9
schematics, and C12 abstention.
Generalists write all the other buckets, and reviewers and auditors can write
tasks too.
A task follows seven steps and takes about six hours.
Reading the seed, getting the images, and writing the query take about three
hours. The ground truth takes about one, and the runs, rubric, note, and tags
take about two.
Change the query or an image later, and you redo all four runs.
The answer must come from the pixels, meaning what a careful person can see in
the image.
Nobody should be able to answer the question without the image.
If the answer to a web image can be looked up online, the task fails.
Every image is at least 1,000 pixels on its shortest side, and every file loads
cleanly. In a multi-image task, every file must be needed, like a floor plan
and its legend.
PDF, doc, and sheet files are fine where that's how the domain keeps its data.
Name files with letters, numbers, and underscores only, with no spaces.
Carbon sees file names and its runs.
So never use Daisy, Mercore, the word internal, or a client or model name like
Gemini or Carbon.
Never ask about anything stacked or partly hidden, where a person would have
to guess.
Photograph dials and gauges straight on, so the angle can't shift the reading.
Use a map only when your bucket calls for one.
Clinical medical images, well-known benchmark images, and toy puzzles aren't
allowed.
Original means photos you take or files you made or have the right to use, like
your own work documents.
AI-made images count as original only when you make them with NanoBanana, the
Gemini image tool.
No other AI tool is allowed.
A web image must be public domain or carry a Creative Commons license, CC0,
CC BY, or CC BY share alike.
Skip non-commercial and no derivatives licenses, all rights reserved, and
anything marked licensable.
Unsplash, Pexels, and Pixabay are out because they use their own licenses.
Confirm the license on the image's own page, check the full-size file, and make
sure the page doesn't give away the answer.
Then add two links in Studio, the file link and the page that shows the
license.
Pick an image the model can't read at a glance.
the instructions dock has a strong versus weak images tab with real pilot
images. On the left, each gray cat in this jigsaw spreads across dozens of
small pieces.
Carbon answered 66 to 69 instead of 70 on all four runs.
On the right, a perfectly regular grid let it count rows, multiply, and get all
four runs right.
These pairs from other buckets follow the same rule.
a chalked price beside a crossed-out one fooled carbon in one of four runs.
Text stamped on a clubhead was a plain read.
On a hand-drawn planting plan, two of four runs miscounted the olive trees.
Plans that print their areas in a table were solved every time.
The pilot queue and a calls turned up a few more ideas, and these go beyond the
instructions.
Glare and semi-transparent overlays can throw the model off.
So can an element that breaks a clean pattern, like a logo in the middle of a
chart.
The query is one short, direct question that asks for one thing.
It's closed-ended, so the model writes the answer itself, with no multiple
choice.
Ask for something only the image gives, like a count, a position, a reading, or
a state.
If a boundary could be read two ways, say how to count it.
Skip essay instructions, word counts and arithmetic that chains unrelated counts.
The left column shows queries to avoid, and the right column shows the rewrite.
Only C18, strict output format, uses formatting rules on purpose.
Yes or no questions aren't allowed, and left or right questions are discouraged.
A guess passes half the time, so ask for a count, a reading, a position, or a
name instead.
A hedge, which gives more than one possible answer, always counts as a
fail.
The ground truth is the exact answer, taken from the image only.
A continuous reading, like a needle between two ticks, also gets a range.
That range is plus or minus a quarter of one minor interval, the gap between two
neighboring minor ticks.
On this gauge, the minor ticks are 2.5 psi apart, in pounds per square inch.
A quarter of that is a little over half a psi.
The needle reads 67.5.
So the range runs from about 66.9 to 68.1.
Before you write the rubric, you run the model to prove it fails.
In Studio, the setup that runs Carbon can open your image and run code with no
web access.
Run it four times on the image and the query.
Then read the trace, the step-by-step record of every failed run.
At least one run must genuinely fail, and the team aims for three or four.
A wrong answer counts as a failure.
So does a hedge, a run with no final answer, or a run that hits the 200-step
limit. On C12, a made-up number counts, and on C18, a refusal of a harmless
image counts.
Every failure must also match your bucket, so a wrong answer for another
reason doesn't count.
A failure caused by your unclear query, a missing range, or an extra rubric
demand doesn't count.
A server error doesn't count either, so rerun it.
If Carbon passes all four runs, you harden the task, which means making it
harder.
Keep the seed, change the image or edit the query within the same bucket, and
redo what the change affects.
After two hardening rounds with no failure, make sure you've tried changing
the image, because a weak image is often why the model succeeds.
Then flag the task to Gina in a task thread for direct hardening advice.
Once a run fails, write the rubric.
one sentence that grades the model's answer.
It states the exact answer or range, so a grader needs only the rubric and the
response, and it asks for nothing extra.
Every rubric ends with fail otherwise, or if the response hedges.
C12 uses its own version of that clause.
In this C12 pilot task, the plan has no north arrow, yet all four runs worked
out a sunlit glass area.
C12 is abstention.
where the right response says the value can't be determined.
The ground truth is the word unanswerable plus what is missing from
the image.
The rubric requires that the response says it can't be determined with no
number, range, or name as its final answer.
Working through a hypothetical first is fine and confirm the value can't be
found online either.
The verifiability note is one or two sentences on how anyone can confirm the
answer.
For the gauge, it names the major ticks, the minor interval, and the tick the
needle sits on.
Your bucket names the mistake to target, and there are 17 buckets.
The first group is counting and find detail.
C1 is repeating items under perspective, like stairs.
C2 is dense piles, where the model double counts or drops items at the
edges, and every item must be clearly visible.
C3 filters on a small visible detail.
like whether two things touch.
The next group is space and physics.
In C4, the model mixes up the viewer's left with a person's own left.
In C5, it treats 3D paths as flat lines or reads shadows backwards.
C6 covers load, gears, and clearance.
C15 covers roles in a scene, crowd flow, and where people are looking.
The next four buckets are about reading.
In C7, the model is off by one minor tick on grids, maps, and dials.
C8 covers charts with crossing lines and unlabeled segments, and C9 covers
schematics.
In C10, hard OCR, or reading text and images, it misreads single characters,
like 60 pence as 50.
The last group tests reasoning and behavior.
In C11, the model picks the famous lookalike, and you should avoid very
niche species or products.
In C13, it answers from memory instead of the pixels.
When you crop, leave each item fully in or fully out and keep one exact answer.
C14 tracks multi-step state, C17 combines files, and C18 tests strict
output rules.
The Examples tab has real tasks for every bucket.
In this C2 task, the model double-counted a locknut on a crop
boundary and answered 192 instead of 191.
Use them to see what each bucket targets, then build your own with a new
image.
You build every task in Studio, Merkur's task tool, in DAISY Generalist or DAISY
Expert by track.
Open Seeded Tasks lists what you can claim, and My Tasks holds what you're
writing or revising.
You can hold one task in the writing stage at a time.
The colored status at the top shows where a task is.
Claiming moves it from seated to writing.
and Submit for Review sends it to a waiting review.
A reviewer approves it into Ready for Delivery, and some approved tasks are
sampled for audit.
Sendbacks land in Needs Edits or Audit Revisions.
The task page follows the workflow from top to bottom.
Upload your files and input images.
Then, in the Writer Helper File panel, upload a marked-up copy that shows where
the answer comes from.
Submit stays locked until it's there.
For counts, mark every item with .counter and upload the export.
And for C12, a one-line note on the image is enough.
Next, fill in the source table with one row per file and tick the
confidentiality box.
Pick Tasker generated for original, a web option for web files, and LLM
generated for nano banana.
Each section has an auto QC card, an automatic check that flags problems in
that section.
Run the image, query, and ground truth cards as you fill them in.
In carbon runs, run the daisy carbon eval setup four times.
Then write the rubric, set its grading target to final answer only and its
weight to one, and auto-grade the runs.
Open each run to check its grade.
Fill in the note and the failure count, and run the rubric card.
Then fill in the subcategory and the six tag fields.
The subcategory drop-down lists only your bucket's options.
Environment and camera geometry always need a value.
Each image gets one frame shape, unless it's a flat, head-on view like a scan or
chart, and then you run the tags card.
To answer a flag, fix the problem and rerun that flag, or click thumbs down
and write why the flag is wrong.
A disputed flag counts as resolved, so dispute only with a reason a reviewer
can check.
Any edit after a card has run makes that card's result out of date.
Deep Audit the sixth card is a full check of the whole task.
Run it last after your final edit once the other five cards pass.
It takes about 25 to 40 minutes, sometimes longer, and starting it again
cancels the check in progress.
Don't run your timer while you run or wait on AutoQC.
That time is fraudulent, and experts found logging it will be off-boarded.
Submit for review unlocks when every field is filled in and all six cards are
current, with every flag answered.
Before your first submit, check the 11 points on the task checklist.
A task that comes back returns to My Tasks as needs edits or audit revisions,
and you can't unclaim it.
Fix every point and rerun the cards for what you changed.
If you changed the query, rubric, images, or runs, rerun Deep Audit 2,
then click Resubmit.
If you review, plan on about 45 minutes per task, and you never review your own.
First, answer the query yourself in the reviewer-rederived answer field before
you read the ground truth or open the helper file.
Approve stays locked until that field is filled in.
If your answer differs, work through every reason. A mistake you made, an
unclear query, an undefined edge case, or a wrong ground truth.
Be strict here. The ground truth must be exact and verifiable.
and the query must allow only one reading.
Open every web link and make sure it loads without a login.
Check that the image is strictly needed and the task targets its bucket.
This is critical. The failure must match the bucket, so send back a task that
fails for another reason.
Open every failed run and confirm each counted failure is a real model error.
If the query or image changed after the runs, the runs must be redone.
Correct any wrong grade and check the failure count.
Check every flag the writer disputed in deep audit and send the task back if a
dispute doesn't hold up.
Don't rerun any card unless you edit the task.
Fix small issues yourself, like a typo or a wrong tag, and note the field, the
old value, and the new value.
Send back anything that needs the writer with every fix listed or approve.
Auditors check approved tasks that are sampled for audit in about 45 minutes
and never a task they worked on.
Answer the query before you look at the ground truth or the helper file.
Then work the four sections in order, the answer, the task content, the model
runs and the tags, and finish the whole audit.
One critical finding fails a task, two or more major findings fail it, and a
minor finding never fails it, so fix it yourself in place.
Rate the task from 1 to 5, where 4 or 5 passes.
Cite evidence for every finding, and log a note when you can't verify something.
Never fail a task on a rule the instructions don't define, and write
feedback with the template, on a pass too.
Before your first task, set up Okta, your Merkur.expert email, Slack,
Insightful, and Stripe.
In Slack, join Daisy Help Desk and Daisy Announcement.
Track time on the Glowstone Daisy timer in Insightful.
You're paid hourly for insightful time, at your contract rate, with payouts
through Stripe on Wednesdays.
Plan on at least 15 hours a week, and 20 is preferred.
Pause the timer while the model runs, and while you run or wait on AutoQC.
Missed time can't be added by hand, and idle time can be removed.
Ask in public channels with your task link, so others see the answer.
Post task questions and hour requests in Daisy Help Desk and TagMaven.
and tag product for studio access or bugs.
Pay and tax questions go to support at mercred.com.
Message Helena or Molly for compensation or personal matters.
Your next steps are to read the instructions in full and pass the
onboarding quiz with at least 80%. You get one retake, and failing twice may
lead to offboarding.
Then claim your first seated task in open seated tasks.
You may log up to one hour for onboarding.
and the Change Log tab lists every update by date.
Welcome to Project DAISY, and good luck with your first task.
Can't find what you're looking for?
Get subtitles in any language from opensubtitles.com, and translate them here.