About the SVG Junk Drawer

What are SVG images?

An SVG is a drawing written in code: a list of shapes, coordinates, and colors.

Unlike most AI-generated art, SVGs are vector images. This means they can be edited in programs like Adobe Illustrator or scaled to any size without losing sharpness.

They are also a fun way to test a model’s coding skills, because mistakes in the code show up in the drawing.

Each drawing is graded

Every drawing has been evaluated using a taxonomy I designed.

For the latest round, I wrote 128 prompts, and sent them to four leading language models. I evaluated each drawing that came back, both on its own and compared to the other three.

What happens to a prompt

The Junk Drawer can also generate new images using prompts from visitors like you.

After the visitor submits a prompt, a server-side pipeline sends it to four models via API:

• Claude Opus 5.5
• GPT-6.1 Sol
• Kimi K3
• Gemini 3.1 Pro

Every model gets identical system instructions: draw only what the prompt asks for on a transparent background.

Filed by the intake clerk

At the same time, another subagent, which I call the intake clerk, catalogues the prompt. It gives it a title and applies three types of tags:

• Subject: what the prompt asks to have drawn, like an object, a creature or a plant.
• Treatment: how the prompt asks the drawing to look, like cartoon, realistic or a named style.
• Probe: what the prompt tests in the drawing model, like structure, layering or texture.

The drawings are anonymized for grading

Once all four models have finished, the user sees the drawings together, without knowing which model made which.

Then the user grades each drawing on its own.

The data collection instrument

Each drawing is graded one question at a time in five categories, then an overall grade using this instrument.

Feel free to click through it to try it out (or keep scrolling!)

For demonstration purposes only. Nothing entered here is saved or recorded.

The taxonomy

Drawings are first graded in these five categories.

After that, users leave an Overall Grade on a five-point scale (the same one the USDA uses for beef!)

The gold standard

This succulent drawn by Claude Fable shows you what a Prime grade SVG looks like.

The parts fit together, the layers stack correctly, and the pot has the tasteful Scandinavian style the prompt asked for.

It earned top marks in every category.

Bad Example: Understanding Assignment

This prompt asked for the American flag as it existed in 1819, drawn in magic marker. However, this drawing does not have the correct arrangement of stars for any version of the flag that existed in 1819.

The stars are otherwise well drawn — it’s just the wrong flag. This makes it a textbook example of an Understanding Assignment problem.

Bad Example: Structural Coherence

In this drawing, the model seems to have understood the assignment and attempted to draw an asteroid hitting the earth as instructed. However, the model positioned the asteroid off to the side of the earth, instead of having it strike the planet.

This is an issue with Structural Coherence, because the parts of the image are not fitting together like they should.

Bad Example: Layering

This drawing has a major Layering problem. Notice how the leaves are hidden behind the pot, when they should be sprouting from its mouth.

Press the ▶ button under the drawing to see what I mean. The replay shows the model drawing the leaves just fine, but then covering them with the pot’s mouth. Textbook layering issue.

Bad Example: Paintwork

The Paintwork category is used to identify issues related to color, shading, lighting effects, strokes, and texture.

In this example, Gemini 3.1 Pro generated a ghost in an opaque shade of dark navy blue. This is an issue with Paintwork, because a drawing of a ghost ought to be partially transparent or a lighter color.

This is a loss pattern I noticed several times in Gemini’s drawings in particular — that model has a tendency to darken colors to excess or add shadows where they don’t belong.

Je ne sais quoi

How to define Je ne sais quoi? It’s style. It’s taste. It’s that intangible something that makes a drawing greater than the sum of its parts. An image can be fine in every other category and still not have it.

Take the booger on the left, drawn by Kimi K3, for example. It doesn’t have any major problems. It’s…fine.

But fine is all it is. And compared to Claude’s slime-green masterpiece on the right? Kimi’s is a dud.

That’s another loss pattern I noticed: Kimi’s drawings tend to be plain and dull even when they are free from technical defects.

Preference ranking

After evaluating all four drawings individually, visitors then rank order all four from best to worst.

To do this, the user drags each drawing from the tray onto a podium, placing it into position in the rankings.

Strength of preference

Then, for each pair of neighbors in the ranking, the visitor adjusts the height of the pedestals to record how much better the higher-ranking drawing is.

Based on the height of the pedestal, the higher-ranking drawing is recorded as being much better, better, slightly better, or negligibly better (about the same).

Derived comparison scores

The process of adjusting pedestal height allows us to derive side-by-side comparison scores for each pair of drawings.

By adding up the margins between any two, we end up with six pairwise comparison scores on a seven-point Likert scale.

Where it all lands

After the visitor finishes, everything they submit ends up in a SQL database: the prompt, each drawing, the tokens and cost of each call, and every rating.

In other words, the Junk Drawer is a fully functioning data labeling project, front-to-back. The only thing it’s missing is actual users (other than myself!)

The results of my 128-prompt set

Using this system, I sent 128 prompts to the four models and graded all four drawings that came back.

This chart shows the distribution of the head-to-head comparison scores on the seven-point Likert scale.

Two pairs, pretty much tied

The results split the four models into two pairs. Claude Opus 5.5 and GPT-6.1 Sol beat the other two most of the time, but are about even against each other. Gemini 3.1 Pro and Kimi K3 are nearly tied too.

Leagues apart

Here’s another way to look at it. Using these scores, we can estimate each model’s overall strength by calculating its chance of beating the average model.

Here too, Claude Opus 5.5 and GPT-6.1 Sol are nearly tied with each other, while Gemini 3.1 Pro and Kimi K3 are in the minor leagues.

Where they go wrong

These charts show each model’s issue rate, or the share of drawings that had a problem in a given category as a percentage.

For three of the categories, I decided to only count moderate and major problems, because if you look closely enough, you can find a minor issue in almost any drawing.

For four out of five categories, a high issue rate is a bad thing. The exception is Je ne sais quoi, because it measures a positive attribute. In that category, the green bars show how often a model’s drawings had it, so the higher the percentage, the better.

Gemini loss pattern

Remember Gemini’s dark navy blue ghost? That darkening wasn’t a one-off — drawings generated by Gemini had problems with Paintwork 38% of the time.

That’s more than any of the other models by 10 percentage points.

Kimi K3 loss pattern

The data also reveals that Kimi K3 suffers from a statistically significant lack of Je ne sais quoi.

Where to look next

The intake clerk tags every prompt with “probe” tags that describe the sorts of capabilities a prompt tests. These tags could also be used to identify loss patterns, because a model might struggle to fulfill certain types of prompts more than others.

Unfortunately, a sample of only 128 prompts means there isn’t enough data in many categories to draw any meaningful conclusions. Nevertheless, this heatmap shows how tags like these could provide valuable insights if applied to a larger sample.

Thanks for digging through the drawer

Of course, this is not meant to be a scientific study.

I built it as a portfolio piece to show the kind of work I do in data operations, including developing taxonomies, designing data collection instruments, working with SQL, and telling stories with data.

If you’d like to talk more about evaluation and the art of data collection, email me at tysonwelsh@gmail.com or find me on LinkedIn.