Why we read every sold comp twice

Short answer: an eBay "sold" search finds sales that share your item's words, not sales of your item. A search for a commercial ice machine came back full of $23–$53 thermostat kits. Every one of them really sold, and none of them was an ice machine. So before a sold comp is allowed to price a GovDeals lot, we now check it one more time: is this actually the same product? Across 1,228 lots and 83,293 sold comps, the answer was "no" 40% of the time.

We've written before about why real sold data beats an AI's guess. This post is about the uncomfortable follow-up. Real sold data is only better if the sales are of the right thing, and keyword search can't tell the difference between the thing and everything sold to go with it.

Real sales of the wrong thing

Here is what the comp pile looked like for some ordinary government-surplus items. Every row is real: the comps the search returned, and what the price looked like before and after the wrong ones were removed.

Notice the direction. Most of the time the wrong comps are cheaper than the real thing: a part sells for a fraction of the machine it fits. That's the expensive failure for a reseller, because it hides the deal. An ice machine "worth $36" is a hard pass on any bid, and that is exactly the lot you want to be bidding on. But it runs the other way too. The XCover Pro was being priced partly off two newer, pricier models, which would have made a mediocre lot look like a win.

Why our filters didn't catch these

We already filter comps hard before they price anything. Lots and bundles go, because we price one unit and multiply. Anything that reads like a part or accessory goes ("charger for…", "replacement…", "fits…"). Comps missing the model number go. Salvage "for parts or not working" sales go. Those rules remove a lot of junk.

But rules match patterns, and the comps above beat them. "Hoshizaki Ice Machine Bin Control" contains every word in "Hoshizaki ice machine with bin". It doesn't say "for" or "replacement". By every keyword test, it's a perfect match. A person sees in half a second that it's a $40 switch. A regex never will.

A second opinion on every comp

So after the rules run, each surviving comp now goes to Jev, a product-matching classifier from TypeSafe AI. It isn't a chatbot: it takes a short piece of text plus one typed question and returns calibrated answers with probabilities. We ask it one question per comp. Here is the GovDeals listing, and here is an eBay sale: is that sale a fair per-unit comparison for what's being auctioned? It must pick one of five answers:

• Same product: a fair comp, keep it.
• Different model: same kind of item, but it conflicts with a model, size or generation the listing names.
• Accessory or part: something for the product, not the product.
• Multi-unit lot: the sale itself was several units.
• Unrelated: a golf bag, a fridge magnet.

A comp stays only if Jev rates it at least 50% likely to be the same product. If fewer than three comps survive, the lot gets no eBay-sold price at all, and it falls back to our other sources or to an honest "unknown". It never falls back to the comps we just rejected.

One design choice matters more than it looks: Jev never sees the auction's price or the comp's price. The question is about identity only. A reseller's whole edge is a real comp that's surprisingly cheap or an auction that's surprisingly low. If the checker could see prices, it might learn that "cheap looks wrong" and throw out exactly the deals we exist to find.

We got the question wrong first

The first version of the question rejected 84% of comps. That sounds like a great filter until you read what it threw out.

GovDeals titles are vague and often plural: "dell latitude computers", "groen tilting skillet gas". Our first wording asked whether a comp matched the listing's brand, model and variant as one complete unit. Jev took that literally. A single Dell Latitude laptop is not "computers", so it was rejected as a multi-unit mismatch. A genuine Groen gas tilting skillet was rejected as a "different model", because the listing never named a model to match. Had we switched it on, one laptop lot's comp price would have collapsed from $219 to $25. A filter meant to remove wrong comps would have buried real deals.

We never let that version touch a price. Every change we make to pricing runs first in shadow mode: it calculates what it would do and records it beside the live price, and we read the results by hand before anything changes. The second version of the question was rewritten around how auction listings are actually written:

• Judge the eBay sale, not the auction listing. The auction being a lot never makes a comp a "multi-unit lot".
• One unit is the right comparison for a lot of several, because we multiply by quantity afterwards.
• Call something a different model only when it conflicts with a detail the auction listing actually names.

What the second version found

That shape is what you'd hope for from a correctness fix. Most lots already had clean comps and didn't move. The ones that moved were mostly the ice-machine kind: whole products priced off their own spare parts. When we read the rejected comps of the biggest movers by hand, the rejections were overwhelmingly right.

"Overwhelmingly" is not "always". In one spot check Jev rejected a "Herman Miller Mirra 2 Stool" as a different model against a listing for a Mirra work stool. A person would probably keep that comp. No checker is perfect, which is why the rules stay in front of it, the "unknown" verdict stays behind it, and we keep reading samples by hand.

After those numbers and that read-through, we switched it on for live pricing this week.

The other half of the problem: the search itself

Vetting fixes comps that are wrong. While measuring it, we found a different problem: lots with no comps at all. Through mid-October, only about one sold search in four came back with anything usable. We assumed eBay was blocking our data provider. It wasn't. Their logs showed 99.8% of our requests succeeding. The searches were succeeding and finding nothing, for two reasons that were entirely our own doing:

• We were demanding every word. We searched with the whole listing title in "match every word" mode. Titles like "of lens 1 2 fa 35mm f1 2 manual lock screws" match nothing on earth. Switching to eBay's "fewer words" matching, with our own filters doing the strict part, took usable results from 23–31% of searches to 41–55%.
• Sellers' warehouse codes were being treated as model numbers. Government sellers tag titles with inventory codes: "Conference Phones (227804 BT)", "Cisco Phones (ID# 47779) (J11-2R) (26-2472-94) 662". Our relevance filter treated every number-like token as a model number a real comp must contain, and no eBay listing has ever contained "47779". Those searches often did return dozens of real sales, about 85 on average, and we then threw every one away. We now strip inventory codes, shipping notes and "MSRP $…" before searching.

The obvious next step is to let the classifier rescue those misses: when our word filters reject everything, ask it which of the returned sales are actually the product. That is built and running in shadow mode right now. It goes live only after we've read enough of what it keeps. The risk there is the opposite one, keeping a wrong product rather than rejecting a right one, so the samples we check are the comps it keeps.

Doing it 20,000 times a day, carefully

One unglamorous lesson to finish. Our first full run with the checker asked 20,000 questions and hit our own safety cap partway through. Each eBay search can return up to 200 sales, and we were asking about all of them, every time we refreshed a lot. Most of a lot's sales are the same from one week to the next, so most of those questions had already been answered. The checker now handles each lot's 40 most recent sales and remembers its verdicts, so a weekly refresh only asks about sales it hasn't seen. Comps past that 40 are simply left out of the price. They are never used without being checked.

None of this is a breakthrough. It's what "use real sold data" turns out to mean once you do it thousands of times a day: read every comp, measure each change before trusting it, and admit "unknown" when the real comps run out.

What this means if you flip

• Read the comps, not just the number. When you search sold listings yourself, scroll the actual titles. If the cheapest results are kits, pads, switches or manuals, your "market price" is a parts price.
• Suspiciously low is often a matching problem, not a bad item. A commercial appliance "worth $36" deserves a second look before you scroll past it.
• Watch for newer models in your comps. They inflate value just as quietly as parts deflate it.
• Strip the seller's noise before you search. Inventory tags, bin codes and "shipping available" narrow your search to nothing.

Related: Why AI alone can't price your flips · What sold comps are · Receipts →