Duplicate detection and fraud gatekeeping: what to stop before distribution

A bad lead that reaches a buyer costs you three times: the payout you clawed back, the time your team spent arguing about it, and the buyer's confidence in your inventory. That last one is the expensive one, and it doesn't show up in any report.

Gatekeeping is the work of catching those leads before they leave the building. It's unglamorous, it's mostly configuration, and it's the highest-leverage thing in an operation that's growing faster than its quality controls.

This is how to think about the gate: what to check, in what order, and — the part most operators get wrong — what to not reject.

Pipeline order is the whole design

Before any specific check, understand the sequence. Gatekeeping isn't a switch, it's a pipeline, and the order of stages determines what each stage can even see.

A well-built ingestion pipeline runs roughly this order:

Authenticate and resolve — verify the source's credentials, resolve which campaign this lead belongs to.
Source caps — is this source over its volume allowance?
Create the record — the lead now exists, in a pending state, so everything after this is auditable.
Validate the payload — required fields present, types correct, values in range.
Blacklist / DNC scrub — the hard gate. Cheapest and most legally consequential.
Credit and eligibility checks — where applicable to the vertical.
Lead filters — your own business rules.
Duplicate check — have we seen this person?
IP and device fraud — proxy, VPN, bot, datacenter.
Phone and email validation — is this contact reachable and real?
Buyer selection and distribution — only now does anyone outside your system see the lead.

Two things fall out of that ordering, and both matter more than they look.

Everything through step 10 is pre-distribution. A duplicate never reaches a buyer. A blacklisted number never reaches a buyer. If your gate is configured correctly, the failure modes your buyers complain about are ones you chose to allow.

A check can only see what ran before it. This is the ordering trap, and it generates more support tickets than any other gatekeeping issue. If your filters run before your fraud vendors, your filters cannot reference fraud scores — the fields don't exist yet. Operators write a filter on a fraud score, watch it never fire, and conclude the integration is broken. It isn't. The filter is just standing upstream of the data it wants.

When a filter that references enriched data won't fire, the first question is never "is the vendor working." It's "does this field exist yet at the point this filter runs."

Duplicate detection: three decisions

Duplicate checking looks like one setting. It's three, and they're independent.

Decision 1: Scope

What do you compare against?

Campaign-wide — has anyone in this campaign seen this person? Prevents ever selling the same person twice, to anyone.
Buyer-level — has this specific buyer seen this person? The same lead can go to a different buyer without issue.

This is a business decision disguised as a config field. Campaign-wide is conservative: you protect the buyer relationship at the cost of volume you could have sold. Buyer-level is permissive: the same consumer legitimately shops multiple carriers, and selling that person to a second buyer who's never seen them isn't fraud, it's the market working.

Which is right depends on what you told your buyers. If you sold exclusivity, campaign-wide is the only honest setting. If your buyers know they're getting shared inventory, buyer-level is leaving less money on the table.

Note what's not on the list: cross-campaign deduplication. Duplicate scope is bounded by the campaign. If you're running the same vertical across two campaigns and expecting dedupe between them, you won't get it, and that's a structural fact to design around rather than configure away.

Decision 2: Matching strategy

All configured fields must match — strict. Same phone and same email and same IP. Very few false positives, more misses.
Any single field matches — loose. Same phone is enough. Catches far more, at the cost of collisions.

The loose setting has a real failure mode people underestimate: shared IP addresses. Put IP in a one-field-match dupe check and you will reject the second person who submits from a household, an office, a university, or a mobile carrier's NAT pool. Those are real leads. You just deleted them and logged it as a duplicate.

IP is a fraud signal, not an identity signal. It belongs in your fraud checks, not usually in your dupe key.

Decision 3: Lookback window

How far back do you search? Thirty days is the common default and a reasonable starting point for most verticals.

The window is a direct expression of how long you think a consumer's intent persists. Someone who filled out an auto insurance form 25 days ago and fills one out today is arguably the same lead. At 90 days, they're arguably shopping again, and refusing to sell that is refusing revenue. At 7 days, you're selling the same person to the same buyer four times a month and they will notice.

Set it per vertical, not globally. Intent decays at different rates for a mortgage refi and a roof replacement.

Duplicates are two different problems

Here's a distinction that changes how you act on the data.

Pre-distribution duplicates are ones you caught. Your dupe check fired, the lead was rejected, nobody outside your system ever saw it. These are working as designed. They cost you nothing except the source payout you may or may not owe.

Buyer-reported duplicates are ones you missed. The lead went out, the buyer's own CRM said "we already have this person," and they rejected it or returned it. These are the ones that matter, because each one is a gap between your dupe key and theirs.

The second number is the one to watch. A rising buyer-reported dupe rate means your matching is looser than that buyer's — usually because they're matching on something you aren't, or their lookback is longer than yours. That's a fixable configuration difference, and it's worth asking the buyer directly what they dedupe on rather than guessing at it for a quarter.

If you're only looking at your own dupe rejection count, you're measuring the leads you caught and ignoring the ones that cost you money.

Fraud gatekeeping: gates versus enrichment

Not every check should reject. This is the distinction that separates a gate that helps from a gate that quietly destroys inventory.

Gates reject the lead outright. Use these where a hit is unambiguous and the consequence of being wrong is severe:

Blacklist and DNC scrubs. Binary and legally consequential. If the number is on a suppression list, there is no version of this where you sell it. Run it early — it's cheap, and it's the check with actual liability attached.
Litigator lists. Same logic, more so.

Scores are inputs to a rule, not verdicts. IP fraud scores, proxy and VPN flags, bot detection, disposable email detection — these produce a number or a signal, and the rejection threshold is yours to set:

A fraud score above some threshold, reject.
VPN and proxy together, reject; either alone, maybe not.
A datacenter IP on a form fill at 3am, probably reject. The same signal on a legitimate corporate VPN, less obvious.

Every one of those thresholds is a dial between fraud you eat and good leads you delete. There's no correct setting, only a setting you've measured.

Enrichment doesn't reject anything. Some checks exist purely to add data — a credit band, a demographic signal — that your filters and your buyers can then act on. The same vendor is often both: with rules configured it's a gate, with no rules it's enrichment. Know which mode each of yours is in, because an "integration that isn't working" is very often an enrichment integration doing exactly what it was configured to do, which is nothing.

Filters: order matters, first match wins

Your own business rules run as an ordered list. The mechanics that matter:

Filters evaluate in order, and the first match stops evaluation. Order is not cosmetic. It's the logic.
Within one filter, conditions are AND. All conditions must match for the filter to fire. Multiple conditions in one filter is a narrower rule, not a broader one.
Nothing matching means the lead passes. The default is accept. If you intended a whitelist — only these states, only this age band — you must write the explicit reject rule at the bottom yourself. A list of accept rules with no closing reject is a list of accept rules that lets everything else through.

That last one is the single most common filter bug in lead operations. It fails open, silently, and looks fine on the screen.

What to reject versus what to sell cheaper

The instinct is to make gatekeeping binary: good lead, bad lead. But a large share of your marginal inventory isn't bad, it's worth less, and rejecting it is a decision to earn zero on it.

A lead from a state where your best buyer has no coverage isn't fraudulent. A lead with a thin credit profile isn't fake. A lead from a source with a mediocre history is still a lead. All of these are candidates for conditional pricing — accept the lead, attach a lower payout, route it to a buyer who wants that segment at that price.

The framing that works:

Reject what's illegal to contact, provably fake, or genuinely duplicate. There's no price at which these are worth selling.
Price down what's real but marginal. Somebody will buy it. Your job is to find out at what number, not to decide in advance that the answer is zero.
Enrich and pass what you're unsure about. Attach the signal, let the filters and the buyers decide.

Operators who tighten the gate in response to a buyer complaint usually over-tighten, because rejection is visible and lost revenue isn't. The leads you rejected don't appear in any report as a cost. They just quietly aren't there.

Common mistakes

Mistake 1: Ordering filters before enrichment

Covered above, but it earns its place on the list because it's so common. A filter can't see data that hasn't been fetched yet.

The fix: know your pipeline order. When a filter on enriched data won't fire, check the ordering before you check the vendor.

Mistake 2: Putting IP in the duplicate key

Shared IPs are normal. Households, offices, mobile carriers. A one-field-match dupe check on IP deletes real people.

The fix: dedupe on identity (phone, email). Use IP for fraud.

Mistake 3: A whitelist with no closing reject

Accept rules with nothing at the bottom means everything else passes. You built a whitelist and shipped a passthrough.

The fix: every whitelist ends with an explicit catch-all reject. Test it with a lead you expect to be rejected.

Mistake 4: Tuning the gate on anecdotes

One buyer complains about one lead, and somebody tightens a threshold that afternoon. Nobody measures what that threshold now rejects.

The fix: change thresholds against measured rejection volume, not against the most recent complaint. Know what a change costs before you make it.

Mistake 5: Measuring rejections instead of misses

Your dupe rejection count measures your successes. Your buyer-reported dupe rate measures your failures. Most operators only watch the first.

The fix: track buyer-reported dupes and returns as your gate's real scorecard. That's the number that costs money.

Mistake 6: Treating gatekeeping as set-and-forget

Fraud patterns move. Sources degrade. The thresholds that were right last year are calibrated to traffic you no longer have.

The fix: review rejection reasons monthly. The distribution shifting is your early warning that something upstream changed.

The takeaway

Gatekeeping is a pipeline, not a switch. Order determines what each check can see, and most "broken integration" reports are really ordering problems.

Reject only what has no price — illegal to contact, provably fake, genuinely duplicate. Price down everything that's merely marginal, because a lead you rejected earns exactly zero and doesn't appear on any report as a loss.

And measure the gate by what gets past it, not by what it catches. Your buyers are already measuring it that way.