Why your keyword alert is noise: a taxonomy of false positives

Every keyword monitoring tool follows the same arc. Week one it is thrilling. Week three you skim it. Week six it is a folder rule. Nothing broke: the tool is doing exactly what it did on day one, which is return everything containing your string.

The problem is that "contains the string" and "worth your attention" overlap far less than anyone expects, and they overlap in a way that gets worse over time as your keywords broaden to catch things you were missing.

What follows is the taxonomy: ten distinct reasons a candidate should be thrown away. They are worth separating because they have different costs, different fixes, and, critically, different implications when a feed goes quiet.

Cheap first: the ordering is the design

The single most important property of a filter chain is that it is ordered by cost. Classification by a language model is the expensive step. Everything above it exists to make sure the expensive step runs on as few candidates as possible.

The order below is roughly the order these run. The first six cost effectively nothing, the seventh costs a little, and the last three are the ones you are paying for.

1. Wrong shape of page

The retrieved URL is not a post at all. It is a profile page, a documentation index, a tag listing, a category page, a store listing.

This is the most common single class of garbage from any search-backed system and the cheapest to remove, because you can tell from the URL and the page type without reading anything. A tool that surfaces "r/somesubreddit, 45k members" as a signal is not filtering on shape.

Fix: structural, per source. Nothing the user should ever have to configure.

2. Matched an exclusion

The candidate hit a term you specifically said you did not want. Your own brand name, a homonym of your product, a market you do not serve, a recruiter's vocabulary.

The homonym case is the one that quietly destroys precision. If your product is called Atlas, or Pulse, or Arc, a meaningful fraction of every result is about something else entirely, and no amount of model quality fixes it: the model is correctly identifying a relevant conversation about a different Atlas.

Fix: an exclusion list, which is the highest-leverage thing most users never touch. Ten minutes here beats any other single change.

3. No lexical overlap

The document does not contain any of the vocabulary the listener is built around. Not a keyword, not a synonym, not an adjacent phrase.

This catches the failure mode where a search engine returns something loosely associated by its own semantic model, often genuinely interesting, and about something else. It is a cheap gate and it is deliberately blunt, because the expensive judgement happens later and should not be spent here.

Fix: none needed. This one working is the system working.

4. Blocked domain

Your own site, your own blog, your own community, or a domain on a block list.

Sounds trivial, and is the single most embarrassing false positive in the category: a tool that reports your own marketing page as a buying signal. It happens more than vendors admit, particularly when a company's content ranks well for its own category terms.

Fix: automatic for your own domains, manual for the rest.

5. Outside the window

The post is real, relevant, and from nineteen months ago.

Intent decays, and it decays at different rates for different evidence. A conversation about a problem is roughly a 30-day artefact: after that the person has either solved it, given up, or bought something. A review of a local business is more like 90 days, because the operational problem underneath moves more slowly than a thread does. A tool with one window for everything is either throwing away good reviews or serving you stale complaints.

Fix: per-listener windows, set from what kind of evidence it is.

6. Duplicate

Already seen, by URL or by title. The same thread surfacing under three queries, a cross-post, a syndicated copy, an aggregator reprint.

Deduplication is unglamorous and it is a large share of perceived noise. Three copies of one good signal reads as a spammy feed, and it also inflates every count you might use to judge whether the listener is working.

Fix: structural. Dedupe on URL first, then on normalised title.

7. Put out of contention by the reranker

The candidate survived every cheap gate and is plausibly on topic, but when scored against the query alongside everything else retrieved, it did not make the cut.

This is the first judgement call in the chain and it is the one that makes the economics work: retrieve broadly, rank cheaply, classify narrowly. Without a reranking step you either retrieve narrowly (and miss the posts that do not use your vocabulary, which are the interesting ones) or you classify everything, which does not survive contact with a monthly bill.

Fix: none by hand. This is the step doing the work.

8. A happy customer

Specific to review mining, and a genuinely counterintuitive class. A five-star review of a business you were hoping to sell to is a correct, relevant, well-retrieved document that is the opposite of a lead.

Gating on rating before spending a model call is both a precision and a cost decision: if you are looking for operational pain, reviews above your star ceiling cannot contain it, and paying a classifier to confirm that is money spent to learn nothing. The local prospecting work lives or dies on this gate.

Fix: a star ceiling per listener.

9. Scored below the bar

The classifier read it, understood it, and scored it under the relevance threshold.

This is the honest middle of the distribution and the place where the bar you set matters more than the model does. Set it high and you get four things a week and miss some. Set it low and you are back to skimming. There is no correct answer, only your answer, which is why rating signals is worth doing: it moves the bar toward your judgement rather than leaving it where a model guessed on day one.

Fix: the threshold, and your thumbs.

10. Noise

The classifier read it and judged it to be nothing: a joke, a bot, a listicle, a press release, a tutorial, a marketing post with no person behind it to talk to.

This is the largest bucket in any honest system, and a vendor whose noise category is small is either retrieving very narrowly or not classifying.

Fix: none. This is the product.

The reason for recording which one applied

Here is what the taxonomy is actually for, and it has nothing to do with elegance.

A quiet feed has two possible causes with opposite fixes. Either the searches returned almost nothing (in which case your vocabulary is too narrow, your sources are wrong, or the market genuinely is not talking) or they returned plenty and the filters discarded all of it, in which case your thresholds are too aggressive or your exclusions are too broad.

From the outside those look identical. Both are an empty inbox on Tuesday.

If every discarded candidate keeps its reason, they look nothing alike. One is a retrieval report with small numbers. The other is a rejection histogram with a spike in low_score or off_topic. The first tells you to widen. The second tells you to loosen. Guess wrong and you make it worse.

This is also the only way to measure false negatives at all. False positives announce themselves: they are in your inbox, wasting your time. False negatives are invisible by construction: you never learn about the good post that was dropped. Keeping rejections for a window is what makes it possible to go back, sample them, and find out what the filter is costing you. That number is not knowable in a system that discards silently, which is most of them.

Openpulse records all ten of these per candidate and reports them in a per-listener quality report, along with per-query and per-source yield: which searches earn their cost and which never produce a kept signal. Queries that consistently return nothing get pruned.

What to ask a vendor

Four questions that separate a filter from a funnel:

• When a listener is quiet, can you tell me whether it found nothing or discarded everything? If the answer is no, every quiet week is a guess.
• Do you keep what you rejected? If not, false negatives are permanently invisible.
• Do you judge on full text or on a search snippet? A snippet contains the keyword by construction (that is why it was returned), so it looks relevant by definition, and the disqualifying context is in the part that was cut off.
• Which queries are earning their cost? A tool that cannot say is charging you for searches that have never once produced anything.