Reducing noise in a crowded feed
A wire that carries everything ends up telling you nothing. Between the first alert and the last correction, a single event arrives four or five times: once from the agency that broke it, once from a paper that took the agency copy, once from a broadcaster that rewrote the paper, and once from a columnist reacting to all three. The desk exists to recognise that shape and to keep one clean record of it. Thinning is the name we give that work, and these notes set out how it is done.
Why a crowded feed needs thinning
Volume is not the same as coverage. A hundred items about one storm may hold two facts that matter and ninety-eight restatements of them. The restatement is not wicked; it is how a wire works. Copy is filed once, then carried, trimmed, re-headed and re-dated by every desk that takes it. Each step adds a little drift and subtracts a little context, and a reader who meets the hundredth version first has no way of knowing that the tenth was fuller.
Thinning is the counterweight. It treats the arriving stream as raw material rather than as a finished list, and it asks of every item a single question: what does this add that the record already held does not? The answer is often “nothing”, and saying so plainly is most of the discipline. The mechanics sit inside the topic filter; the reasoning is set out here.
Duplicate stories and the syndicated copy behind them
A duplicate is rarely an exact copy. Two desks handling the same agency file will change the opening line, cut a paragraph, and add a local sentence, so that a machine comparing whole texts sees two different documents and a reader sees the same story twice. The unit that matters is therefore not the document but the event: the same people, the same place, the same figures, inside a short window. Three tells recur often enough to be worth naming.
- The quotation arrives intact. A named official says the same eight words in the same order in four outlets, with the same pause marked by a dash. That is one interview, carried widely.
- The figures do not drift at all. Three outlets print the same damage estimate to the same decimal place. Independent counting almost never agrees that closely.
- The same odd phrase recurs. “A source close to the discussions” appears in five places, unexplained in each. One desk wrote it; the rest inherited it.
Grouping by event rather than by text is what makes the count honest. A reader who is told “eleven outlets reported this” is being told something useful; a reader shown eleven separate items is being shown the same thing eleven times and invited to mistake repetition for weight.
Low-value items, and the test they have to pass
An item is not low value because it is short, dull, or about a small country. It is low value when it adds neither a fact, nor a named voice, nor a consequence that a reader following the thread would want. The table below is the working test, applied item by item before anything is merged into a thread.
| What the item brings | Handling | Reason |
|---|---|---|
| A new figure from an official statement | Kept | Adds a fact to the record |
| A reaction with no new detail | Held | Adds a voice, no fact; noted, not led |
| A re-headed agency paragraph | Dropped | Duplicate of the file already held |
| A correction to an earlier figure | Kept first | Overrides what the thread already said |
| A preview of a scheduled event | Held | Adds timing only; useful once the event opens |
The thinning pass, step by step
- Cluster the arriving items by the names, places and figures they have in common, within a window of a few hours.
- Take the earliest item that carries the full set of facts as the spine of the cluster.
- Admit a later item only if it corrects the spine, adds a fact, or adds a named voice with something to say.
- Attach one tone reading to the item kept, not an average of the tone of every copy.
- Record how many items were merged, so the weight of the coverage stays visible in the thread.
What thinning never does
It never drops a story because the story is awkward for someone, and it never treats a small outlet as noise by default. A local paper with one reporter can be the only desk that stood in the street, and its sentence is often the one that survives. Nor does thinning average two readings into a bland middle: where two versions disagree in pitch, the disagreement is kept and marked, which is the subject of the editorial standards for the desk.
The result is a shorter list that is harder to misunderstand. If you want the vocabulary used above — cluster, spine, merge count — the glossary defines each one, and the reader questions page takes up the cases where the count looks wrong.
Sentiment Analysis Platform is an independent editorial service. It is not connected to, endorsed by, or acting on behalf of any broadcaster, publisher, newsroom, agency, or platform named anywhere on this site. Nothing published here is professional advice of any kind.