Deduplicating Retrieved Chunks

A retriever often returns the same passage more than once — the same sentence indexed under slightly different casing or spacing, or a chunk that overlaps a neighbor. Feeding those duplicates into the prompt wastes context budget and biases the model toward whatever got repeated. Before the chunks reach the LLM, you dedup them while keeping the best-ranked copy of each.

Task

Implement dedup_chunks(chunks):

1. Compute a normalized key for each chunk: lowercase it, then collapse every run of whitespace (spaces, tabs, newlines) into a single space and strip the ends.
2. Walk the list in order. The first time a normalized key is seen, keep that chunk; every later chunk with an already-seen key is dropped.
3. Return the kept chunks as their original (un-normalized) strings, in the original order.

Because the input is ranked best-first, keeping the first occurrence keeps the highest-ranked copy.

Example

chunks = [
    "The Fed raised rates.",
    "the   fed raised rates.",     # duplicate of #1 after normalization
    "Inflation cooled in Q2.",
    "The Fed raised rates.",       # exact duplicate of #1
]

dedup_chunks(chunks)
# -> ["The Fed raised rates.", "Inflation cooled in Q2."]

Constraints

• Do not call any real LLM, network, or external service — this is pure string processing.
• Normalization affects only the comparison key; the returned strings must be the original chunk text.
• Preserve the input order of the surviving chunks.
• An empty input returns an empty list.

Approach

Make one pass over the ranked list. For each chunk compute a normalized key (lowercase, whitespace collapsed) and check it against a seen set. Because the list is ordered best-first, the first time a key appears is the highest-ranked copy, so we append the original chunk and record the key; later chunks with a seen key are skipped. This keeps order, keeps the best copy, and runs in a single linear scan.

Solution

def dedup_chunks(chunks: list[str]) -> list[str]:
    def normalize(text: str) -> str:
        # lowercase, then collapse any whitespace run to one space and strip ends
        return " ".join(text.lower().split())

    seen: set[str] = set()
    out: list[str] = []
    for chunk in chunks:
        key = normalize(chunk)
        if key in seen:
            continue                  # a lower-ranked duplicate — drop it
        seen.add(key)
        out.append(chunk)             # keep the original, un-normalized text
    return out

Walkthrough

On the example list:

1. "The Fed raised rates." normalizes to "the fed raised rates.". Not in seen, so we add the key and append the original.
2. "the fed raised rates." normalizes to the same "the fed raised rates." (the three spaces collapse to one). The key is already in seen, so it is skipped.
3. "Inflation cooled in Q2." normalizes to "inflation cooled in q2.". New key, so it is kept.
4. "The Fed raised rates." normalizes to "the fed raised rates.", already seen, so it is dropped.

Result: ["The Fed raised rates.", "Inflation cooled in Q2."] — the highest-ranked copy of each distinct passage, in order.

Complexity & notes

• Time: O(n · m) for n chunks of average length m — each chunk is normalized once (.lower() and .split() are linear in its length) and set lookups are O(1) on the resulting key. Space: O(n · m) for the seen keys plus the output list.
• " ".join(text.split()) is the compact idiom for collapse-and-strip: str.split() with no argument splits on any run of whitespace and discards empties, so tabs, newlines, and repeated spaces all normalize identically.
• This catches exact-after-normalization duplicates, not semantic paraphrases. If the interviewer pushes on “the Fed hiked rates” vs “the Fed raised rates,” that is near-duplicate detection by embedding cosine similarity or MinHash/shingling — a different, heavier tool. Say so; the cheap normalized-key pass is the right first line of defense and usually removes most retriever redundancy.
• Keeping the original string (not the normalized key) matters: the LLM should see the real passage, and downstream citation/offsets depend on the untouched text.