Metadata filters: who may see what

Filters restrict retrieval to documents whose metadata matches, before ranking. A customer's question should never retrieve a staff-only policy.

Example

public = {"field": "meta.audience", "operator": "==", "value": "public"}
for document in retriever.run(query="who can approve refunds", filters=public)["documents"]:
    print(document.meta["audience"], "|", document.content)

A filter is a dictionary: a field, an operator and a value. Only public documents were ranked, so the staff rule about approving refunds cannot appear in a customer's answer, however well it matches.

Example

staff_refunds = {
    "operator": "AND",
    "conditions": [
        {"field": "meta.topic", "operator": "==", "value": "refunds"},
        {"field": "meta.audience", "operator": "in", "value": ["staff", "public"]},
    ],
}
print(len(store.filter_documents(filters=staff_refunds)))
print([d.content[:20] for d in retriever.run(query="refunds", filters=staff_refunds)["documents"]])

AND, OR and NOT combine conditions. Operators include ==, !=, in, not in, > and <. filter_documents uses the same filters without a query.

Filters are the permission check

Decide the filter from who is asking, in your code, from their login. Never let the question text choose the audience: "show me staff rules" must still run with the customer's filter.

Try it yourself

Pass filters= when creating the retriever and run without one.
Filter on a field no document has.
Write a filter for public documents that are not about delivery.

PreviousBM25 retrieval: keyword search

Slow is fine. Stopping is the only problem.