Guarantees

Grepvine makes a small number of promises and states each one so you can check it.

Recall.

A query returns every match in every searchable document. Grepvine applies no top-k cutoff, no approximate index and no sampling. The index only rules out documents that can’t match, and it never drops a real match. To check this, run a literal query and grep -F with the same text over the files you uploaded, and compare the results.

Recall covers the whole set of pages, not any single page. A page can hold fewer results than you asked for, even none, when the query used up its time budget for that page. Keep following next_page_id until has_next is false. paginate in both SDKs and gv search do this for you.

Three things are outside a query’s reach, and each one is reported:

• Documents that aren’t searchable. Content that isn’t valid UTF-8, or is over 100 MiB, is stored but never matched. A listing reports it with searchable: false.
• Long lines. A result returns up to 64 KiB of text. Text cut at either end is marked text_truncated.
• Very many matches on the same lines. A result lists up to 4,096 matches. If it leaves any out, it is marked matches_truncated.

Precision.

Every result is a real match. Grepvine checks each candidate against the bytes you uploaded, using the pattern’s exact semantics, before it returns it.

• Regex mode. Uses RE2 syntax and semantics, so matching takes linear time in the length of the input. It supports Unicode classes and Unicode case folding; \b and \w are ASCII only.
• Literal mode. Matches the pattern’s bytes exactly, or under simple case folding when case_sensitive is false.
• Offsets. Grepvine never normalizes content. Every offset counts bytes from the start of the document you uploaded.

Consistency.

When a write is acknowledged, every later query, read and listing sees it, on any server. Grepvine has no refresh interval and no eventually consistent window for documents.

• Pages read one snapshot. The first page of a search fixes a snapshot of the index, and every later page reads the same snapshot. Writes that land while you page through never cause a duplicate or a gap; they show up in your next search. A page token is valid for 15 minutes, and only for the same caller, index and query, including amount.
• Conditional writes. If-Match: <revision> writes only if the document still has that revision, and If-None-Match: * only if the key doesn’t exist. Unconditional writes to the same key are ordered, and the last one wins.
• Batches. If any document in a batch is invalid, Grepvine refuses the whole batch and writes nothing. Otherwise each document succeeds or fails on its own: its result carries a new revision, or an error such as precondition_failed when it was left unchanged.
• Retries. Repeating a write with the same Idempotency-Key returns the original result and applies the write at most once, even if a server crashed in between. Repeating an interrupted batch with the same key finishes the documents it didn’t reach.

Two kinds of state can lag:

• Index counts in listings. A listing of indexes can show counts a few seconds behind the latest writes. Reading one index always returns current counts.
• Access and catalog changes. Revoking an API key, changing a member’s role or deleting an index reaches every API server within 10 seconds. It takes effect at once on the server that handled the request.

Durability.

Grepvine acknowledges a write only after it is stored in object storage (Amazon S3). From that point it survives the loss of any Grepvine server, because servers hold only a cache. The durability of your data is therefore the durability of the object store.