Storage and compute

Grepvine keeps storage and compute apart. Every index lives in object storage (Amazon S3), and stateless servers answer queries from copies of it cached on local NVMe drives. Object storage keeps your data durable, and the cache keeps queries fast. Neither one depends on the other to do its job.

flowchart LR
  agent["Agents and clients"] -->|"HTTPS"| api["Grepvine API"]
  subgraph compute["Compute: stateless, replaceable"]
    s1["Server<br/>NVMe cache"]
    s2["Server<br/>NVMe cache"]
    s3["Server<br/>NVMe cache"]
  end
  api --> s1 & s2 & s3
  s1 & s2 & s3 <-->|"immutable files"| store[("Object storage<br/>durable")]:::hl

Object storage holds every index; stateless servers keep cached copies of its immutable files on NVMe.

Object storage holds everything durable.

Object storage is the only durable copy of an index: your documents, their metadata and the index built over them. A write is acknowledged only after it is stored there, so from that moment it survives the loss of any server. A server that fails takes nothing with it except its cache.

Compute holds no state.

Any server can serve any index. Servers can be added, replaced or lost without copying data, because a new server reads what it needs from object storage. Compute can therefore scale with query load, independently of how much data you store. An idle index takes up no machines: it is files in object storage until the next query arrives.

Stateless doesn’t mean every query starts cold. Grepvine’s scheduler keeps each index’s data close to the servers that query it, and warms their caches before queries need it:

• Stable placement. The scheduler sends each index to the same small set of servers, so its files stay in their caches. Every API server computes the same placement on its own, with no coordinator. When a server leaves, its indexes move to the next servers in a fixed order.
• Warming ahead of queries. When background optimization produces new files for an index, the servers that serve it fetch them before any query asks for them.
• Planning data kept ready. Each server keeps the parts of an index that every query reads first ready for 30 minutes after the index’s last request. A query that arrives in that time can start work without a trip to object storage.

The NVMe cache makes queries fast.

Each server keeps a cache on local NVMe drives. When a query’s data is cached, the server answers from local disk. When it isn’t, the server fetches only the parts the query needs from object storage, answers, and then caches more of the index in the background. A server also caches what it writes, so documents you just wrote are already warm on that server.

The cache holds index files, never query results. Every query runs against the index itself, so a result never depends on an earlier query and can’t be stale.

Immutable files make the cache safe.

Grepvine writes an index as a set of immutable files. Once written, a file never changes:

• A write adds new files.
• Background optimization combines small files into larger ones and swaps them in.
• A query reads a fixed set of files: a consistent snapshot of the index.

Because a file never changes, a cached copy is always correct. Servers need no cache invalidation and don’t coordinate with each other, and any number of them can cache the same file. Every file is checked when it enters a cache and when a server first reads it. A copy that fails the check is fetched again from object storage.

Optimization runs in the background, never while a request waits. It keeps the work each query does within fixed bounds, however many writes the index has taken. After a large upload, reading the index reports optimization_progress_percent until optimization is done. Queries return the same results throughout, and get faster as optimization finishes.

Indexes split and combine automatically.

As an index grows, Grepvine may split it into parts that different servers serve, and combine the parts’ answers into one result. This happens automatically, while the index keeps taking reads and writes. You always address one index, and a query returns the same results in the same order whether the index is in one part or many. Nothing in the API, the clients or the results refers to the parts.

What this means for you.

• Durability is object storage’s. Guarantees says exactly what is promised.
• Writes wait for object storage. A write takes longer to acknowledge than a write to memory, so send documents in batches. See tradeoffs.
• A cold query is slower than a warm one. The first query after an index has left the cache reads from object storage. See tradeoffs.