Achieving High Throughput and Elasticity in a Larger-than-Memory Store

Millions of sensors, mobile applications and machines are now generating billions of events. This has led to specialized many-core key-value stores (KVSs) that can ingest and index these events at high rates (100 Mops/s/machine). These systems are efficient if events are generated on the same machine, but, to be practical and cost-effective they must ingest events over the network and scale across cloud resources elastically. We present Shadowfax, a new distributed key-value store that transparently spans DRAM, SSDs, and cloud blob storage while serving 130 Mops/s/VM over commodity Azure VMs using conventional Linux TCP. Beyond high single-VM performance, Shadowfax uses a unique approach to distributed reconfiguration that avoids any server-side key ownership checks or cross-core coordination both during normal operation and migration. Hence, Shadowfax can shift load in 17 s to improve cluster throughput by 10% with little disruption. Compared to the state-of-the-art, it has 8x better throughput (than Seastar+memcached) and scales out 6x faster.