RFC: Cartridge bench tool for Tarantool

[filonenko-mikhail]

Cartridge bench tool for Tarantool

User Story

User setup Tarantool single node or cluster and try to understand «How many specific traffic Tarantool can handle on this hardware»

The same official things are for redis, postgresql and aerospike.

• https://redis.io/topics/benchmarks
• https://www.postgresql.org/docs/current/pgbench.html
• https://docs.aerospike.com/docs/client/go/benchmarks.html

Propolsal

Make a bench module for cartridge cli. Bench module makes some load for Tarantool.

MVP

• Server

tarantool -e 'box.cfg()'

• Benchmark

cartridge-cli bench

• Output

1m rps

The simplest case "Single Node Forever Alone"

Bench tool

• Connects to single-node tarantool
• Creates _cartridge_bench_kv

  key	value
  string	string

• Opens 10 connections
• Makes 50% write 50% read requests
  • with 1024 byte random key
  • with 10*1024 bytes random value
• With rate 10 simultaneous requests in every connection
• Measures
  • requests per second
  • bytes per second

"Twix Pair Candy Popkorn"

User sets up tarantool leader and tarantool follower with replication.
Bench tool makes the same scenario as for "Single Node", but read requests go to the follower.

"Splinter Cell"

User sets up vshard cluster with one router and two storages.
Bench tool

• Connects to all of nodes
• Sets up spaces on shards _cartridge_bench_sharded_kv

  key	value	bucket_id
  string	string	long

• bucket_id = mpcrc32(key)
• Opens 10 connections
• Makes 50% write 50% read requests
  • with 1024 byte random key
  • with 10*1024 bytes random value
• With rate 10 simultaneous requests in every connection
• User vshard.router.* functions for requests
• Measures
  • requests per second
  • bytes per second

[filonenko-mikhail]

MVP PoC

• 10 simultanios connection
• keep 10 requests on the fly
• during 20 seconds
• wait for connection closing
• count total request done

Material for

• Redis Benchmark from box
  • https://github.com/redis/redis/blob/unstable/src/redis-benchmark.c
• Golang
  • https://habr.com/ru/post/268585/

Status — DONE

[filonenko-mikhail]

Next — Single Node Forever Alone

• create space from golang
• make load profile
  • 50% write
  • 50% read requests
• make main as cartridge-cli module

  cartridge bench --url 127.0.0.1:3301 --duration 20 --connections 10
  

• make command line args for:
  • url to server
    • user
    • password
  • connection count
  • request simultaneus (on the fly)
  • time duration
  • key size
  • payload size
• research for internet wide benchmark reports (probably JMeter, Ya.Tank)

[filonenko-mikhail]

Plan For MIFI

• Learning Tarantool and Golang connector infrastructure
• Make Golang connection to Tarantool with defer closing
• Make function for request using golang connection
• Make functions as goroutines for multiplexing requests to Tarantool
• Synchronizing goroutines
• Make timer for waiting connection pool
• Make load profile (Сделать настроиваемый профиль нагрузки)
• Выполнить кластеризацию Tarantool
• Балансировать нагрузку на репликасет Tarantool
• Сделать шардированный кластер Tarantool
• Маршрутизировать запросы к шардированному кластеру

[ligurio]

CH benchmark https://clickhouse.com/docs/en/operations/utilities/clickhouse-benchmark/

[LeonidVas]

Hi! I have a conversation with Dima and looked at the descriptions of the redis-benchmark and clickhouse-benchmark and I would like to write some thoughts.
As I understand it, redis-benchmark generates the database itself in accordance with the test scenarios, and clickhouse-benchmark runs the given workload on a previously prepared database.
According to the code previously proposed in the PR, we want to implement the redis-benchmark approach. As for me, this is a more interesting approach, but we need to understand what and how we plan to measure, because if there are various requests (insert, update, select, delete), some starvation may begin (for example, we have a lot of delete, but small amount of insert, respectively, we will spend a lot of time waiting insert). It is possible that it is necessary to slightly change the approach to the filling of the DB, measurements.
For example, we can also measure the average (moving average) execution time of a request depending on its type (this will not give us information about ​​the throughput, but it will give information about ​​the latency for this type of load). We can also measure the duration of the benchmark in the number of requests and set the time limit as a limitation, then we can prepare a DB in which starvation will be unlikely / impossible.
Also, a couple of points on which I think it is important to clarify:

• The work of the tool should be predictable from launch to launch, because if the results when launching the tool with the same parameters differ by 10 times, then it will not be clear what you are measuring, the difference in hardware or how lucky the launch was.
• Tool should be positioned as giving an approximate picture of the DB performance on a given hardware with a given workload, and not the best performance that can be achieved.

I plan to take a closer look at the redis-benchmark and see how the DB is filled there in various test scenarios.