This is the beginning of a page for tracking
a) What's expected when we say verify performance
b) How to go about completing that goal.
Performance Expectations:
1) Define tests at the beginning of the release as part of the design process that will validate the performance based on the use cases asked for.
Common knobs include
- Object graph size
- number of nodes
- number of threads
- read rate
- write rate
- locality of reference
2) Once you have your tests defined you should at a minimum do:
- 2 4 and 8 node testing
- Create a graph varying the important nobs
- Understand why we are not linear on any of the dimensions to 8 nodes and document it. Things to look at include:
- Lock bottlenecks
- cpu bottlenecks
- thread starvation
- network bottlneck
- boadcast bottleneck
- pipeline stalls
- disk bottleneck
- GC bottleneck
These are some tips and remarks that were sent around in internal emails:
It depends if the client nodes are competing for resources or not. So, provided we have enough perf boxes,
1. if each client is running on it's own box, and the cpu and memory (GC) is not a bottleneck,
2. and the test is partitioned to eliminate locks bouncing around or lock contention,
3. you should be able to hit the server ceiling in terms of throughput.
The server ceiling can be 1. Disk IO, 2. Memory (GC), 3. The L2 process itself (locks, concurrency etc.)
I normally tune 1 by running 1 L1 and 1 L2, then concentrate on 2 by adding 1 or 2 more L1s and then move to three to exponentially add more L1s. Keep monitoring the L2 memory and CPU at all times of course.
So, typically, you should be able to show linear increase in throughput till you hit the max of what server can handle.
All of the above assumes that the network throughput and bandwidth is not the bottleneck.
Of course, there is disk IO on the server as well to look at.
