Tags

, , , , ,

So the problem statement: I have a server, I have a SOLR index, how do I optimize Java which is running my SOLR instance? How do I know this parameter is better than another parameter? I’ll run experiment. Measure response times (with other parameters) and repeat the experiment(s) again and again.

The lucene benchmark is the right way to measure ‘things’. I looked at the source code, wanted to extend it, but hmm, cough, it is bit hard to read :). But I know JMeter and I have used it before to measure server responses (and like statistics, so I want to see details of distributions, not just nice pictures).  Looking around more, I cannot not find anything else that satisfies my needs – persistence, reliability and being rigorous. The solrmeter is a bunch of utilites, nice, true, but not better than JMeter, so why to use it? I won’t.

The sematext monitoring platform looks great (I would like to use it, but first I have to convince our team), though it seems better fit for for real-time monitoring. And I want to run my experiments and look at very detailed info, for example such as  this one

latencies-over-time

This innocuous graph shows how long our users are waiting when I use the standard Java garbage collector. The test in this experiment ran only for 1m, but I already know that the large spike there is not random. And it is NOT innocent! It can be pretty bad… latencies-over-time

This graph shows latencies for query of x NEAR5 y (randomly generated), the two large spikes are garbage collecotor swipes. And I think I can’t see all these details in Sematext SPK, so I am forced to write my own tool to measure SOLR performance (besides, as I discovered later, 3 loops of 4 scenarios produced 720MB of data … eeeek, I don’t think SPK has that much of data)

 

So here it is:

https://github.com/romanchyla/solrjmeter

I have decided to test it on the problem of tuning Java VM for Solr. There are several posts in the SOLR mailing list with recommendations for different  Java VM parameters.

First thing first: this will run against an index with ~9.5M docs, the index has 200GB, is on a machine running CentOs 6, the SOLR will use 20GB of RAM, the machine has 64GB of RAM, the rest is for OS caching – I won’t bother you with other details :)

I have generated thousands of queries, they are of different types, exercise boolean operations, proximity operators, they search in fields, as well as across fields (unfielded search, using edismax). All of that is very specific to the way how we are parsing queries at ADS (Astrophysics Data System), but you should assume it makes sense to us to test these random queries ;-) So the 4 scenarios are:

  1. Standard Java garbage collector (without custom settings)
  2. CMS with custom settings (thanks to Shawn Heisey, who shared his setup with the mailing list0): -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:NewRatio=3 -XX:MaxTenuringThreshold=8 -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -XX:+UseLargePages -XX:+AggressiveOpts
  3. New G1 Garbage collector without parameters
  4. Custom G1 (again recommende on the list, perhaps by Otis Gospodenic, but I am not sure):-XX:+AggressiveOpts -XX:+UseG1GC -XX:+UseStringCache -XX:+OptimizeStringConcat -XX:-UseSplitVerifier -XX:+UseNUMA -XX:MaxGCPauseMillis=50 -XX:GCPauseIntervalMillis=1000

It ran last night, each test around 1.5 hours (theare are many queries, my dev machine was doing the measurement and generating graphs – so it takes long time to finish), the server is restarted with every run, there is a rampup period of 30 secs per every query class, then 6 threads are doing search for 60secs. The loop of 30 classes was repeated 3 times.

As expected, the default Java garbage collector had the worst spread (the variation as measured by standard deviation – ie. how much and how many times the response times were far away from the average response time).

cms-is-worst

The screenshot is not showing labels, but the default CMS is the light yellow. So I decided to remove it from comparison and look only at the remaining three. And they all look quite similar! I had to drill down at the details per each scenario. This is from the middle run no# 2

G1 (default)

+------------------------------------------+-------+-------+-------+-------+--------+--------+-------+------+--------+-----+--------+
 |                                     Test |   %90 |   %95 |   %98 |   %99 | %error |    avg | count |  max | median | min |  stdev |
 +------------------------------------------+-------+-------+-------+-------+--------+--------+-------+------+--------+-----+--------+
 |            booleanHighFreqAndHighFreq.aq |  41.0 |  54.0 |  69.0 |  83.0 |  0.032 | 17.000 |  7872 |  325 |     10 |   3 | 21.916 |
 |   booleanHighFreqAndHighFreqUnfielded.aq |  43.0 |  55.0 |  74.0 |  91.0 |  0.033 | 18.000 |  7690 |  364 |     11 |   3 | 22.067 |
 |             booleanHighFreqAndLowFreq.aq |  22.0 |  32.0 |  47.0 |  62.0 |  0.021 | 10.000 | 11198 |  292 |      5 |   2 | 14.297 |
 |    booleanHighFreqAndLowFreqUnfielded.aq |  21.0 |  30.0 |  43.0 |  55.0 |  0.021 | 10.000 | 11405 |  260 |      5 |   2 | 11.426 |
 |             booleanHighFreqAndMedFreq.aq |  42.0 |  63.0 |  98.0 | 133.0 |  0.035 | 21.000 |  5123 | 1047 |     10 |   2 | 53.822 |
 |    booleanHighFreqAndMedFreqUnfielded.aq |  43.0 |  64.0 | 103.0 | 137.0 |  0.035 | 22.000 |  5141 | 1177 |     10 |   2 | 55.122 |
 |          booleanHighFreqNear2HighFreq.aq |  61.0 | 143.0 | 291.0 | 562.0 |  0.181 | 31.000 |  6190 |  875 |      9 |   2 | 82.651 |
 | booleanHighFreqNear2HighFreqUnfielded.aq |  64.0 | 142.0 | 292.0 | 574.0 |  0.181 | 32.000 |  6192 |  865 |      9 |   2 | 83.321 |
 |           booleanHighFreqNear2LowFreq.aq |  16.0 |  23.0 |  49.0 |  70.0 |  0.000 |  9.000 | 12013 | 5074 |      6 |   3 | 47.976 |
 |  booleanHighFreqNear2LowFreqUnfielded.aq |  15.0 |  21.0 |  46.0 |  67.0 |  0.000 |  9.000 | 12924 |  254 |      6 |   3 | 11.578 |
 |           booleanHighFreqNear2MedFreq.aq |  31.0 |  51.0 |  73.0 |  81.0 |  0.058 | 14.000 | 10586 |  306 |      8 |   2 | 19.352 |
 |  booleanHighFreqNear2MedFreqUnfielded.aq |  29.0 |  53.0 |  71.0 |  79.0 |  0.058 | 13.000 | 10585 |  249 |      8 |   2 | 17.330 |
 |          booleanHighFreqNear5HighFreq.aq |  68.0 | 122.0 | 279.0 | 464.0 |  0.170 | 30.000 |  6501 |  604 |      9 |   2 | 69.919 |
 | booleanHighFreqNear5HighFreqUnfielded.aq |  74.0 | 124.0 | 281.0 | 455.0 |  0.170 | 31.000 |  6340 |  680 |      9 |   2 | 70.063 |
 |           booleanHighFreqNear5LowFreq.aq |  21.0 |  27.0 |  47.0 |  66.0 |  0.000 | 10.000 | 11705 |  255 |      6 |   3 | 13.651 |
 |  booleanHighFreqNear5LowFreqUnfielded.aq |  21.0 |  28.0 |  49.0 |  69.0 |  0.000 | 10.000 | 11739 |  268 |      6 |   3 | 13.623 |
 |           booleanHighFreqNear5MedFreq.aq |  35.0 |  53.0 |  71.0 | 100.0 |  0.039 | 15.000 | 10173 |  296 |      8 |   2 | 20.671 |
 |  booleanHighFreqNear5MedFreqUnfielded.aq |  33.0 |  53.0 |  70.0 | 105.0 |  0.039 | 14.000 | 10408 |  277 |      8 |   2 | 20.575 |
 |            booleanHighFreqNotHighFreq.aq |  54.0 |  74.0 | 133.0 | 201.0 |  0.041 | 26.000 |  6556 |  425 |     17 |   2 | 32.122 |
 |   booleanHighFreqNotHighFreqUnfielded.aq |  51.0 |  71.0 | 128.0 | 197.0 |  0.041 | 25.000 |  6657 |  519 |     16 |   2 | 31.576 |
 |             booleanHighFreqNotLowFreq.aq |  63.0 |  84.0 |  93.0 |  98.0 |  0.000 | 35.000 |  5253 |  290 |     32 |   3 | 22.410 |
 |    booleanHighFreqNotLowFreqUnfielded.aq |  62.0 |  81.0 |  93.0 |  95.0 |  0.000 | 35.000 |  5269 |  292 |     31 |   3 | 21.747 |
 |             booleanHighFreqNotMedFreq.aq |  54.0 |  73.0 | 137.0 | 197.0 |  0.039 | 28.000 |  6263 |  404 |     20 |   3 | 30.826 |
 |    booleanHighFreqNotMedFreqUnfielded.aq |  55.0 |  74.0 | 138.0 | 198.0 |  0.039 | 29.000 |  6192 |  450 |     20 |   3 | 31.909 |
 |             booleanHighFreqOrHighFreq.aq |  89.0 | 113.0 | 138.0 | 223.0 |  0.043 | 38.000 |  4734 |  388 |     22 |   2 | 43.684 |
 |    booleanHighFreqOrHighFreqUnfielded.aq |  90.0 | 111.0 | 134.0 | 219.0 |  0.043 | 38.000 |  4746 |  392 |     22 |   2 | 42.164 |
 |              booleanHighFreqOrLowFreq.aq | 101.0 | 181.0 | 241.0 | 267.0 |  0.010 | 52.000 |  3705 |  401 |     38 |   3 | 52.070 |
 |     booleanHighFreqOrLowFreqUnfielded.aq |  96.0 | 178.0 | 239.0 | 262.0 |  0.010 | 51.000 |  3719 |  443 |     37 |   3 | 50.728 |
 |              booleanHighFreqOrMedFreq.aq |  73.0 |  92.0 | 217.0 | 261.0 |  0.048 | 37.000 |  5288 |  339 |     24 |   2 | 42.742 |
 |     booleanHighFreqOrMedFreqUnfielded.aq |  73.0 |  90.0 | 208.0 | 257.0 |  0.048 | 36.000 |  5349 |  337 |     23 |   3 | 42.115 |
 +------------------------------------------+-------+-------+-------+-------+--------+--------+-------+------+--------+-----+--------+

G1-customized

 +------------------------------------------+-------+-------+-------+-------+--------+--------+-------+------+--------+-----+--------+
 |                                     Test |   %90 |   %95 |   %98 |   %99 | %error |    avg | count |  max | median | min |  stdev |
 +------------------------------------------+-------+-------+-------+-------+--------+--------+-------+------+--------+-----+--------+
 |            booleanHighFreqAndHighFreq.aq |  44.0 |  57.0 |  75.0 |  90.0 |  0.033 | 18.000 |  7719 |  389 |     10 |   3 | 22.858 |
 |   booleanHighFreqAndHighFreqUnfielded.aq |  43.0 |  55.0 |  71.0 |  81.0 |  0.033 | 18.000 |  7608 |  311 |     12 |   3 | 20.056 |
 |             booleanHighFreqAndLowFreq.aq |  27.0 |  37.0 |  53.0 |  70.0 |  0.021 | 11.000 | 10836 |  508 |      6 |   2 | 16.697 |
 |    booleanHighFreqAndLowFreqUnfielded.aq |  27.0 |  37.0 |  54.0 |  71.0 |  0.021 | 11.000 | 10875 |  262 |      5 |   3 | 15.767 |
 |             booleanHighFreqAndMedFreq.aq |  53.0 |  76.0 | 127.0 | 213.0 |  0.036 | 25.000 |  5047 | 1235 |     10 |   3 | 63.891 |
 |    booleanHighFreqAndMedFreqUnfielded.aq |  45.0 |  67.0 |  98.0 | 156.0 |  0.035 | 23.000 |  5094 | 1290 |     10 |   3 | 62.324 |
 |          booleanHighFreqNear2HighFreq.aq |  70.0 | 144.0 | 285.0 | 536.0 |  0.183 | 32.000 |  6122 |  858 |      9 |   2 | 81.670 |
 | booleanHighFreqNear2HighFreqUnfielded.aq |  60.0 | 147.0 | 289.0 | 524.0 |  0.181 | 31.000 |  6186 |  935 |      9 |   2 | 83.910 |
 |           booleanHighFreqNear2LowFreq.aq |  18.0 |  29.0 |  56.0 |  70.0 |  0.000 | 10.000 | 12444 |  311 |      6 |   3 | 15.384 |
 |  booleanHighFreqNear2LowFreqUnfielded.aq |  17.0 |  29.0 |  56.0 |  66.0 |  0.000 |  9.000 | 12523 |  271 |      6 |   3 | 14.220 |
 |           booleanHighFreqNear2MedFreq.aq |  29.0 |  53.0 |  78.0 |  96.0 |  0.058 | 13.000 | 10625 |  256 |      8 |   2 | 18.725 |
 |  booleanHighFreqNear2MedFreqUnfielded.aq |  29.0 |  51.0 |  71.0 |  89.0 |  0.058 | 13.000 | 10390 |  244 |      8 |   2 | 17.707 |
 |          booleanHighFreqNear5HighFreq.aq |  76.0 | 125.0 | 278.0 | 448.0 |  0.170 | 31.000 |  6338 |  632 |      9 |   2 | 71.548 |
 | booleanHighFreqNear5HighFreqUnfielded.aq |  76.0 | 126.0 | 300.0 | 455.0 |  0.170 | 31.000 |  6341 |  642 |      9 |   3 | 72.610 |
 |           booleanHighFreqNear5LowFreq.aq |  23.0 |  32.0 |  54.0 |  71.0 |  0.000 | 11.000 | 11526 |  671 |      6 |   3 | 15.748 |
 |  booleanHighFreqNear5LowFreqUnfielded.aq |  26.0 |  36.0 |  59.0 |  72.0 |  0.000 | 11.000 | 11418 |  297 |      6 |   3 | 16.288 |
 |           booleanHighFreqNear5MedFreq.aq |  37.0 |  55.0 |  74.0 | 100.0 |  0.038 | 15.000 | 10097 |  318 |      8 |   3 | 20.688 |
 |  booleanHighFreqNear5MedFreqUnfielded.aq |  39.0 |  56.0 |  70.0 |  87.0 |  0.038 | 15.000 | 10084 |  285 |      8 |   2 | 21.326 |
 |            booleanHighFreqNotHighFreq.aq |  61.0 |  80.0 | 138.0 | 210.0 |  0.042 | 28.000 |  6344 |  376 |     18 |   2 | 34.587 |
 |   booleanHighFreqNotHighFreqUnfielded.aq |  59.0 |  79.0 | 133.0 | 205.0 |  0.042 | 28.000 |  6414 |  473 |     18 |   2 | 34.226 |
 |             booleanHighFreqNotLowFreq.aq |  66.0 |  88.0 |  95.0 | 108.0 |  0.000 | 38.000 |  5090 |  312 |     34 |   4 | 24.931 |
 |    booleanHighFreqNotLowFreqUnfielded.aq |  66.0 |  88.0 |  96.0 | 109.0 |  0.000 | 38.000 |  5034 |  303 |     34 |   4 | 25.333 |
 |             booleanHighFreqNotMedFreq.aq |  57.0 |  76.0 | 140.0 | 200.0 |  0.038 | 30.000 |  6134 |  306 |     20 |   3 | 32.044 |
 |    booleanHighFreqNotMedFreqUnfielded.aq |  58.0 |  79.0 | 140.0 | 200.0 |  0.038 | 31.000 |  6119 |  293 |     21 |   2 | 32.188 |
 |             booleanHighFreqOrHighFreq.aq |  97.0 | 118.0 | 148.0 | 255.0 |  0.042 | 41.000 |  4491 |  387 |     24 |   3 | 46.294 |
 |    booleanHighFreqOrHighFreqUnfielded.aq |  96.0 | 118.0 | 149.0 | 250.0 |  0.041 | 41.000 |  4566 |  389 |     24 |   3 | 45.882 |
 |              booleanHighFreqOrLowFreq.aq |  99.0 | 180.0 | 243.0 | 271.0 |  0.010 | 53.000 |  3691 |  471 |     37 |   3 | 51.968 |
 |     booleanHighFreqOrLowFreqUnfielded.aq | 113.0 | 194.0 | 251.0 | 277.0 |  0.010 | 57.000 |  3524 |  464 |     39 |   3 | 56.207 |
 |              booleanHighFreqOrMedFreq.aq |  75.0 |  95.0 | 226.0 | 274.0 |  0.048 | 38.000 |  5198 |  342 |     25 |   2 | 43.792 |
 |     booleanHighFreqOrMedFreqUnfielded.aq |  74.0 |  94.0 | 231.0 | 265.0 |  0.047 | 37.000 |  5241 |  359 |     24 |   2 | 44.198 |
 +------------------------------------------+-------+-------+-------+-------+--------+--------+-------+------+--------+-----+--------+

CMS-customized

+------------------------------------------+-------+-------+-------+-------+--------+--------+-------+------+--------+-----+--------+
 |                                     Test |   %90 |   %95 |   %98 |   %99 | %error |    avg | count |  max | median | min |  stdev |
 +------------------------------------------+-------+-------+-------+-------+--------+--------+-------+------+--------+-----+--------+
 |            booleanHighFreqAndHighFreq.aq |  41.0 |  54.0 |  69.0 |  82.0 |  0.032 | 17.000 |  7874 |  328 |     10 |   2 | 20.510 |
 |   booleanHighFreqAndHighFreqUnfielded.aq |  45.0 |  58.0 |  78.0 |  93.0 |  0.033 | 19.000 |  7463 |  358 |     12 |   3 | 22.407 |
 |             booleanHighFreqAndLowFreq.aq |  24.0 |  32.0 |  42.0 |  49.0 |  0.021 | 10.000 | 10982 |  623 |      6 |   2 | 13.924 |
 |    booleanHighFreqAndLowFreqUnfielded.aq |  25.0 |  34.0 |  44.0 |  53.0 |  0.021 | 10.000 | 11174 |  289 |      5 |   2 | 15.003 |
 |             booleanHighFreqAndMedFreq.aq |  48.0 |  68.0 | 104.0 | 168.0 |  0.035 | 23.000 |  5101 | 1176 |     10 |   3 | 57.798 |
 |    booleanHighFreqAndMedFreqUnfielded.aq |  48.0 |  72.0 | 116.0 | 213.0 |  0.036 | 24.000 |  5022 | 1307 |     10 |   2 | 64.380 |
 |          booleanHighFreqNear2HighFreq.aq |  62.0 | 137.0 | 284.0 | 529.0 |  0.181 | 31.000 |  6281 |  870 |      9 |   2 | 79.851 |
 | booleanHighFreqNear2HighFreqUnfielded.aq |  65.0 | 143.0 | 287.0 | 559.0 |  0.181 | 32.000 |  6194 |  875 |      9 |   2 | 81.449 |
 |           booleanHighFreqNear2LowFreq.aq |  18.0 |  26.0 |  39.0 |  50.0 |  0.000 |  9.000 | 12787 |  302 |      6 |   3 | 12.519 |
 |  booleanHighFreqNear2LowFreqUnfielded.aq |  17.0 |  25.0 |  39.0 |  49.0 |  0.000 |  9.000 | 12962 |  251 |      6 |   3 | 11.857 |
 |           booleanHighFreqNear2MedFreq.aq |  28.0 |  41.0 |  66.0 |  80.0 |  0.058 | 13.000 | 10839 |  274 |      7 |   2 | 17.402 |
 |  booleanHighFreqNear2MedFreqUnfielded.aq |  30.0 |  44.0 |  68.0 |  82.0 |  0.058 | 13.000 | 10756 |  308 |      7 |   2 | 18.348 |
 |          booleanHighFreqNear5HighFreq.aq |  67.0 | 119.0 | 269.0 | 455.0 |  0.170 | 29.000 |  6536 |  611 |      9 |   3 | 68.869 |
 | booleanHighFreqNear5HighFreqUnfielded.aq |  66.0 | 118.0 | 270.0 | 458.0 |  0.171 | 29.000 |  6590 |  639 |      9 |   2 | 68.033 |
 |           booleanHighFreqNear5LowFreq.aq |  22.0 |  29.0 |  37.0 |  46.0 |  0.000 | 10.000 | 11819 |  281 |      6 |   3 | 12.898 |
 |  booleanHighFreqNear5LowFreqUnfielded.aq |  23.0 |  30.0 |  41.0 |  48.0 |  0.000 | 10.000 | 11722 |  300 |      6 |   3 | 13.517 |
 |           booleanHighFreqNear5MedFreq.aq |  35.0 |  50.0 |  65.0 |  85.0 |  0.038 | 15.000 | 10278 |  284 |      8 |   3 | 20.559 |
 |  booleanHighFreqNear5MedFreqUnfielded.aq |  33.0 |  49.0 |  63.0 |  77.0 |  0.039 | 14.000 | 10422 |  259 |      8 |   2 | 18.036 |
 |            booleanHighFreqNotHighFreq.aq |  53.0 |  73.0 | 136.0 | 206.0 |  0.041 | 26.000 |  6586 |  426 |     17 |   2 | 32.367 |
 |   booleanHighFreqNotHighFreqUnfielded.aq |  57.0 |  76.0 | 136.0 | 205.0 |  0.042 | 27.000 |  6476 |  353 |     17 |   2 | 33.097 |
 |             booleanHighFreqNotLowFreq.aq |  64.0 |  83.0 |  93.0 |  97.0 |  0.000 | 36.000 |  5220 |  294 |     33 |   3 | 22.540 |
 |    booleanHighFreqNotLowFreqUnfielded.aq |  66.0 |  86.0 |  94.0 |  99.0 |  0.000 | 37.000 |  5139 |  335 |     33 |   3 | 23.835 |
 |             booleanHighFreqNotMedFreq.aq |  57.0 |  75.0 | 139.0 | 203.0 |  0.038 | 30.000 |  6117 |  313 |     21 |   3 | 31.941 |
 |    booleanHighFreqNotMedFreqUnfielded.aq |  57.0 |  78.0 | 150.0 | 207.0 |  0.039 | 31.000 |  6072 |  414 |     21 |   3 | 34.089 |
 |             booleanHighFreqOrHighFreq.aq |  90.0 | 109.0 | 131.0 | 223.0 |  0.042 | 37.000 |  4771 |  367 |     22 |   2 | 41.495 |
 |    booleanHighFreqOrHighFreqUnfielded.aq |  95.0 | 115.0 | 136.0 | 232.0 |  0.041 | 40.000 |  4596 |  378 |     24 |   2 | 43.960 |
 |              booleanHighFreqOrLowFreq.aq | 104.0 | 178.0 | 248.0 | 270.0 |  0.010 | 54.000 |  3689 |  363 |     37 |   3 | 52.516 |
 |     booleanHighFreqOrLowFreqUnfielded.aq | 100.0 | 173.0 | 245.0 | 270.0 |  0.010 | 53.000 |  3695 |  548 |     38 |   3 | 51.579 |
 |              booleanHighFreqOrMedFreq.aq |  76.0 |  97.0 | 239.0 | 274.0 |  0.048 | 39.000 |  5138 |  332 |     27 |   3 | 43.823 |
 |     booleanHighFreqOrMedFreqUnfielded.aq |  74.0 |  92.0 | 226.0 | 264.0 |  0.048 | 37.000 |  5272 |  417 |     25 |   2 | 43.144 |
 +------------------------------------------+-------+-------+-------+-------+--------+--------+-------+------+--------+-----+--------+

I didn’t like the fact that the G1 had a query that took 5s and it had more of the maxes as well. So I decided to concentrate on the customized G1 and CMS. But they do seem to be remarkably similar!

Their distribution for the X OR Y queries are almost exactly the same, only that custom G1 finished slightly more queries there, but that can be just random thing. When I looked at distribution of one class, I first thought there was a bug in my tool, because they were almost exactly overlapping! But I checked it and there was no bug.

So there is no conclusion for me yet. Both options seem good and much better than the default garbage collector. Sematext produced interesting graph on their experience with G1, which may seem to suggest that G1 is better there: http://blog.sematext.com/2013/06/24/g1-cms-java-garbage-collector/ but I am not yet ready to make such a conclusion.

There are of course many problems with my lipstick test – it is triggering the garbage collector through the unknown number of dead objects that are there during search/fetching results, it is using not a great set of queries, it is running for only 90s per query type. But I am confident my tool is measuring what I want to measure – ie the responsiveness of our servers. It has clearly shown latency problems with the default CMS garbage collector and indicated problems with default G1. I should prepare better queries, something that loads many hits, does heavy searching and also I may want to let it run for much longer period.

And here the good news, I now have the way to test different scenarios. I don’t need to guess, or rely on somebody’s measurements, I can target our machines and I can process the data using statistical tools (because the data is available after it was harvested – btw, over 500MB of data, I should trim that down).

If you  want to look at your own SOLR instances in the same way, you can. The tool for is available as open-source at: https://github.com/romanchyla/solrjmeter

You can reproduce the same test easily – just change the following file to run against your standard SOLR server and execute the scenario: https://github.com/romanchyla/solrjmeter/blob/master/scenarios/comparing-garbage-collectors.sh

Here are few more screenshots, first comparison of standard means per different scenarios:

comparison-view

Next, the view on the details of one of the scenario runs:

day-view

And the last, the detailed view on one query type:

test-view

Advertisements