------
TOPIC:
------
Dynamic backoff throttle is been introduced in the Jewel timeframe to produce
a stable and improved performance out from filestore. This should also improve 
the average and 99th latency significantly.

-----------
WHAT IS IT?
-----------
The old throttle scheme in the filestore is to allow all the ios till the 
outstanding io/bytes reached some limit. Once crossed the threshold, it will not
allow any more io to go through till the outstanding ios/bytes goes below the 
threshold. That's why once it is crossed the threshold, the write behavior 
becomes very spiky. 
The idea of the new throttle is, based on some user configurable parameters 
it can start throttling (inducing delays) early and with proper parameter
value we can prevent the outstanding io/bytes to reach to the threshold mark.
This dynamic backoff throttle is implemented in the following scenarios within
filestore.

  1. Filestore op worker queue holds the transactions those are yet to be applied.
     This queue needs to be bounded and a backoff throttle is used to gradually 
     induce delays to the queueing threads if the ratio of current outstanding 
     bytes|ops and filestore_queue_max_(bytes|ops) is more than 
     filestore_queue_low_threshhold. The throttles will block io once the 
     outstanding bytes|ops reaches max value (filestore_queue_max_(bytes|ops)).

     User configurable config option for adjusting delay is the following.
  
        filestore_queue_low_threshhold:
          Valid values should be between 0-1 and shouldn't be > *_high_threshold.

        filestore_queue_high_threshhold:
          Valid values should be between 0-1 and shouldn't be < *_low_threshold.

        filestore_expected_throughput_ops:
          Shouldn't be less than or equal to 0. 

	filestore_expected_throughput_bytes:
          Shouldn't be less than or equal to 0.
 
        filestore_queue_high_delay_multiple:
	  Shouldn't be less than 0 and shouldn't be > *_max_delay_multiple.
          
	filestore_queue_max_delay_multiple:
	  Shouldn't be less than 0 and shouldn't be < *_high_delay_multiple

  2. journal usage throttle is implemented to gradually slow down queue_transactions 
     callers as the journal fills up. We don't want journal to become full as this
     will again induce spiky behavior. The configs work very similarly to the
     Filestore op worker queue throttle.

        journal_throttle_low_threshhold
	journal_throttle_high_threshhold
	filestore_expected_throughput_ops
	filestore_expected_throughput_bytes
	journal_throttle_high_multiple
	journal_throttle_max_multiple

This scheme will not be inducing any delay between [0, low_threshold].
In [low_threshhold, high_threshhold), delays should be injected based on a line
from 0 at low_threshold to high_multiple * (1/expected_throughput) at high_threshhold.
In [high_threshhold, 1), we want delays injected based on a line from
(high_multiple * (1/expected_throughput)) at high_threshhold to 
(high_multiple * (1/expected_throughput)) + (max_multiple * (1/expected_throughput)) 
at 1.
Now setting *_high_multiple and *_max_multiple to 0 should give us similar effect
like old throttle (this is default). Setting filestore_queue_max_ops and 
filestore_queue_max_bytes to zero should disable the entire backoff throttle.
          
-------------------------
HOW TO PICK PROPER VALUE ?
-------------------------

General guideline is the following.

  filestore_queue_max_bytes and filestore_queue_max_ops:
  -----------------------------------------------------

  This directly depends on how filestore op_wq will be growing in the memory. So,
  user needs to calculate how much memory he can give to each OSD. Bigger value
  meaning current/max ratio will be smaller and throttle will be relaxed a bit.
  This could help in case of 100% write but in case of mixed read/write this can
  induce more latencies for reads on the objects those are not yet been applied 
  i.e yet to reach in it's final filesystem location. Ideally, once we reach the
  max throughput of the backend, increasing further will only increase memory
  usage and latency.

  filestore_expected_throughput_bytes and filestore_expected_throughput_ops:
  -------------------------------------------------------------------------

  This directly co-relates to how much delay needs to be induced per throttle.
  Ideally, this should be a bit above (or same) your backend device throughput.

  *_low_threshhold *_high_threshhold :
  ----------------------------------

  If your backend is fast you may want to set this threshold value higher so that
  initial throttle would be less and it can give a chance to back end transaction
  to catch up.

  *_high_multiple and *_max_multiple:
  ----------------------------------

  This is the delay tunables and one should do all effort here so that the current
  doesn't reach to the max. For HDD and smaller block size, the old throttle may
  make sense, in that case we should set these delay multiple to 0.

It is recommended that user should use ceph_smalliobenchfs tool to see the effect 
of these parameters (and decide) on your HW before deploying OSDs. 

Need to supply the following parameters.

  1. --journal-path
     
  2. --filestore-path
    (Just mount the empty data partition and give the path here)

  3. --op-dump-file 

'ceph_smalliobenchfs --help' will show all the options that user can tweak.
In case user wants to supply any ceph.conf parameter related to filestore,
it can be done by adding '--' in front, for ex. --debug_filestore

Once test is done, analyze the throughput and latency information dumped in the
file provided in the option --op-dump-file.