+++ /dev/null
-------
-TOPIC:
-------
-Dynamic backoff throttle is been introduced in the Jewel timeframe to produce
-a stable and improved performance out from filestore. This should also improve
-the average and 99th latency significantly.
-
------------
-WHAT IS IT?
------------
-The old throttle scheme in the filestore is to allow all the ios till the
-outstanding io/bytes reached some limit. Once crossed the threshold, it will not
-allow any more io to go through till the outstanding ios/bytes goes below the
-threshold. That's why once it is crossed the threshold, the write behavior
-becomes very spiky.
-The idea of the new throttle is, based on some user configurable parameters
-it can start throttling (inducing delays) early and with proper parameter
-value we can prevent the outstanding io/bytes to reach to the threshold mark.
-This dynamic backoff throttle is implemented in the following scenarios within
-filestore.
-
- 1. Filestore op worker queue holds the transactions those are yet to be applied.
- This queue needs to be bounded and a backoff throttle is used to gradually
- induce delays to the queueing threads if the ratio of current outstanding
- bytes|ops and filestore_queue_max_(bytes|ops) is more than
- filestore_queue_low_threshhold. The throttles will block io once the
- outstanding bytes|ops reaches max value (filestore_queue_max_(bytes|ops)).
-
- User configurable config option for adjusting delay is the following.
-
- filestore_queue_low_threshhold:
- Valid values should be between 0-1 and shouldn't be > *_high_threshold.
-
- filestore_queue_high_threshhold:
- Valid values should be between 0-1 and shouldn't be < *_low_threshold.
-
- filestore_expected_throughput_ops:
- Shouldn't be less than or equal to 0.
-
- filestore_expected_throughput_bytes:
- Shouldn't be less than or equal to 0.
-
- filestore_queue_high_delay_multiple:
- Shouldn't be less than 0 and shouldn't be > *_max_delay_multiple.
-
- filestore_queue_max_delay_multiple:
- Shouldn't be less than 0 and shouldn't be < *_high_delay_multiple
-
- 2. journal usage throttle is implemented to gradually slow down queue_transactions
- callers as the journal fills up. We don't want journal to become full as this
- will again induce spiky behavior. The configs work very similarly to the
- Filestore op worker queue throttle.
-
- journal_throttle_low_threshhold
- journal_throttle_high_threshhold
- filestore_expected_throughput_ops
- filestore_expected_throughput_bytes
- journal_throttle_high_multiple
- journal_throttle_max_multiple
-
-This scheme will not be inducing any delay between [0, low_threshold].
-In [low_threshhold, high_threshhold), delays should be injected based on a line
-from 0 at low_threshold to high_multiple * (1/expected_throughput) at high_threshhold.
-In [high_threshhold, 1), we want delays injected based on a line from
-(high_multiple * (1/expected_throughput)) at high_threshhold to
-(high_multiple * (1/expected_throughput)) + (max_multiple * (1/expected_throughput))
-at 1.
-Now setting *_high_multiple and *_max_multiple to 0 should give us similar effect
-like old throttle (this is default). Setting filestore_queue_max_ops and
-filestore_queue_max_bytes to zero should disable the entire backoff throttle.
-
--------------------------
-HOW TO PICK PROPER VALUE ?
--------------------------
-
-General guideline is the following.
-
- filestore_queue_max_bytes and filestore_queue_max_ops:
- -----------------------------------------------------
-
- This directly depends on how filestore op_wq will be growing in the memory. So,
- user needs to calculate how much memory he can give to each OSD. Bigger value
- meaning current/max ratio will be smaller and throttle will be relaxed a bit.
- This could help in case of 100% write but in case of mixed read/write this can
- induce more latencies for reads on the objects those are not yet been applied
- i.e yet to reach in it's final filesystem location. Ideally, once we reach the
- max throughput of the backend, increasing further will only increase memory
- usage and latency.
-
- filestore_expected_throughput_bytes and filestore_expected_throughput_ops:
- -------------------------------------------------------------------------
-
- This directly co-relates to how much delay needs to be induced per throttle.
- Ideally, this should be a bit above (or same) your backend device throughput.
-
- *_low_threshhold *_high_threshhold :
- ----------------------------------
-
- If your backend is fast you may want to set this threshold value higher so that
- initial throttle would be less and it can give a chance to back end transaction
- to catch up.
-
- *_high_multiple and *_max_multiple:
- ----------------------------------
-
- This is the delay tunables and one should do all effort here so that the current
- doesn't reach to the max. For HDD and smaller block size, the old throttle may
- make sense, in that case we should set these delay multiple to 0.
-
-It is recommended that user should use ceph_smalliobenchfs tool to see the effect
-of these parameters (and decide) on your HW before deploying OSDs.
-
-Need to supply the following parameters.
-
- 1. --journal-path
-
- 2. --filestore-path
- (Just mount the empty data partition and give the path here)
-
- 3. --op-dump-file
-
-'ceph_smalliobenchfs --help' will show all the options that user can tweak.
-In case user wants to supply any ceph.conf parameter related to filestore,
-it can be done by adding '--' in front, for ex. --debug_filestore
-
-Once test is done, analyze the throughput and latency information dumped in the
-file provided in the option --op-dump-file.
Code Review / stor4nfv.git / blobdiff