tower/retry/budget/
mod.rs

1//! A retry "budget" for allowing only a certain amount of retries over time.
2//!
3//! # Why budgets and not max retries?
4//!
5//! The most common way of configuring retries is to specify a maximum
6//! number of retry attempts to perform before giving up. This is a familiar idea to anyone
7//! who’s used a web browser: you try to load a webpage, and if it doesn’t load, you try again.
8//! If it still doesn’t load, you try a third time. Finally you give up.
9//!
10//! Unfortunately, there are at least two problems with configuring retries this way:
11//!
12//! **Choosing the maximum number of retry attempts is a guessing game.**
13//! You need to pick a number that’s high enough to make a difference when things are somewhat failing,
14//! but not so high that it generates extra load on the system when it’s really failing. In practice,
15//! you usually pick a maximum retry attempts number out of a hat (e.g. 3) and hope for the best.
16//!
17//! **Systems configured this way are vulnerable to retry storms.**
18//! A retry storm begins when one service starts to experience a larger than normal failure rate.
19//! This causes its clients to retry those failed requests. The extra load from the retries causes the
20//! service to slow down further and fail more requests, triggering more retries. If each client is
21//! configured to retry up to 3 times, this can quadruple the number of requests being sent! To make
22//! matters even worse, if any of the clients’ clients are configured with retries, the number of retries
23//! compounds multiplicatively and can turn a small number of errors into a self-inflicted denial of service attack.
24//!
25//! It's generally dangerous to implement retries without some limiting factor. [`Budget`]s are that limit.
26//!
27//! # Examples
28//!
29//! ```rust
30//! use std::{future, sync::Arc};
31//!
32//! use tower::retry::{budget::{Budget, TpsBudget}, Policy};
33//!
34//! type Req = String;
35//! type Res = String;
36//!
37//! #[derive(Clone, Debug)]
38//! struct RetryPolicy {
39//!     budget: Arc<TpsBudget>,
40//! }
41//!
42//! impl<E> Policy<Req, Res, E> for RetryPolicy {
43//!     type Future = future::Ready<()>;
44//!
45//!     fn retry(&mut self, req: &mut Req, result: &mut Result<Res, E>) -> Option<Self::Future> {
46//!         match result {
47//!             Ok(_) => {
48//!                 // Treat all `Response`s as success,
49//!                 // so deposit budget and don't retry...
50//!                 self.budget.deposit();
51//!                 None
52//!             }
53//!             Err(_) => {
54//!                 // Treat all errors as failures...
55//!                 // Withdraw the budget, don't retry if we overdrew.
56//!                 let withdrew = self.budget.withdraw();
57//!                 if !withdrew {
58//!                     return None;
59//!                 }
60//!
61//!                 // Try again!
62//!                 Some(future::ready(()))
63//!             }
64//!         }
65//!     }
66//!
67//!     fn clone_request(&mut self, req: &Req) -> Option<Req> {
68//!         Some(req.clone())
69//!     }
70//! }
71//! ```
72
73pub mod tps_budget;
74
75pub use tps_budget::TpsBudget;
76
77/// For more info about [`Budget`], please see the [module-level documentation].
78///
79/// [module-level documentation]: self
80pub trait Budget {
81    /// Store a "deposit" in the budget, which will be used to permit future
82    /// withdrawals.
83    fn deposit(&self);
84
85    /// Check whether there is enough "balance" in the budget to issue a new
86    /// retry.
87    ///
88    /// If there is not enough, false is returned.
89    fn withdraw(&self) -> bool;
90}
tower/retry/budget/mod.rs

tower/retry/budget/
mod.rs
