I was wondering if it’s a good idea to let such requests fail, especially when dealing with critical systems that handle tasks like payments or other sensitive operations. Do businesses generally allow such failures? Or is there typically a backup process or alternative mechanism in place to handle these scenarios?
The payments are no exception to fault tolerance. Let's say if any bank's network is down, we will not be able to do successful transactions via that bank. We see such kinds of "bank network down" prompts in the Google Pay app all the time. These transactions need to be retried by the client side or there can be an async worker who picks all these transactions and pushes them to a final state: approved/canceled/refunded.
I hope this helps. Let me know if you have further questions.
1. The initial check to see if the circuit is in Open state - it doesn’t seem thread safe. The watcher goroutine (or another goroutine running Execute) might be updating it, we might read a stale value.
2. For HalfOpen case, when there’s an error - we are updating the failureCount too. Is that needed? Shouldn’t we just dump a message into the channel, move the state back to Open and return the error?
EDIT:
Am I right to infer that the channel would have at-most 1 message at any given time? A message is dumped constrained by a mutex lock, it also updates the state of the circuit. So once a routine updates the state to Open (either from HalfOpen or Closed, doesn’t matter), next one would fail fast.
I ask this because I was thinking about a condition where watcher reads from a channel, waits and then updates the state to HalfOpen. But it doesn’t take into consideration that other goroutines by that time might have dumped another message into the channel and we would have unnecessarily flipped the state to HalfOpen. In fact we’d see circuit flipping between Open and HalfOpen (though I am inferring >1 messages is not possible, hence the implementation is correct)
Great job on making such a technical concept easy to understand and follow :)
Thanks Neha, glad you found it useful!
I was wondering if it’s a good idea to let such requests fail, especially when dealing with critical systems that handle tasks like payments or other sensitive operations. Do businesses generally allow such failures? Or is there typically a backup process or alternative mechanism in place to handle these scenarios?
Thanks for providing implementation as well.
Hey Sayan, great question!
The payments are no exception to fault tolerance. Let's say if any bank's network is down, we will not be able to do successful transactions via that bank. We see such kinds of "bank network down" prompts in the Google Pay app all the time. These transactions need to be retried by the client side or there can be an async worker who picks all these transactions and pushes them to a final state: approved/canceled/refunded.
I hope this helps. Let me know if you have further questions.
awesome article on circuit breakers
Thanks for the article!
Couple of things related to “Execute” method:
1. The initial check to see if the circuit is in Open state - it doesn’t seem thread safe. The watcher goroutine (or another goroutine running Execute) might be updating it, we might read a stale value.
2. For HalfOpen case, when there’s an error - we are updating the failureCount too. Is that needed? Shouldn’t we just dump a message into the channel, move the state back to Open and return the error?
EDIT:
Am I right to infer that the channel would have at-most 1 message at any given time? A message is dumped constrained by a mutex lock, it also updates the state of the circuit. So once a routine updates the state to Open (either from HalfOpen or Closed, doesn’t matter), next one would fail fast.
I ask this because I was thinking about a condition where watcher reads from a channel, waits and then updates the state to HalfOpen. But it doesn’t take into consideration that other goroutines by that time might have dumped another message into the channel and we would have unnecessarily flipped the state to HalfOpen. In fact we’d see circuit flipping between Open and HalfOpen (though I am inferring >1 messages is not possible, hence the implementation is correct)