Google APIs Flood buster


Please Share

Google DevelopersAre you working on an application that uses one of the Google APIs. Have you encounter one of the following errors?

  • 403 userRateLimitExceeded
  • 403 rateLimitExceeded
  • 429 RESOURCE_EXHAUSTED

These errors are basically flood protection. Your application is running to fast. When you see these errors Google recommends you implement Exponential Backoff.

Implementing exponential backoff



Exponential Backoff is where your application periodically retries a failed request over an increasing amount of time. It is a standard error handling strategy for network applications. The Google API’s are designed with the expectation that clients which choose to retry failed requests do so using exponential backoff. Besides being “required”, using exponential backoff increases the efficiency of bandwidth usage, reduces the number of requests required to get a successful response, and maximizes the throughput of requests in concurrent environments.

The flow for implementing simple exponential backoff is as follows.

  • Make a request to the API
  • Receive an error response that has a retry-able error code
  • Wait 1s + random_number_milliseconds seconds
  • Retry request
  • Receive an error response that has a retry-able error code
  • Wait 2s + random_number_milliseconds seconds
  • Retry request/li>
  • Receive an error response that has a retry-able error code
  • Wait 4s + random_number_milliseconds seconds
  • Retry request
  • Receive an error response that has a retry-able error code
  • Wait 8s + random_number_milliseconds seconds
  • Retry request
  • Receive an error response that has a retry-able error code
  • Wait 16s + random_number_milliseconds seconds
  • Retry request
  • If you still get an error, stop and log the error.

In the above flow, random_number_milliseconds is a random number of milliseconds less than or equal to 1000. This is necessary to avoid certain lock errors in some concurrent implementations. random_number_milliseconds must be redefined after each wait. If you are using any of the Google Client libraries you shouldn’t have to worry about this most of them have already implemented this for you.

Taking it one step farther

Exponential backoff works just fine. My application detects an error and waits a few milliseconds and then tries it again. The great thing is that 90% of the time I only need to wait and retry once before I get data back. The problem is that it still bothers me that I am getting the initial error message to begin with.

So I have come up with a way of effectually keeping track of how many request my application was making and then slowing it down just a little before Google would begin to detect flooding.

GoogleFlood Buster

This class is designed to allow you to keep track of the number of request you are making to Google. The time each request is made is logged by the class it tracks how many requests have been made within a given amount of time. If to many have been made then it sleeps just enough to prevent flooding the server. This does not work 100% of the time but it will reduce the number of flooding errors you get significantly.

Conclusion

Google recommends that we implement Exponential Backoff when we encounter any of the RateLimitExceeded errors. I personally like to take things one step further and try to limit the amount of flooding my applications do. For the most part Google allows us to access there APIs free of charge I like to be polite when accessing their systems.

Please Share


Linda Lawton

About Linda Lawton

My name is Linda Lawton I have more than 20 years experience working as an application developer and a database expert. I have also been working with Google APIs since 2012 and I have been contributing to the Google .Net client library since 2013. In 2013 I became a a Google Developer Experts for Google Analytics.

Leave a comment

Your email address will not be published. Required fields are marked *