Monitoring Quota usage for Google APIs
Do you have an application accessing one of the Google APIs? Does the API that you are using have a Quota attached? (As far as I know they all do.) Have you seen this error returned from the API “dailyLimitExceeded” or maybe this one “userRateLimitExceeded” or how about this one “quotaExceeded”. Have you checked the Google Developer console trying to find some information about how much of the quota your application is using only to see the following.
Android / iOS users
If your application is android or iOS Google has made things a bit easier for you. They have released SDKs to help you on your way.
- Google Analytics SDK for iOS v3 – Getting Started
- Google Analytics SDK v4 for Android – Getting Started
If your application isn’t Android or iOS Google has left you like me hanging a little, until now that is because I am going to show you what I have done. If you are using Android and iOS i encourage you to continue reading because I am going to help you understand a few things you may want to consider also sending. But you can skip the part about how to use the measurement protocol the hard way.
Setting up an Application Google Analytics account
Go to the Admin section of your Google Analytics account, feel free to put it under the same account as your website, we might as well keep things organised. Click on the name of what ever property shows up first and you will find a drop down where you can then click on “Create new property”.
copy this Tracking ID it should start with UA-. Save it we will need it in a minute.
Note: It can take up to 48 hours before your new account will start showing data. You may want to wait and continue this tutorial later or you can continue and I will try and show you an example using a PHP script and my own tracking id I created along with this tutorial. You can send what ever you like to it I wont tell anyone 🙂
What to send?
Well the hole point of this post was that we wanted to see how much of our quota we where using right? I have found two ways to do that each gives different information in the Application account.
My first approach was in my opinion the most obvious. Sending a request and getting the results back from the API was an event.
Events are user interactions with content that can be tracked independently from a web page or a screen load. Downloads, mobile ad clicks, gadgets, Flash elements, AJAX embedded elements, and video plays are all examples of actions you might want to track as Events.
Can we agree these are events?
- Google Drive – requesting a list of fines, inserting a file, updating a file
- Google Calendar – requesting a list of calendars, inserting a new calendar.
- Google Analytics – Requesting data, requesting management account summary list.
See any request you make to an API could be considered an event.
Depending upon the type of application you have it may or may not have screens. Mine in fact runs automated part of the time with no user interaction and at other times the user may load a GUI. I have decided to consider even the automated section also a screen but have given it a different name. Screen Names “UI Load” vs “Automation” this way i can tell if the call came from a user or from the system running automated. (Not strictly quota related)
Events Vs ScreenViews
An Event will let you log the number of rows returning from the API as the event value, Screenview will let you let you see things like how long it took by sending a start and an end screen view and then checking time on screen to judge how long it took for the process to run. I also like to send some custom variables about the request with my screen view. Personally i don’t feel information about the request like size of the file and sampling level are valid to be sent with an event but they are valid to send with the screen view. Sending custom variables is not really necessary to help you figure out your quota but it will help you see exactly what your users are doing so that you can ensure your application is helping the users as much as it can.
You can log any errors your application throws also to Google Analytics. Then you can see what version of the application encountered an error. If you add some info extra info about what type of machine the application is running on you may even be able to track down the error to a specific operating system, language … (Not strictly quota related)
You can log how long the different parts of your application took to run. (Not strictly quota related)
How to send data to the Measurement protocol
So you are still here? Have I inspired you to log your application to a Google Analytics Application account? Good lets get started.
Sending the Request
As I said before this can be done in any programming language that can send a HTTP POST. You should probably look for a way to send the post asynchronous. The reason for sending the post asynchronous is to prevent slowing your application down. We don’t want to wait for a response from Google we just want to send it if the server registers it fine if it doesn’t we don’t care. The only response it returns is a 1 x 1 pixel Gif anyway.
When testing these examples you can always create a HTTP GET string and paste them in a Web Browser they will work. It is really up to you if you want to send it as a HTTP GET or a HTTP POST in your application both will work. Google recommends a POST because if the length of what you are sending is greater then the max length of a HTTP GET data will be lost. But for testing you can always just create a HTTP GET string and test it in a web browser.
GET /collect?payload_data HTTP/1.1
The first three main values that must be sent with any type of request wither it is an application account or a web account are the following.
- v – Protocol version the value you should send will always be 1. Example: v=1
- tid – Tracking ID This is the id that you got from creating the new Google Analytics Account . Example: tid=UA-XXX-Y
- cid – Client id This one is a little harder to explain i like to think of it as a session identifier. You should only create this once when your application starts. Then when you send data for the user it will all be part of the same session. Cid should be some generic number like a UUID. In .net i just use new guid(); But if your application is registered there registration key would work here as well. example:cid=35009a79-1a05-49d7-b876-2b884d0f825b
Putting it all together in a Get would be: v=1&tid=UA-XXX-Y&cid=35009a79-1a05-49d7-b876-2b884d0f825b
Application account required fields
There are a few fields that you need to send as well the values of them really don’t matter as far as I can tell. They are required Google Analytics uses them in some way that if you don’t send them no data appears in the standard reports. This is undocumented and was a pain to figure out trust me!
- an – Application name example: an=My%20Awesom%20APP
- aid – application id example: aid=com.daimto.awesom.app
- av – application version example: av=1.0.0
- aiid – Application Installer ID example: aiid=come.daimto.awesom.installer
Putting it all together for a HTTP GET would be: an=My%20Awesom%20APP&aid=com.daimto.awesom.app&av=1.0.0&aiid=come.daimto.awesom.installer
Like I said the values don’t really matter send what you like. With the exception of the version I don’t bother with them much. Version is very nice for seeing who upgraded or testing if an error comes with a specific version.
Notice how i have placed %20 in place of a space for Application name. All of your strings must be HTTP encoded.
The last thing you need to tell Google is what type of hit this is, is it a screenview, event, exception, or timing for example.
- t – Hit type example: t=screenview
Sending a ScreenView to the measurement protocol
This doesn’t do much but it just logs that someone viewed a screen. It could be improved upon with some custom variables.
Custom Dimensions and metrics have to be set up in the admin section of your Google Analytics. They are numbered 1 – 20. Lets say you created a custom dimension called cid. You could then add &cd1=35009a79-1a05-49d7-b876-2b884d0f825b to the request string above. By adding a custom dimension for cid you will then be able to track different users anonymously via the session id. The limits are endless with this really. You could create a custom metric for number of rows returned. Size of file downloaded.
Sending an Event to the measurement protocol
There are 4 extra parameters you need to send to log an event also hit type will be event.
- ec – Event Category
- ea – Event Action
- el – Event Label
- ev – Event Value
Lets say your application just uploaded a file to Google Drive. When a file is uploaded to Google Drive it returns a file resource within that resource you will find a file size. Wouldn’t it be nice to know the size of the file they uploaded?
Note: I haven’t tested this I am not actually sure the limit of the Event Value field, but I have sent it over a million so this should work.
Remember this is an event so we send t=event
Lets say we just request a list of the Users Google Analytics accounts. Do we need to know how many accounts they have probabbly not but you could send it as the event value
Now that we are logging every request to Google Analytics as an event some interesting things start to appear in Google Analytics. Check the Events over view report. All we have to do is look at Total Events for the day to know what our quota usage was for that day. I want to add that this number appears to be different from the number in the Developer console. I am not sure if its not kept up to date or if its because the number in developer console I have been told resets at midnight pacific east cost USA time.
Sending an Exception
Sending an exception is quite easy. Hit type will be sent as exception. example: t=exception
- exd – exception description: this field is huge feel free to send the full exception if you like. just remember to http encode it. example: exd=Database error
- edf – Exception Fatal: this is a Boolean value 0 or 1: example: exf=0
Lets say our application encounter an error due to the user inserting the wrong value.
Using a Google Analytics application account to monitor your quota usage is much better then using the Google Developer Console to decided it it is time to increase your quota. Using the Google Analytics measurement protocol allows to monitor your application usage gives you a lot of information about your users. You can look more into the measurement protocol and send things like language, screen resolution …. you can even create a user agent that will allow you to see a lot of information about the system your application is running on. I encourage you to look more into what can be sent with the measurement protocol. NOTE: Do not send uid, trust me just don’t, if you do and can tell me what its doing and why its doing with it please email me. Personally I found it only made a mess of my data.