Please Share

Google Drive API V3

This Tutorial was completely rewritten in April, 2021

What is pagination?

Methods like the files.list in the Google Drive V3 API return a limited number of results per request. Each request returns a page of results. With the file.list method the maximum number of results or files returned per request is 1000. If there are more than 1000 rows to be returned to you, then the response will also include a nextPageToken, this token can be passed with the next request as a pageToken pareamter. This will give you the next page of results for your request, in this manner paginating over the requests can be a little cumbersome in my opinion.

Several years ago I mentioned this to Jon Skeet who is one of the other contributors on the Google .net client library. Together we came up with an alternate solution. I like to call it the PAGINATOR, Jon however named it file PageStreamer.

Create .net core console application

For this I am going to create a .net core console application, but you can use service accounts, with libraries or with asp .net core its up to you which type of project you create but for this example we are going to keep it simple and just use a console app.
I will be using service account authentication in order to access data stored on my personal drive account without need for user authorization.

Create Service Account

If you would like to follow along you will need to go to Google developer console and create a new service account. Download the json key file and don’t forget to enable the google drive api under libraires. If you have any issues I have a video which you can go check out and it will walk you through creating
service account credential key file.

How to create Google Oauth2 Service account credentials.

Create .net core console application

Nuget package

You will need to add the Google.apis.drive.v3 package to your project.

Constant

I have added a single constant to the top of my program.cs file, this is the path to the service account Json credentials file which I downloaded from Google Developer console. The service account json key file contains all the information needed by my application to use the service account to authorize to google.

private const string PathToServiceAccountKeyFile = @"C:\Youtube\dev\ServiceAccountCred.json";

Load service account using GoogleCredential.FromFile

To load the service account is a single like of code, first we load the service account key file using FromFile, then we set which scope of access the service account will need in order to do what it is we intend to do. With service accounts I normally just request full access.

// Load the Service account credentials and define the scope of its access.
var credential = GoogleCredential.FromFile(PathToServiceAccountKeyFile)
    .CreateScoped(DriveService.ScopeConstants.Drive);

Create DriveService

Once we have created our service account credential we can then go about creating our DriveService object. The DriveService object is the object we will use to make all of our calls to the Google Drive API v3.

// Create the  Drive service.
var service = new DriveService(new BaseClientService.Initializer()
    {
     HttpClientInitializer = credential
    });

File.list search files in directory

Now we need to set up our FileListRequest, in this case we would like to list all of the files in the folder on my Google drive account which i have shared with the service account. After that we would also like to see 1000 rows per page this is just to save the number of requests we will be making. The default page size for the file.list method is 100, as I uploaded 5000 dummy files to my drive account, I think it would be better to set the PageSize parameter a bit higher.

var request = service.Files.List();                                  // Create file list request.
request.Q = "parents in  '10krlloIS2i_2u_ewkdv3_1NqcpmWSL1w'";  // List only files in this folder
request.PageSize = 1000;                                         // Return 1000 files per page.

Configure PageStreamer

Now we will need to configure the PageStreamer method. The pagesteramer method takes several parameters. First it needs requestModifier which is an Action to modify a request to include the specified page token, then
a tokenExtractor which is a function used to extract the next page token from a response, followed by a
resourceExtractor which is also a function used to extract a sequence of resources from a response.

var pageStreamer = new Google.Apis.Requests.PageStreamer<google.apis.drive.v3.data.file, filesresource.listrequest,="" google.apis.drive.v3.data.filelist,="" string="">(
                    (req, token) => request.PageToken = token,
                    response => response.NextPageToken,
                    response => response.Files);
</google.apis.drive.v3.data.file,>

Store the results

Finally we will need to consider what we would like to do with the response. Personally I like to store it all into a single variable. By storing the fill file response from the API I can then sort the results locally as I see fit.

 var all = new Google.Apis.Drive.v3.Data.FileList();
 all.Files = new List();

foreach (var result in await pageStreamer.FetchAllAsync(request, CancellationToken.None))
{
all.Files.Add(result);
}

Display the results

Finally i will just run though each of the files and write them to the screen.

foreach (var file in all.Files.OrderBy(file => file.Name))
            {
                Console.WriteLine(file.Name);
            }

Conclusion

Paginating over the nextpagetoken can be a little annoying, however the Google .net client library has implemented an option for you called pagestereamer which makes fetching the results over multiple pages much easier.

Please Share