How to use the NextPageToken with the C# 3


Please Share

Google Drive API V3

Google Drive API V3

This Tutorial was completely rewritten in April, 2021

What is pagination?

Methods like the files.list in the Google Drive V3 API return a limited number of results per request. Each request returns a page of results. With the file.list method the maximum number of results or files returned per request is 1000. If there are more than 1000 rows to be returned to you, then the response will also include a nextPageToken, this token can be passed with the next request as a pageToken pareamter. This will give you the next page of results for your request, in this manner paginating over the requests can be a little cumbersome in my opinion.

Several years ago I mentioned this to Jon Skeet who is one of the other contributors on the Google .net client library. Together we came up with an alternate solution. I like to call it the PAGINATOR, Jon however named it file PageStreamer.

Create .net core console application

For this I am going to create a .net core console application, but you can use service accounts, with libraries or with asp .net core its up to you which type of project you create but for this example we are going to keep it simple and just use a console app.
I will be using service account authentication in order to access data stored on my personal drive account without need for user authorization.

Create Service Account

If you would like to follow along you will need to go to Google developer console and create a new service account. Download the json key file and don’t forget to enable the google drive api under libraires. If you have any issues I have a video which you can go check out and it will walk you through creating
service account credential key file.

How to create Google Oauth2 Service account credentials.

Create .net core console application

Nuget package

You will need to add the Google.apis.drive.v3 package to your project.

Constant

I have added a single constant to the top of my program.cs file, this is the path to the service account Json credentials file which I downloaded from Google Developer console. The service account json key file contains all the information needed by my application to use the service account to authorize to google.

private const string PathToServiceAccountKeyFile = @"C:\Youtube\dev\ServiceAccountCred.json";

Load service account using GoogleCredential.FromFile

To load the service account is a single like of code, first we load the service account key file using FromFile, then we set which scope of access the service account will need in order to do what it is we intend to do. With service accounts I normally just request full access.

// Load the Service account credentials and define the scope of its access.
var credential = GoogleCredential.FromFile(PathToServiceAccountKeyFile)
    .CreateScoped(DriveService.ScopeConstants.Drive);

Create DriveService

Once we have created our service account credential we can then go about creating our DriveService object. The DriveService object is the object we will use to make all of our calls to the Google Drive API v3.

// Create the  Drive service.
var service = new DriveService(new BaseClientService.Initializer()
    {
     HttpClientInitializer = credential
    });

File.list search files in directory

Now we need to set up our FileListRequest, in this case we would like to list all of the files in the folder on my Google drive account which i have shared with the service account. After that we would also like to see 1000 rows per page this is just to save the number of requests we will be making. The default page size for the file.list method is 100, as I uploaded 5000 dummy files to my drive account, I think it would be better to set the PageSize parameter a bit higher.

var request = service.Files.List();                                  // Create file list request.
request.Q = "parents in  '10krlloIS2i_2u_ewkdv3_1NqcpmWSL1w'";  // List only files in this folder
request.PageSize = 1000;                                         // Return 1000 files per page.

Configure PageStreamer

Now we will need to configure the PageStreamer method. The pagesteramer method takes several parameters. First it needs requestModifier which is an Action to modify a request to include the specified page token, then
a tokenExtractor which is a function used to extract the next page token from a response, followed by a
resourceExtractor which is also a function used to extract a sequence of resources from a response.

var pageStreamer = new Google.Apis.Requests.PageStreamer<google.apis.drive.v3.data.file, filesresource.listrequest,="" google.apis.drive.v3.data.filelist,="" string="">(
                    (req, token) => request.PageToken = token,
                    response => response.NextPageToken,
                    response => response.Files);
</google.apis.drive.v3.data.file,>

Store the results

Finally we will need to consider what we would like to do with the response. Personally I like to store it all into a single variable. By storing the fill file response from the API I can then sort the results locally as I see fit.

 var all = new Google.Apis.Drive.v3.Data.FileList();
 all.Files = new List();

foreach (var result in await pageStreamer.FetchAllAsync(request, CancellationToken.None))
{
all.Files.Add(result);
}

Display the results

Finally i will just run though each of the files and write them to the screen.

foreach (var file in all.Files.OrderBy(file => file.Name))
            {
                Console.WriteLine(file.Name);
            }

Conclusion

Paginating over the nextpagetoken can be a little annoying, however the Google .net client library has implemented an option for you called pagestereamer which makes fetching the results over multiple pages much easier.

Please Share


About Linda Lawton

My name is Linda Lawton I have more than 20 years experience working as an application developer and a database expert. I have also been working with Google APIs since 2012 and I have been contributing to the Google .Net client library since 2013. In 2013 I became a a Google Developer Experts for Google Analytics.

Leave a Reply to Andreas Geier Cancel reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

3 thoughts on “How to use the NextPageToken with the C#

    • Linda Lawton Post author

      PageStreamer if memory serves is part of Google.Apis and yes if you import any of the API NuGet packages they will automatically grab Google.APIs (Google.Apis.Drive.V2 || Google.Apis.Drive.V3).

      Yes it was release a while ago one of those stealth releases that the team is becoming very good at. I really need to get on them about creating actual change log release announcements.

      Note: If you are running a project with Framework 4.0 then you probably wont have this it was probably released some where around version 1.12 and support for Framework 4.0 stopped at 1.10. In that case yes you can just grab the PageStreamer method and add it to your project manually it “should” work.

  • Andreas Geier

    Hi Linda,
    thank you very much for your Pagination Solution. I found your article and started to use it immediatly. It works perfect for me now. I use the Calendar API v3 and I face one problem:
    I do a full sync of all Events of a particular google calendar with about 7000 entries to a SQL-Server-Database.
    After I received the last entry, I expect to get also the “nextSyncToken” Property in the eventsList-Result.
    But it is always null.
    Any toughts?
    Thanks in advance.
    Kind regards
    Andreas