Pagination feature feedback

[sindresorhus]

Try out the new pagination feature and let us know how it works. We will improve it based on your feedback. The feature lets you more easily handle the case where you need to request multiple pages from an API.

• https://github.com/sindresorhus/got#pagination

---

Here are relevant issues:

• Suggestion: rename pagination() to pagination.each()
• Changes to the .pagination API

[andymantell]

Just used this with the WordPress Rest API and it just works. I've come back to the documentation to check because it was too easy and I don't know if I believe it! I feel like I've missed something - I didn't even have to specify any of the callbacks.

let posts = await got.paginate(`${process.env.CRAWL_URL}wp-json/wp/v2/posts`);

console.log(posts.length);  // outputs 152

for await (const post of posts) {
  ...
}

[szmarczak]

I think you confused got.paginate with got.paginate.all. The correct example is:

let posts = await got.paginate.all(`${process.env.CRAWL_URL}wp-json/wp/v2/posts`);

console.log(posts.length);  // outputs 152

for (const post of posts) {
  ...
}

Otherwise posts.length will be undefined as posts would be an iterator.

[andymantell]

Yes sorry you are right, the example I pasted isn't quite what I've ended up with - I trimmed it down for pasting into here and effectively broke it. got.paginate.all looks good though, I hadn't seen that.

Thanks!

[Slaventsiy]

An amazing feature, works great with GitLab API.

However, it doesn't play well with responseType and resolveBodyOnly options. If I have got instance with options { responseType: 'json', resolveBodyOnly: true } and even provide _pagination.transform without JSON.parse, got crashes on

got/source/index.ts

Line 79 in 74d66b2

if (!Reflect.has(response.headers, 'link')) {

[fiznool]

I've tried the pagination feature out today on a Google API and it works really nicely. Thank you for implementing this ❤️

I have a couple of bits of feedback:

• It would be nice to somehow type the response body, as the current types are defaulting to Response<unknown>, meaning type casting is necessary inside the paginate and transform functions.
• It's not really obvious what the difference is between pagination and pagination.all. After much trial and error I realised that the former is if you want to use for await on the returned generator, whereas the latter must be awaited directly and returns the regular array of results.

As a sidenote, I would imagine the major use case for the pagination feature is for pagination.all, i.e. get all of the results from the endpoint? for await syntax is not as well known, and (unless I'm missing something) will likely only be used in advanced scenarios. Hence, would it make sense for pagination.all to be the recommended way of paginating? And, if so, should it be used as the normal pagination function, with the for await version being renamed to (e.g.) pagination.each? I came to this expecting pagination to work just like pagination.all does today, given that most of the rest of got has this architecture (call the function, await it, get the result directly).

Edit: after refining the code, I am now using pagination directly with for await after all. However my comment above still stands, that I would expect the pagination API to work like the rest of the library by default, with an opt-in to the for await version.

[fiznool]

For reference here's my code snippet for interfacing with the Google My Business API, using TypeScript.

interface Account {
  name: string;
  accountName: string;
}

interface AccountsListResponseBody {
  accounts: Account[];
  nextPageToken?: string;
}

async function fetchLocations(accessToken: string) {
  const myBusinessAPIClient = got.extend({
    prefixUrl: 'https://mybusiness.googleapis.com/v4',
    headers: {
      authorization: `Bearer ${accessToken}`,
    },
    responseType: 'json',
  });

  const accountsSearchParams = {
    pageSize: 20,
  };
  const accounts = await myBusinessAPIClient.paginate.all<Account>(
    'accounts',
    {
      searchParams: accountsSearchParams,
      pagination: {
        transform: (response) =>
          (response.body as AccountsListResponseBody).accounts,
        paginate: (response) => {
          const { nextPageToken } = response.body as AccountsListResponseBody;
          if (!nextPageToken) {
            return false;
          }

          return {
            searchParams: {
              ...accountsSearchParams,
              pageToken: nextPageToken,
            },
          };
        },
      },
    }
  );

  console.log(accounts);
}

[szmarczak]

It would be nice to somehow type the response body, as the current types are defaulting to Response, meaning type casting is necessary inside the paginate and transform functions.

You can (and should) cast it via a template:

got/source/types.ts

Lines 53 to 63 in 3da16e0

	export interface GotPaginate {
	<T>(url: string | URL, options?: OptionsWithPagination<T>): AsyncIterableIterator<T>;
	all<T>(url: string | URL, options?: OptionsWithPagination<T>): Promise<T[]>;
	// A bug.
	// eslint-disable-next-line @typescript-eslint/adjacent-overload-signatures
	<T>(options?: OptionsWithPagination<T>): AsyncIterableIterator<T>;
	// A bug.
	// eslint-disable-next-line @typescript-eslint/adjacent-overload-signatures
	all<T>(options?: OptionsWithPagination<T>): Promise<T[]>;
	}

It's not really obvious what the difference is between pagination and pagination.all. After much trial and error I realised that the former is if you want to use for await on the returned generator, whereas the latter must be awaited directly and returns the regular array of results.

all stands for all items, which obviously is an array.

As a sidenote, I would imagine the major use case for the pagination feature is for pagination.all, i.e. get all of the results from the endpoint? for await syntax is not as well known, and (unless I'm missing something) will likely only be used in advanced scenarios.

I strongly disagree. People may want to use pagination(...) instead of pagination.all(...) to transform data on the fly. Storing 10k items in memory is not a good idea.

for await version being renamed to (e.g.) pagination.each?

IMO that's a good idea. +1 on this one. You can open an issue about this so we don't forget.

I came to this expecting

That's what many people do. They expect their code to just work. If they don't use TypeScript, they should always read the docs unless they have memorized the API. Related with #920

[PopGoesTheWza]

I am using paginate on Sugar CRM api with Typescript node applications.

First thing, it's a brilliant feature which would benefit from more documentation, example, etc. (then again it is still maturing)

Using got@10.x it works like a charm (aside some minor typing things maybe and possibly a heap crash when processing a large set of pages, but I have not fully investigate this yet)

Migrating to got@11.x is promising but currently broken for my usage. I opened several issues. Once I can have got@11.x to work, I will likely contribute a got template for Sugar CRM.

Thanks for the many great repos.

[fiznool]

@szmarczak thank you for your comments.

You can (and should) cast it via a template

Yes, that is a good feature - however the casting is only for the list of results. In some cases (such as the Google My Business API), the response body is an object with a property containing the list, like so:

{
  "accounts": [...],
  "nextPageToken": "some_token"
}

I believe the typings currently account for this possible difference, but they use the Response type without a generic, and so the response body defaults to Response<unknown> - hence the need for the manual casting inside transform and paginate.

got/source/as-promise/types.ts

Lines 73 to 81 in 3da16e0

	export interface PaginationOptions<T> {
	pagination?: {
	transform?: (response: Response) => Promise<T[]> | T[];
	filter?: (item: T, allItems: T[], currentItems: T[]) => boolean;
	paginate?: (response: Response, allItems: T[], currentItems: T[]) => Options | false;
	shouldContinue?: (item: T, allItems: T[], currentItems: T[]) => boolean;
	countLimit?: number;
	};
	}

That's what many people do. They expect their code to just work.

I understand where you are coming from, but I actually did read the documentation on the pagination API several times, and still didn't fully understand until I had stepped through the code to see what was going on. I think this is simply an issue with the documentation - I would be happy to submit a PR to improve this if it would be acceptable. As also discussed above, I think that having pagination.all and pagination.each as the two methods would also help to clear up any misunderstandings.

I hope my original comment didn't offend in any way - I think this is a fantastic feature which has saved me a lot of time, and for that I thank all of the contributors. 💜

[szmarczak]

In some cases (such as the Google My Business API), the response body is an object with a property containing the list, like so:

That's an interesting one. But I think you can just return response.body.accounts in transform() and before you do that just response.request.options.context = {token: nextPageToken};. Then you can access the token in the paginate() function.

hence the need for the manual casting inside transform and paginate.

Exactly!

I think this is simply an issue with the documentation - I would be happy to submit a PR to improve this if it would be acceptable.

A PR would be lovely, don't hestitate to send one ;)

I hope my original comment didn't offend in any way

No, it did not. You don't have anything to worry about. Actually your points are good!