Skip to content

Add hybrid memory/disk cache storages to optimise Airtable API data fetching during server initial build #92

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 18 commits into from
Jun 5, 2020
Merged

Add hybrid memory/disk cache storages to optimise Airtable API data fetching during server initial build #92

merged 18 commits into from
Jun 5, 2020

Conversation

Vadorequest
Copy link
Member

@Vadorequest Vadorequest commented Jun 4, 2020 •
edited
Loading

Issue

Fetching Airtable API performs 3 API requests per page, for each page. This generates around 140 API calls.

In-memory cache doesn't work, as it's reset between pages.

Implementation

This PR implements a hybrid memory/disk cache storage.
Initially, a memory cache had been written, but it didn't work because of the above mentioned behaviour. So, it has been extended to allow disk storage too.

Tested locally and on Vercel. Has unit tests.

See vercel/next.js#13765

@Vadorequest
Copy link
Member Author

[GitHub Actions]
Deployment FAILED
Commit 7f6d111 failed to deploy to (click to see logs)

@Vadorequest
Copy link
Member Author

[GitHub Actions]
Deployment SUCCESS
Commit 93cb7fd successfully deployed to https://nrn-v2-mst-aptd-at-lcz-sty-c1-5zdzxz5dv.now.sh
Deployment aliased as https://nrn-v2-mst-aptd-at-lcz-sty-c1-airtable-api-ca.now.sh

@Vadorequest
Copy link
Member Author

[GitHub Actions]
E2E tests FAILED
Download artifacts (screenshots + videos) from checks section at the top

@Vadorequest Vadorequest mentioned this pull request Jun 4, 2020
Open
@Vadorequest
Copy link
Member Author

[GitHub Actions]
Deployment FAILED
Commit b9b359f failed to deploy to (click to see logs)

@Vadorequest
Copy link
Member Author

[GitHub Actions]
Deployment SUCCESS
Commit 8ed03d1 successfully deployed to https://nrn-v2-mst-aptd-at-lcz-sty-c1-gctj7g7j2.now.sh
Deployment aliased as https://nrn-v2-mst-aptd-at-lcz-sty-c1-airtable-api-ca.now.sh

@Vadorequest
Copy link
Member Author

[GitHub Actions]
Deployment SUCCESS
Commit 3204cd2 successfully deployed to https://nrn-v2-mst-aptd-at-lcz-sty-c1-p7tb74vr4.now.sh
Deployment aliased as https://nrn-v2-mst-aptd-at-lcz-sty-c1-airtable-api-ca.now.sh

@Vadorequest
Copy link
Member Author

[GitHub Actions]
Deployment SUCCESS
Commit 62e8463 successfully deployed to https://nrn-v2-mst-aptd-at-lcz-sty-c1-ev19yklkb.now.sh
Deployment aliased as https://nrn-v2-mst-aptd-at-lcz-sty-c1-airtable-api-ca.now.sh

@Vadorequest Vadorequest marked this pull request as ready for review June 5, 2020 21:49
@Vadorequest
Copy link
Member Author

[GitHub Actions]
Deployment SUCCESS
Commit 17a37d0 successfully deployed to https://nrn-v2-mst-aptd-at-lcz-sty-c1-ndzqgnk22.now.sh
Deployment aliased as https://nrn-v2-mst-aptd-at-lcz-sty-c1-airtable-api-ca.now.sh

@Vadorequest
Copy link
Member Author

[GitHub Actions]
Deployment SUCCESS
Commit 5eab392 successfully deployed to https://nrn-v2-mst-aptd-at-lcz-sty-c1-5xpusb5b7.now.sh
Deployment aliased as https://nrn-v2-mst-aptd-at-lcz-sty-c1-airtable-api-ca.now.sh

@Vadorequest
Copy link
Member Author

[GitHub Actions]
Deployment SUCCESS
Commit e9d07ff successfully deployed to https://nrn-v2-mst-aptd-at-lcz-sty-c1-31owutixg.now.sh
Deployment aliased as https://nrn-v2-mst-aptd-at-lcz-sty-c1-airtable-api-ca.now.sh

@Vadorequest Vadorequest changed the title Add cache for fetching Airtable API Add hybrid memory/disk cache storages to optimise Airtable API data fetching during server initial build Jun 5, 2020
@Vadorequest Vadorequest merged commit e19a3d8 into v2-mst-aptd-at-lcz-sty Jun 5, 2020
@Vadorequest Vadorequest deleted the airtable-api-cache branch June 5, 2020 22:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@Vadorequest