[BUG]: Endpoints union types causes tsc to abort with out of memory error

[MisanthropicBit]

What happened?

I'm building docker images for two projects which both has a dependency (my project as well) that in turn depends on a bunch of @octokit packages including @octokit/types. However, it started getting oom errors from tsc --emitDeclarationOnly.

When debugging the issue, I ran tsc --emitDeclarationOnly --generateTrace trace-folder to get a trace.json and types.json file (see instructions here). Examining the trace revealed that tsc is has very deep recursive events for type-checking the Endpoints type in the checkSourceFile with an event called traceUnionsOrIntersectionsTooLarge_DepthLimit.

I haven't found a viable solution since I can't tell typescript to ignore type-checking of @octokit/types only. The --skipLibCheck option has no effect, possibly because the package is indirectly included in the source code through imports. Adding NODE_OPTIONS="--max-old-space-size=4096" to the docker build or increasing the memory limit in Docker Desktop solves the issue but is more symptom treatment than a cure. This might be an issue with the typescript compiler, although a large recursive type would explain the oom error, so I'll submit some info to them as well.

Interestingly, eslint also runs out of memory at times. EDIT: This might be because I also have typescript-eslint installed which calls the typescript's typechecking APIs to enable type-aware linting.

An aside: The link to your code of conduct in the issue template is broken.

Versions

Versions as given in a direct dependency for both projects.

• @octokit/auth-app: "^7.1.5"
• @octokit/core: "^6.1.4"
• @octokit/plugin-paginate-rest: "^11.4.2"
• @octokit/plugin-retry: "^7.1.4"
• @octokit/plugin-throttling: "^9.4.0"
• @octokit/types: "^13.8.0"
• typescript: "^5.5.3"
• @typescript-eslint/eslint-plugin: "7.6.0",
• @typescript-eslint/parser: "7.6.0",

Relevant log output

0.822 > tsc --emitDeclarationOnly
0.822
57.23
57.23 <--- Last few GCs --->
57.23
57.23 [19:0x56cf2e0]    55901 ms: Scavenge 2044.1 (2079.4) -> 2043.1 (2081.4) MB, 2.45 / 0.00 ms  (average mu = 0.264, current mu = 0.241) allocation failure;
57.23 [19:0x56cf2e0]    55925 ms: Scavenge 2045.1 (2081.4) -> 2044.0 (2081.4) MB, 3.10 / 0.00 ms  (average mu = 0.264, current mu = 0.241) allocation failure;
57.23 [19:0x56cf2e0]    55942 ms: Scavenge 2045.1 (2081.4) -> 2044.2 (2082.7) MB, 8.14 / 0.00 ms  (average mu = 0.264, current mu = 0.241) allocation failure;
57.23
57.23
57.23 <--- JS stacktrace --->
57.23
57.23 FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
57.23 ----- Native stack trace -----
57.23
57.24  1: 0xb8d0a3 node::OOMErrorHandler(char const*, v8::OOMDetails const&) [node]
57.24  2: 0xf062e0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
57.24  3: 0xf065c7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
57.24  4: 0x1118165  [node]
57.24  5: 0x11186f4 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [node]
57.24  6: 0x112f5e4 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [node]
57.24  7: 0x112fdfc v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
57.24  8: 0x1131f5a v8::internal::Heap::HandleGCRequest() [node]
57.25  9: 0x109d5c7 v8::internal::StackGuard::HandleInterrupts() [node]
57.25 10: 0x153fec2 v8::internal::Runtime_StackGuardWithGap(int, unsigned long*, v8::internal::Isolate*) [node]
57.25 11: 0xffffb9d59ef6
57.34 Aborted

Code of Conduct

• I agree to follow this project's Code of Conduct

[github-actions]

👋 Hi! Thank you for this contribution! Just to let you know, our GitHub SDK team does a round of issue and PR reviews twice a week, every Monday and Friday! We have a process in place for prioritizing and responding to your input. Because you are a part of this community please feel free to comment, add to, or pick up any issues/PRs that are labeled with Status: Up for grabs. You & others like you are the reason all of this works! So thank you & happy coding! 🚀

[wolfy1339]

I've never experienced this myself.

Are you sure that you don't inadvertently have multiple copies of @octokit/types with differing versions in your node_modules?

Can you share a minimal reproduction so I can debug this

[MisanthropicBit]

Hey. Thanks for getting back to me. I've never experienced anything like this myself either.

Are you sure that you don't inadvertently have multiple copies of @octokit/types with differing versions in your node_modules?

I did try to run npm explain @octokit/types and got a lot of output. It seems like the type lib is being included from the other octokit packages I listed above from the (inhouse) dependency I mentioned which isn't surprising.

In that same dependency, all octokit packages are dependencies except for @octokit/types which is a development dependency. I also tried to do npm dedupe but to no avail. Let me know if I can test/cleanup this in some other way.

Can you share a minimal reproduction so I can debug this

The projects are closed source but I can try to make one and get back to you.

[wolfy1339]

@octokit/types is used pretty extensively throughout alot of the Octokit packages. It is expected to see it multiple times in the dependency tree, but it should be deduped by NPM and installed once.

npm explain probably isn't the best, npm ls is better for this case

[MisanthropicBit]

npm explain probably isn't the best, npm ls is better for this case

I don't think npm ls will help me here since the package isn't reported as its included through a dependency.

I looked at the trace info again and here is a bit of the output where the event traceUnionsOrIntersectionsTooLarge_DepthLimit is reported:

{"id":23315,"recursionId":21249,"unionTypes":[22313,22314,22315,22316,22317,22318,22319,22320,22321,22322,22323,22324,22325,22326,22327,22328,22329,22330,22331,22332,22333,22334,22335,22336,22337,22338,22339,22340,22341,22342,22343,22344,22345,22346,22347,22348,22349,22350,22351,22352,22353,22354,22355,22356,22357,22358,22359,22360,22361,22362,22363,22364,22365,22366,22367,22368,22369,22370,22371,22372,22373,22374,22375,22376,22377,22378,22379,22380,22381,22382,22383,22384,22385,22386,22387,22388,22389,22390,22391,22392,22393,22394,22395,22396,22397,22398,22399,22400,22401,22402,22403,22404,22405,22406,22407,22408,22409,22410,22411,22412,22413,22414,22415,22416,22417,22418,22419,22420,22421,22422,22423,22424,22425,22426,22427,22428,22429,22430,22431,22432,22433,22434,22435,22436,22437,22438,22439,22440,22441,22442,22443,22444,22445,22446,22447,22448,22449,22450,22451,22452,22453,22454,22455,22456,22457,22458,22459,22460,22461,22462,22463,22464,22465,22466,22467,22468,22469,22470,22471,22472,22473,22474,22475,22476,22477,22478,22479,22480,22481,22482,22483,22484,22485,22486,22487,22488,22489,22490,22491,22492,22493,22494,22495,22496,22497,22498,22499,22500,22501,22502,22503,22504,22505,22506,22507,22508,22509,22510,22511,22512,22513,22514,22515,22516,22517,22518,22519,22520,22521,22522,22523,22524,22525,22526,22527,22528,22529,22530,22531,22532,22533,22534,22535,22536,22537,22538,22539,22540,22541,22542,22543,22544,22545,22546,22547,22548,22549,22550,22551,22552,22553,22554,22555,22556,22557,22558,22559,22560,22561,22562,22563,22564,22565,22566,22567,22568,22569,22570,22571,22572,22573,22574,22575,22576,22577,22578,22579,22580,22581,22582,22583,22584,22585,22586,22587,22588,22589,22590,22591,22592,22593,22594,22595,22596,22597,22598,22599,22600,22601,22602,22603,22604,22605,22606,22607,22608,22609,22610,22611,22612,22613,22614,22615,22616,22617,22618,22619,22620,22621,22622,22623,22624,22625,22626,22627,22628,22629,22630,22631,22632,22633,22634,22635,22636,22637,22638,22639,22640,22641,22642,22643,22644,22645,22646,22647,22648,22649,22650,22651,22652,22653,22654,22655,22656,22657,22658,22659,22660,22661,22662,22663,22664,22665,22666,22667,22668,22669,22670,22671,22672,22673,22674,22675,22676,22677,22678,22679,22680,22681,22682,22683,22684,22685,22686,22687,22688,22689,22690,22691,22692,22693,22694,22695,22696,22697,22698,22699,22700,22701,22702,22703,22704,22705,22706,22707,22708,22709,22710,22711,22712,22713,22714,22715,22716,22717,22718,22719,22720,22721,22722,22723,22724,22725,22726,22727,22728,22729,22730,22731,22732,22733,22734,22735,22736,22737,22738,22739,22740,22741,22742,22743,22744,22745,22746,22747,22748,22749,22750,22751,22752,22753,22754,22755,22756,22757,22758,22759,22760,22761,22762,22763,22764,22765,22766,22767,22768,22769,22770,22771,22772,22773,22774,22775,22776,22777,22778,22779,22780,22781,22782,22783,22784,22785,22786,22787,22788,22789,22790,22791,22792,22793,22794,22795,22796,22797,22798,22799,22800,22801,22802,22803,22804,22805,22806,22807,22808,22809,22810,22811,22812,22813,22814,22815,22816,22817,22818,22819,22820,22821,22822,22823,22824,22825,22826,22827,22828,22829,22830,22831,22832,22833,22834,22835,22836,22837,22838,22839,22840,22841,22842,22843,22844,22845,22846,22847,22848,22849,22850,22851,22852,22853,22854,22855,22856,22857,22858,22859,22860,22861,22862,22863,22864,22865,22866,22867,22868,22869,22870,22871,22872,22873,22874,22875,22876,22877,22878,22879,22880,22881,22882,22883,22884,22885,22886,22887,22888,22889,22890,22891,22892,22893,22894,22895,22896,22897,22898,22899,22900,22901,22902,22903,22904,22905,22906,22907,22908,22909,22910,22911,22912,22913,22914,22915,22916,22917,22918,22919,22920,22921,22922,22923,22924,22925,22926,22927,22928,22929,22930,22931,22932,22933,22934,22935,22936,22937,22938,22939,22940,22941,22942,22943,22944,22945,22946,22947,22948,22949,22950,22951,22952,22953,22954,22955,22956,22957,22958,22959,22960,22961,22962,22963,22964,22965,22966,22967,22968,22969,22970,22971,22972,22973,22974,22975,22976,22977,22978,22979,22980,22981,22982,22983,22984,22985,22986,22987,22988,22989,22990,22991,22992,22993,22994,22995,22996,22997,22998,22999,23000,23001,23002,23003,23004,23005,23006,23007,23008,23009,23010,23011,23012,23013,23014,23015,23016,23017,23018,23019,23020,23021,23022,23023,23024,23025,23026,23027,23028,23029,23030,23031,23032,23033,23034,23035,23036,23037,23038,23039,23040,23041,23042,23043,23044,23045,23046,23047,23048,23049,23050,23051,23052,23053,23054,23055,23056,23057,23058,23059,23060,23061,23062,23063,23064,23065,23066,23067,23068,23069,23070,23071,23072,23073,23074,23075,23076,23077,23078,23079,23080,23081,23082,23083,23084,23085,23086,23087,23088,23089,23090,23091,23092,23093,23094,23095,23096,23097,23098,23099,23100,23101,23102,23103,23104,23105,23106,23107,23108,23109,23110,23111,23112,23113,23114,23115,23116,23117,23118,23119,23120,23121,23122,23123,23124,23125,23126,23127,23128,23129,23130,23131,23132,23133,23134,23135,23136,23137,23138,23139,23140,23141,23142,23143,23144,23145,23146,23147,23148,23149,23150,23151,23152,23153,23154,23155,23156,23157,23158,23159,23160,23161,23162,23163,23164,23165,23166,23167,23168,23169,23170,23171,23172,23173,23174,23175,23176,23177,23178,23179,23180,23181,23182,23183,23184,23185,23186,23187,23188,23189,23190,23191,23192,23193,23194,23195,23196,23197,23198,23199,23200,23201,23202,23203,23204,23205,23206,23207,23208,23209,23210,23211,23212,23213,23214,23215,23216,23217,23218,23219,23220,23221,23222,23223,23224,23225,23226,23227,23228,23229,23230,23231,23232,23233,23234,23235,23236,23237,23238,23239,23240,23241,23242,23243,23244,23245,23246,23247,23248,23249,23250,23251,23252,23253,23254,23255,23256,23257,23258,23259,23260,23261,23262,23263,23264,23265,23266,23267,23268,23269,23270,23271,23272,23273,23274,23275,23276,23277,23278,23279,23280,23281,23282,23283,23284,23285,23286,23287,23288,23289,23290,23291,23292,23293,23294,23295,23296,23297,23298,23299,23300,23301,23302,23303,23304,23305,23306,23307,23308,23309,23310,23311,23312,23313,23314],"flags":["Union"]},

Looking at ID 23315 (the first type ID in the union):

{"id":23316,"recursionId":21250,"substitutionBaseType":22294,"constraintType":23315,"flags":["Substitution","IncludesInstantiable"]},
{"id":23317,"recursionId":21251,"indexedAccessObjectType":20606,"indexedAccessIndexType":23316,"flags":["IndexedAccess","IncludesWildcard"]},
{"id":23318,"recursionId":21252,"intersectionTypes":[22313,22294],"flags":["Intersection"]},
{"id":23319,"recursionId":21253,"intersectionTypes":[22314,22294],"flags":["Intersection"]},
{"id":23320,"recursionId":21254,"intersectionTypes":[22315,22294],"flags":["Intersection"]},
{"id":23321,"recursionId":21255,"intersectionTypes":[22316,22294],"flags":["Intersection"]},
{"id":23322,"recursionId":21256,"intersectionTypes":[22317,22294],"flags":["Intersection"]},
{"id":23323,"recursionId":21257,"intersectionTypes":[22318,22294],"flags":["Intersection"]},

ID 22294 is included in every intersection type and points to:

{"id":22294,"symbolName":"R","recursionId":20233,"firstDeclaration":{"path":"../types.ts/pkg/dist-types/endpointinterface.d.ts","start":{"line":25,"character":6},"end":{"line":25,"character":21}},"flags":["TypeParameter","IncludesMissingType"]},

Looking at ID 23315:

{"id":22313,"recursionId":20247,"flags":["StringLiteral"],"display":"\"DELETE /app/installations/{installation_id}\""},

which is part of the generated endpoints type.

Would it perhaps make sense to break up the Endpoints per http method? So instead of Endpoints['DELETE /app/installations/{installation_id}'] you would do Endpoints['DELETE']['/app/installations/{installation_id}']. Unfortunately that is a breaking change but it would constrain the depth of recursion per http method. I'll try to test this out locally if I can 🙂

[wolfy1339]

That would certainly be a breaking change and require changes throughout the other packages.

I don't understand how you are running into this issue?