feat(Default Data Loader Node): Add default text splitter #15786

nikhilkuria · 2025-05-28T08:54:13Z

Summary

This PR makes it easy for users to start with Simple Vector Store.
Currently, users have to set up the following nodes in succession

Simple Vector Store
Document Loader
Text Splitter

This makes the users confused on which properties to pick in each of these

To make it simpler, we add a new property to the Document Loaders.

With the "Simple" option, we pick a Text Splitter by default.

With the "Custom" option, the users can pick a Text Splitter of their choice

Related Linear tickets, Github issues, and Community forum posts

https://linear.app/n8n/issue/ADO-3519/default-text-splitting-in-the-data-loaders

Review / Merge checklist

PR title and summary are descriptive. (conventions)
Docs updated or follow-up ticket created.
Tests included.
PR Labeled with release/backport (if the PR is an urgent fix that needs to be backported)

…ttps://github.com/n8n-io/n8n into feat-ado-3519-default-text-splitting-data-loaders

codecov · 2025-05-28T10:37:00Z

Codecov Report

Attention: Patch coverage is 46.66667% with 16 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...efaultDataLoader/DocumentDefaultDataLoader.node.ts	46.66%	7 Missing and 1 partial ⚠️
.../DocumentGithubLoader/DocumentGithubLoader.node.ts	46.66%	7 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

…loaders

…ttps://github.com/n8n-io/n8n into feat-ado-3519-default-text-splitting-data-loaders

cubic-dev-ai

cubic found 16 issues across 8 files. Review them in cubic.dev

_{React with 👍 or 👎 to teach cubic. Tag @cubic-dev-ai to give specific feedback.}

...nodes/document_loaders/DocumentDefaultDataLoader/test/DocumentDefaultDataLoader.node.test.ts

...hain/nodes/document_loaders/DocumentDefaultDataLoader/V1/DocumentDefaultDataLoaderV1.node.ts

...odes-langchain/nodes/document_loaders/DocumentGithubLoader/V1/DocumentGithubLoaderV1.node.ts

...langchain/nodes/document_loaders/DocumentGithubLoader/test/DocumentGithubLoader.node.test.ts

cubic-dev-ai

cubic found 16 issues across 8 files. Review them in cubic.dev

_{React with 👍 or 👎 to teach cubic. Tag @cubic-dev-ai to give specific feedback.}

...nodes/document_loaders/DocumentDefaultDataLoader/test/DocumentDefaultDataLoader.node.test.ts

...hain/nodes/document_loaders/DocumentDefaultDataLoader/V1/DocumentDefaultDataLoaderV1.node.ts

...odes-langchain/nodes/document_loaders/DocumentGithubLoader/V1/DocumentGithubLoaderV1.node.ts

...langchain/nodes/document_loaders/DocumentGithubLoader/test/DocumentGithubLoader.node.test.ts

Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>

…ttps://github.com/n8n-io/n8n into feat-ado-3519-default-text-splitting-data-loaders

dariacodes

On the high-level code looks good, tested it locally - it works! 🚀

shortstacked · 2025-06-03T08:34:54Z

Workflow Test Results 📊 🔴 1 Failed, ⚠️ 4 Warnings, 👍 78 Successful out of 83 total workflows.

Detail: Workflows failing: 237: Workflow contains 1 deleted data. View full workflow run

Tested Ref: 7fcc7f591882fee2025a14e79247ffb2b24cdf2f by @dariacodes

❌ Failed Tests (1)

Workflow ID	Workflow Name	Reason
237	BasicLLMChain:AzureChat	Workflow contains 1 deleted data.

⚠️ Warnings (4)

Workflow ID	Workflow Name	Reason
35	Slack:User:getPresence info:UserProfile:get update...	Workflow contains new data that previously did not exist.
53	ConvertKit:CustomField:create getAll update delete...	Workflow contains new data that previously did not exist.
257	Agent:auto-fix:anthropic	Workflow contains new data that previously did not exist.
48	Asana:Project:getAll get:Task:create update move g...	Workflow contains new data that previously did not exist.

github-actions · 2025-06-03T08:41:47Z

✅ All Cypress E2E specs passed

nikhilkuria · 2025-06-03T10:49:30Z

@dariacodes I had to make a change to how the nodes are versioned to fix a bug. Can you please review again?

shortstacked · 2025-06-03T13:06:37Z

Workflow Test Results 📊 🔴 1 Failed, ⚠️ 4 Warnings, 👍 78 Successful out of 83 total workflows.

Detail: Workflows failing: 237: Workflow contains 1 deleted data. View full workflow run

Tested Ref: c2ff14bd1e6b0b550cd58a7a83a957954edf0264 by @dariacodes

❌ Failed Tests (1)

Workflow ID	Workflow Name	Reason
237	BasicLLMChain:AzureChat	Workflow contains 1 deleted data.

⚠️ Warnings (4)

Workflow ID	Workflow Name	Reason
35	Slack:User:getPresence info:UserProfile:get update...	Workflow contains new data that previously did not exist.
53	ConvertKit:CustomField:create getAll update delete...	Workflow contains new data that previously did not exist.
257	Agent:auto-fix:anthropic	Workflow contains new data that previously did not exist.
48	Asana:Project:getAll get:Task:create update move g...	Workflow contains new data that previously did not exist.

github-actions · 2025-06-03T13:13:40Z

✅ All Cypress E2E specs passed

Cadiac

Looks good to me, tested the functionality and it seemed to work. The "is the new field present" based approach on the inputs & version checks on supplyData is definitely the way to go here, splitting this into two classes would have been overkill here - we usually do smaller node changes like you've done here. 👍

Cadiac · 2025-06-03T14:27:36Z

...langchain/nodes/document_loaders/DocumentDefaultDataLoader/DocumentDefaultDataLoader.node.ts

 export class DocumentDefaultDataLoader implements INodeType {
 	description: INodeTypeDescription = {
 		displayName: 'Default Data Loader',
 		name: 'documentDefaultDataLoader',
 		icon: 'file:binary.svg',
 		group: ['transform'],
-		version: 1,
+		version: [1, 1.1],
+		defaultVersion: 1.1,


By the way, we don't usually define the defaultVersion when we're defining components like this, by default the latest version is used!

Cadiac · 2025-06-03T14:47:09Z

...langchain/nodes/document_loaders/DocumentDefaultDataLoader/DocumentDefaultDataLoader.node.ts

+		inputs.push({
+			displayName: 'Text Splitter',
+			maxConnections: 1,
+			type: 'ai_textSplitter',


Downside of defining this as a function and calling toString() on it instead of inlining it is that you can't use enum values here, like you discovered.

This is why many of our nodes seem to just inline these,

https://github.com/n8n-io/n8n/pull/15915/files#diff-76afc25d81dbfea47a75d0ae4b1555db59b162ade6c650e6ff9baabb320a8ae6R53-R63

with that trick you can still use the enum values, as long as you pass those to the template literal.

But we're unlikely to ever change these enum values and your solution has the benefit of having the function typed so in my opinion we can keep this too, we seem to follow this pattern at ChainLLM node for instance too. 👍

Cadiac · 2025-06-03T14:49:52Z

...nodes/document_loaders/DocumentDefaultDataLoader/test/DocumentDefaultDataLoader.node.test.ts

Nice that you added tests for node this!

shortstacked · 2025-06-03T15:00:34Z

Workflow Test Results 📊 🔴 1 Failed, ⚠️ 3 Warnings, 👍 79 Successful out of 83 total workflows.

Detail: Workflows failing: 237: Workflow contains 1 deleted data. View full workflow run

Tested Ref: 9a66636110445a8253534ef5d30354919fd4d4a1 by @Cadiac

❌ Failed Tests (1)

Workflow ID	Workflow Name	Reason
237	BasicLLMChain:AzureChat	Workflow contains 1 deleted data.

⚠️ Warnings (3)

Workflow ID	Workflow Name	Reason
35	Slack:User:getPresence info:UserProfile:get update...	Workflow contains new data that previously did not exist.
257	Agent:auto-fix:anthropic	Workflow contains new data that previously did not exist.
48	Asana:Project:getAll get:Task:create update move g...	Workflow contains new data that previously did not exist.

github-actions · 2025-06-03T15:06:38Z

✅ All Cypress E2E specs passed

Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>

janober · 2025-06-11T10:06:27Z

Got released with [email protected]

Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>

nikhilkuria added 3 commits May 28, 2025 09:58

feat-ado-3519 add default text splitting

6456f5d

feat-ado-3519 add default text splitting

d5dab5d

Merge branch 'feat-ado-3519-default-text-splitting-data-loaders' of h…

650b69d

…ttps://github.com/n8n-io/n8n into feat-ado-3519-default-text-splitting-data-loaders

n8n-assistant bot added the n8n team Authored by the n8n team label May 28, 2025

nikhilkuria changed the title ~~feat(DataLoader Node): add default text splitter~~ feat(Default Data Loader Node): add default text splitter May 28, 2025

nikhilkuria added 2 commits May 28, 2025 12:28

ado-3519-remove expression in new property

ff74381

feat(ado-3519): refactor methods inside node

50e32ee

nikhilkuria changed the title ~~feat(Default Data Loader Node): add default text splitter~~ feat(Default Data Loader Node): Add default text splitter May 28, 2025

feat(ADO-3519): cleaning up comments

13650b8

nikhilkuria closed this May 30, 2025

nikhilkuria reopened this May 30, 2025

nikhilkuria added 8 commits May 30, 2025 15:01

Merge branch 'master' into feat-ado-3519-default-text-splitting-data-…

21a37a5

…loaders

feat(ADO-3519): adding tests to see if simple property is used properly

1060487

feat(ADO-3519): fix type errors in tests

d716a83

feat(ADO-3519): fix type errors in tests and small refactor

aff5dad

feat(ADO-3519): add versioning to DocumentDefaultDataLoader

1b1c511

feat(ADO-3519): add versioning to DocumentGithubDataLoader

aa8773e

feat(ADO-3519): add versioning to DocumentGithubDataLoader

5426d15

Merge branch 'feat-ado-3519-default-text-splitting-data-loaders' of h…

a1fabe7

…ttps://github.com/n8n-io/n8n into feat-ado-3519-default-text-splitting-data-loaders

nikhilkuria marked this pull request as ready for review June 2, 2025 12:56

feat(ADO-3519): add inputs to DocumentGithubDataLoader

d49cf9c

nikhilkuria requested review from a team and dariacodes June 2, 2025 13:10

cubic-dev-ai bot reviewed Jun 2, 2025

View reviewed changes

nikhilkuria and others added 4 commits June 2, 2025 17:34

feat(ADO-3519): add additional assert

cfa605f

Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>

feat(ADO-3519): more assert statements and fixing typos

4898aad

feat(ADO-3519): reorder option for github loader

78ccab7

merge branch 'feat-ado-3519-default-text-splitting-data-loaders' of h…

1fa6227

…ttps://github.com/n8n-io/n8n into feat-ado-3519-default-text-splitting-data-loaders

nikhilkuria added 2 commits June 2, 2025 21:59

feat(ADO-3519): using simple as the default option

dcd55aa

feat(ADO-3519): removing assert to check for process call

7fcc7f5

dariacodes previously approved these changes Jun 3, 2025

View reviewed changes

feat(ADO-3519): use inline versioning and not separate folders

037f042

nikhilkuria dismissed dariacodes’s stale review via 037f042 June 3, 2025 10:31

feat(ADO-3519): correct version numbers

c2ff14b

dariacodes previously approved these changes Jun 3, 2025

View reviewed changes

feat(ADO-3519): copy change in options

9a66636

nikhilkuria dismissed dariacodes’s stale review via 9a66636 June 3, 2025 13:22

nikhilkuria requested a review from Cadiac June 3, 2025 14:10

Cadiac approved these changes Jun 3, 2025

View reviewed changes

nikhilkuria merged commit 40850c9 into master Jun 3, 2025
36 checks passed

nikhilkuria deleted the feat-ado-3519-default-text-splitting-data-loaders branch June 3, 2025 15:14

Alexandero89 pushed a commit to Alexandero89/n8n that referenced this pull request Jun 4, 2025

feat(Default Data Loader Node): Add default text splitter (n8n-io#15786)

fa3f908

Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>

Alexandero89 pushed a commit to Alexandero89/n8n that referenced this pull request Jun 4, 2025

feat(Default Data Loader Node): Add default text splitter (n8n-io#15786)

71b207a

Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>

This was referenced Jun 11, 2025

🚀 Release 1.98.0 #16216

Closed

🚀 Release 1.98.0 #16224

Closed

🚀 Release 1.98.0 #16225

Merged

janober added the Released label Jun 11, 2025

TianYi0217 pushed a commit to TianYi0217/n8n that referenced this pull request Jun 14, 2025

feat(Default Data Loader Node): Add default text splitter (n8n-io#15786)

1f1b97b

Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>

feat(Default Data Loader Node): Add default text splitter #15786

feat(Default Data Loader Node): Add default text splitter #15786

Uh oh!

Conversation

nikhilkuria commented May 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Related Linear tickets, Github issues, and Community forum posts

Review / Merge checklist

Uh oh!

codecov bot commented May 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

cubic-dev-ai bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dariacodes left a comment

Choose a reason for hiding this comment

Uh oh!

shortstacked commented Jun 3, 2025

Workflow Test Results 📊 🔴 1 Failed, ⚠️ 4 Warnings, 👍 78 Successful out of 83 total workflows.

❌ Failed Tests (1)

⚠️ Warnings (4)

Uh oh!

github-actions bot commented Jun 3, 2025

Uh oh!

nikhilkuria commented Jun 3, 2025

Uh oh!

shortstacked commented Jun 3, 2025

Workflow Test Results 📊 🔴 1 Failed, ⚠️ 4 Warnings, 👍 78 Successful out of 83 total workflows.

❌ Failed Tests (1)

⚠️ Warnings (4)

Uh oh!

github-actions bot commented Jun 3, 2025

Uh oh!

Cadiac left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Cadiac Jun 3, 2025

Choose a reason for hiding this comment

Uh oh!

Cadiac Jun 3, 2025

Choose a reason for hiding this comment

Uh oh!

Cadiac Jun 3, 2025

Choose a reason for hiding this comment

Uh oh!

shortstacked commented Jun 3, 2025

Workflow Test Results 📊 🔴 1 Failed, ⚠️ 3 Warnings, 👍 79 Successful out of 83 total workflows.

❌ Failed Tests (1)

⚠️ Warnings (3)

Uh oh!

github-actions bot commented Jun 3, 2025

Uh oh!

Uh oh!

nikhilkuria commented May 28, 2025 •

edited

Loading

codecov bot commented May 28, 2025 •

edited

Loading

Cadiac left a comment •

edited

Loading