Skip to content
This repository was archived by the owner on Mar 10, 2020. It is now read-only.
This repository was archived by the owner on Mar 10, 2020. It is now read-only.

add with 1MB file triggers Uncaught Error: stream.push() after EOF #967

@olizilla

Description

@olizilla
Contributor

Uploading a file > 1MB, as a pull-stream fails, with the error:

Uncaught Error: stream.push() after EOF
    at new NodeError (errors-browser.js:30)
    at readableAddChunk (_stream_readable.js:269)
    at PullDuplexStream.push../node_modules/readable-stream/lib/_stream_readable.js.Readable.push (_stream_readable.js:240)
    at next (index.js:55)
    at FileReader.loaded (index.js:22)

The issue is occuring in master on webui.ipfs.io since upgrading v30.1.2 from v29.1.0

See: ipfs/ipfs-webui#1010

Activity

olizilla

olizilla commented on Apr 15, 2019

@olizilla
ContributorAuthor

Also of note, in ipfs-webui we use https://github.com/tableflip/pull-file-reader

which has a default chunk size of 1MB https://github.com/tableflip/pull-file-reader/blob/master/index.js#L8 which suspiciously conincides with the threshold for triggering this issue.

olizilla

olizilla commented on Apr 18, 2019

@olizilla
ContributorAuthor

@hugomrdias this is blocking a relase of webui and desktop. Are you digging in to it or should I jump on it?

olizilla

olizilla commented on Apr 18, 2019

@olizilla
ContributorAuthor

investigating this some more...

Screenshot 2019-04-18 at 11 55 48

hugomrdias

hugomrdias commented on Apr 18, 2019

@hugomrdias
Contributor

im on it, will report back asap

olizilla

olizilla commented on Apr 18, 2019

@olizilla
ContributorAuthor

@hugomrdias I'm on it. The problem seems to be in pull-file-reader

olizilla

olizilla commented on Apr 18, 2019

@olizilla
ContributorAuthor

There is nothing in pull-stream-reader to stop it from emitting the cb(true) for when the offset exeeds the filesize, while it is still waiting for a data callback from the FileReader...

Screenshot 2019-04-18 at 12 28 03

in the above, with a 2Mb file, we see it trigger the offset > filesize callback before the final data chunk.

hugomrdias

hugomrdias commented on Apr 18, 2019

@hugomrdias
Contributor

But this worked before right? what changed in ipfs to trigger this problem?

olizilla

olizilla commented on Apr 18, 2019

@olizilla
ContributorAuthor

It sure did! I think something else must have been dealing with it, when it should always have been an error. I just added a quick, "count the pending chunks" check to pull-stream-reader and it fixes it! Will PR a cleaner fix now.

olizilla

olizilla commented on Apr 18, 2019

@olizilla
ContributorAuthor

Screenshot 2019-04-18 at 12 37 28

olizilla

olizilla commented on Apr 18, 2019

@olizilla
ContributorAuthor

@hugomrdias oh no wait, you are right, this is super weird, the pull-stream sink should not be asking for more chunks before it gets a callback from the previous one... that's the weirdness that is causing this. I could guard against it in pull-file-reader, but it seems like we have a rogue pull-stream sink in ipfs now that is just pulling as fast as it can, rather than waiting for each chunk

3 remaining items

ghost removed on Apr 24, 2019
reopened this on Apr 26, 2019
added a commit that references this issue on Apr 26, 2019
ghost assigned on Apr 26, 2019
added a commit that references this issue on Apr 29, 2019
added a commit that references this issue on May 8, 2019
added a commit that references this issue on May 9, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

    Participants

    @olizilla@alanshaw@lidel@hugomrdias

    Issue actions

      add with 1MB file triggers Uncaught Error: stream.push() after EOF · Issue #967 · ipfs-inactive/js-ipfs-http-client