Unable to read big json file from Request.Body

[Tratcher]

From @anfomin on September 30, 2015 15:1

I'm trying to read big JSON file (UTF8) from Request.Body stream:

string content;
using (var sr = new StreamReader(Request.Body, Encoding.Default))
    content = sr.ReadToEnd();

At the beginning it's ok, but content variable contains invalid character sequence after some offset.

Sample repo: https://github.com/anfomin/DNX-MVC-bigfile. There is file sample.json. Just run dnx kestrel and make:

POST http://localhost:5000

with body = sample.json file.

You will find mismatches since line 84.

Source:

Result:

I've tried latest DNX 1.0.0-beta8-15736 from https://www.myget.org/F/aspnetvnext/api/v2 and 1.0.0-rc1-15775 from https://www.myget.org/F/aspnetcidev/api/v2.

Copied from original issue: aspnet/Mvc#3235

[Tratcher]

From @Eilon on September 30, 2015 18:49

@kichalla can you take a look?

[Tratcher]

From @kichalla on September 30, 2015 19:51

@anfomin I suppose you are running on Mono on Linux? was curious if Encoding.Default was doing something here but looks like some content is entirely missing based on your snapshot above...just to isolate the issue, can you try saving the read content to a file and see if the contents are any different? also does this issue repro consistently for you?

[Tratcher]

From @anfomin on September 30, 2015 20:38

@kichalla I tried this sample both on Windows and Mac, result is the same.

Also I tried to save Request.Body to a text file via File.CreateText("..."). Result file contains the same corruptions. You can remove Encoding.Default, the result does not change.

The problem is Request.Body because when I change input stream to File.Open(...) then output is correct.

[Tratcher]

From @kichalla on September 30, 2015 21:2

@anfomin you are right, I was able to repro it on a Windows machine now..investigating...

left -> actual file content
right -> content read from request body and saved to a file

[Tratcher]

From @Eilon on September 30, 2015 21:5

Weird... putting this on RC. Please enlist @halter73 @lodejard @Tratcher @moozzyk if needed because this might even be a Kestrel issue.

[Tratcher]

From @kichalla on September 30, 2015 21:13

Yeah, this seems like a Kestrel issue as I cannot repro it with weblistener

[lodejard]

I'll take a look also

[kichalla]

@loudej Probably you are the best person to take a look at it...I tried to figure it out but I got lost 😄 ..following are some details:

• This issue seems to repro only for specific request sizes and characters...for example, the below repro works fine if the character is ASCII...so my wild guess is that it has something to do with decoder and its flushing...

• You can use the following repro code
  I used the LargeResponseApp in Kestrel repo for this

  app.Run(async (context) =>
          {
              var reader = new StreamReader(context.Request.Body);
              var content = await reader.ReadToEndAsync();
  
              using (var sw = new StreamWriter(File.Create(@"C:\Users\kichalla\Documents\files\actual.txt")))
              {
                  await sw.WriteAsync(content);
              }
          });

  Client app:

  public class Program
  {
      public void Main(string[] args)
      {
          var filePath = @"C:\users\kichalla\documents\files\expected.txt";
          using (var streamWriter = File.CreateText(filePath))
          {
              streamWriter.Write((new string('й', 5 * 1024 * 1024)));
          }
          //--------
          var httpClient = new HttpClient();
          var response = httpClient.PostAsync("http://localhost:5000", new StreamContent(File.OpenRead(filePath))).Result;
          Console.WriteLine(response.ToString());
          Console.WriteLine(response.Content.ReadAsStringAsync().Result);
      }
  }

[lodejard]

Perfect! Thanks @Tratcher, @kichalla those should repro very well in a unit test also