[HttpParser] In absence of `content-length` and `chunked-encoding` headers, wait till server closes the connection

[abhinavsingh]

Describe the bug
NOTE: Happens only for HTTP/1.0 scenarios.

I have seen this bug only while browsing my router gateway via 192.168.1.1. My home gateway doesn't seem to send any content-length header or indicate that the response will be chunked. This breaks HttpParser class which expects one of the header to be present. Resulting in NotImplementedError

https://github.com/abhinavsingh/proxy.py/blob/develop/proxy/http/parser.py#L174-L175

To Reproduce
Steps to reproduce the behavior:

$ python
>>> from proxy.http.parser import HttpParser, httpParserTypes
>>> p = HttpParser(httpParserTypes.RESPONSE_PARSER)
>>> p.parse(b'HTTP/1.0 200 OK\r\nCP=CAO PSA OUR\r\nCache-Control:private,max-age=0;\r\nX-Frame-Options:SAMEORIGIN\r\nX-Content-Type-Options:nosniff\r\nX-XSS-Protection:1; mode=block\r\nContent-Security-Policy:default-src \'self\' \'unsafe-inline\' \'unsafe-eval\'\r\nStrict-Transport-Security:max-age=2592000; includeSubdomains\r\nSet-Cookie: lang=eng; path=/;HttpOnly;\r\nContent-type:text/html;charset=UTF-8;\r\n\r\n<!-- HTML RESPONSE HERE -->')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/abhinavsingh/Dev/proxy.py/proxy/http/parser.py", line 175, in parse
    'Response headers doesn\'t contain content-length header and neither is chunked encoded.')
NotImplementedError: Parser shouldn't have reached here

Expected behavior
Trust server implementation and wait till server has closed the connection.

[sofair]

I am facing the same problem. I didn't find a simple way to override the HttpParser class, so just modified the parser.py file of the package.
I treat it as a complete response. so the code was modified to:

                if b'content-length' in self.headers:
                    'balabala'
                elif self.is_chunked_encoded():
                    'balabala'
                else:
                    print("E*E"*20)
                    self.state = httpParserStates.RCVING_BODY
                    if self.body is None:
                        self.body = b''
                    total_size = len(raw)
                    received_size = len(self.body)
                    self.body += raw[:total_size - received_size]
                    if True:
                        self.state = httpParserStates.COMPLETE
                    more, raw = len(raw) > 0, raw[total_size - received_size:]
                    # comment the exception
                    # raise NotImplementedError('Parser shouldn\'t have reached here')

[abhinavsingh]

Hi @sofair, are you facing this issue for router applications or is it happening for websites in the wild? If for a real website, could you also share raw HTTP response sent by the server for further debugging. Thank you!!!

[sofair]

Hi @sofair, are you facing this issue for router applications or is it happening for websites in the wild? If for a real website, could you also share raw HTTP response sent by the server for further debugging. Thank you!!!

Yes, it happens for a real website, but I'm not coding a general-purpose proxy, I'm just using it to modify some post data in the middle between an app and its web service. In my impression, this happens when the website response is a short JSON string. I'm sorry that I didn't record the raw data.

[abhinavsingh]

I'm sorry that I didn't record the raw data.

No worries we do have a raw sample from my internal home router page which reproduces this bug.

In my impression, this happens when the website response is a short JSON string.

In my experience this happens when neither does the server return a content-length header nor a header indicating that the response will be chunked. Hence, parse currently end up raising raise NotImplementedError('Parser shouldn\'t have reached here')

I am unsure what exactly does the HTTP spec says about this use case. E.g.

1. Should proxy (i.e. client) wait for server to close the connection?
2. Blindly assuming that response is complete can lead to situations where server might have intended a chunked response. But we decided to close the connection in absence of necessary headers.

This leads little more investigation to remove ambiguity and be complaint with the specifications. Targeting this for the next release.

[abhinavsingh]

Found some useful information about it. So client (i.e. our proxy) should continue reading from the server until it closes the connection. We must ignore this behavior for HEAD, 1xx, 204, 304 type responses.

See https://stackoverflow.com/a/15995101/414221