Failure to read file in 1.21.0 with Python 2.7 and non-ascii yaml files.

[spyoungtech]

Yamllint is now failing on files that linted successfully in previous versions. This happens in files that are not entirely ascii. With Python 3.x this issue does not occur.

I believe this is due to a change in cli.py where io.open is used instead of open.

Relevant Traceback:

 $ yamllint -c .lintConfig .
 Traceback (most recent call last):
   File "/usr/local/bin/yamllint", line 11, in <module>
     sys.exit(run())
   File "/usr/local/lib/python2.7/dist-packages/yamllint/cli.py", line 181, in run
     problems = linter.run(f, conf, filepath)
   File "/usr/local/lib/python2.7/dist-packages/yamllint/linter.py", line 237, in run
     content = input.read()
   File "/usr/lib/python2.7/encodings/ascii.py", line 26, in decode
     return codecs.ascii_decode(input, self.errors)[0]
 UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 132: ordinal not in range(128)

[adrienverge]

Thanks for the report, sorry about that. I experience a similar problem on CentOS 7 with Python 3.6.

Can you try with LC_ALL=C.UTF-8 yamllint -c .lintConfig . or LC_ALL=en_US.UTF-8 yamllint -c .lintConfig .?

@bz2 any thoughts? The simple option I see is to revert the commit.

[bz2]

@adrienverge Hmm, annoying, though your suggested work around should be okay for old linux platforms without a sensible encoding set still.

Four options on fixing:

1. Revert change, breaking Python 3 newline handling again.
2. Remove Python 2.7 compatibility as its sunset.
3. Add encoding='utf-8' as an io.open() parameter.
4. Implement the character encoding detection from the yaml spec.

It's pretty easy and mostly correct to just say all yaml files are utf-8 (but not quite, should handle the other unicode encodings) and ignore the platform encoding. Have just added test and it seems that's fine.

[spyoungtech]

Would it be easier to just to read the file in binary mode instead? Based on a quick glance at the source, this should 'just work' -- Unless that would reintroduce the newline problem you mentioned.

[spyoungtech]

Actually, the easiest thing would be to offload this to pyyaml since it's already a dependency... adjusting PR momentarily.

[spyoungtech]

Well. After fiddling with this for far longer than I'd have liked, I arrived at just writing the encoding detection per spec:

A YAML processor must support the UTF-16 and UTF-8 character encodings. If a character stream does not begin with a byte order mark (#FEFF), the character encoding shall be UTF-8. Otherwise it shall be either UTF-8, UTF-16 LE or UTF-16 BE as indicated by the byte order mark