US 3-030 5.2p1 [lex.phases] new-lines after phase 1 #475
Description
Translation phases 2 and 3 assume that lines are terminated by “new-line characters”. However, the current specification of phase 1 does not guarantee that to be true. In particular, for a UTF-8 file the verbatim sequence of source file characters forms the input for phase 2, even on systems where the line terminator is a carriage return. The non-UTF-8 specification is also defective in that it speaks of “introducing” new-line characters, even for encodings like Latin-1 where new-lines might already be present and no “introduction” is needed or appropriate.
Proposed change:
If an input file is determined to be a UTF-8 file, then it shall be a well-formed UTF-8 code unit sequence and it is decoded to produce a sequence of UCS scalar values that constitutes the sequence of elements of the translation character set, representing each line-termination character or character sequence as a new-line character.
For any other kind of input file supported by the implementation, characters are mapped, in an implementation-defined manner, to a sequence of translation character set elements (5.3) (introducing new-line characters for representing end-of-line indicators as new-line characters).