Lexer: check in advance_token to avoid regard spare ## as GardedStrPrefix

[xizheyin]

Fixes #140618

I performed additional checks to avoid treating the two ## after r#‘xxx’#### as GardedStrPrefix.

GardedStrPrefix is introduced in 321a5db.

r? compiler

[xizheyin]

@rustbot author

[rustbot]

Reminder, once the PR becomes ready for a review, use @rustbot ready.

[xizheyin]

@rustbot ready

[xizheyin]

r? compiler

[wesleywiser]

r? compiler

[jdonszelmann]

heya, you now unconditionally look at the previous character every step of tokenization. I'm admittedly somewhat new to review rotation, and don't have a good intuition to how much lexing matters in perf. My expectation is 0 but I'm interested in the outcome of a perf run because of that. Will be reviewing more, but initial pass looks reasonable @bors try @rust-timer queue

[bors]

⌛ Trying commit 3fe5df4 with merge 606582d...

[bors]

☀️ Try build successful - checks-actions
Build commit: 606582d (606582da13e1b4b4ec1c1e80afd6ce9cb2755789)

[rust-timer]

Finished benchmarking commit (606582d): comparison URL.

Overall result: no relevant changes - no action needed

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

@bors rollup=never
@rustbot label: -S-waiting-on-perf -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

Results (primary 5.4%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	5.4%	[5.4%, 5.4%]	1
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	5.4%	[5.4%, 5.4%]	1

Cycles

This benchmark run did not return any relevant results for this metric.

Binary size

Results (secondary 0.0%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	0.0%	[0.0%, 0.0%]	1
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-	-	0

Bootstrap: 752.053s -> 750.495s (-0.21%)
Artifact size: 372.47 MiB -> 372.53 MiB (0.02%)

[xizheyin]

This was based on a very old base, so I rebased it to the new master branch.

I don't think this will have much of an impact on performance, however I'm not quite sure why max-rss is increasing by 5%, is it compared to the latest master branch? Maybe we should rerun @rust-timer queue. But I have no sufficient permissions...

[xizheyin]

I'm curious as to how these permissions were obtained. Because a couple of previous reviewers said r=me after resolving something, yet I don't have permissions to do that either.

[jdonszelmann]

both reviews and perf runs can only be done by team members. Sometimes a reviewer says r=me which effectively means any team member can merge it, the review has been done. For you it means, that's the last problem with your PR, after that someone (maybe them) will merge. Sometimes a reviewer can delegate these powers to non team members, but you can't do this yourself.

[jdonszelmann]

self.prev() is only for debug builds (read its documentation)

[jdonszelmann]

In general, it feels to me like these checks should be part of raw_or_double_quoted_string. i.e., once a string has been parsed, and we see more hashtags, error on that

[xizheyin]

it will parse the last # of #"foo"### as GuardedStrPrefix, which is error.

I think this recognition should be corrected in advance_token. But I also don't know why prev() is only used in debug, do we need to remove that debug attribute or find another way?

[jdonszelmann]

No it's there for a reason, if you look prev is only used in debug assertions. We don't really want to look behind, so you likely want to find another way to do this

[xizheyin]

Okay, I'll re-investigate when I get some free time.