Remove NtIdent and NtLifetime.

The extra span is now recorded in the new `TokenKind::NtIdent` and
`TokenKind::NtLifetime`. These both consist of a single token, and so
there's no operator precedence problems with inserting them directly
into the token stream.

The other way to do this would be to wrap the ident/lifetime in invisible
delimiters, but there's a lot of code that assumes an interpolated
ident/lifetime fits in a single token, and changing all that code to work with
invisible delimiters would have been a pain. (Maybe it could be done in a
follow-up.)

This change might not seem like much of a win, but it's a first step toward the
much bigger and long-desired removal of `Nonterminal` and
`TokenKind::Interpolated`. That change is big and complex enough that it's
worth doing this piece separately. (Indeed, this commit is based on part of a
late commit in #114647, a prior attempt at that big and complex change.)
This commit is contained in:
Nicholas Nethercote
2024-04-22 19:46:51 +10:00
parent 9a63a42cb7
commit 95e519ecbf
11 changed files with 131 additions and 104 deletions

View File

@@ -201,10 +201,17 @@ impl<'a> StripUnconfigured<'a> {
inner = self.configure_tokens(&inner);
Some(AttrTokenTree::Delimited(sp, spacing, delim, inner)).into_iter()
}
AttrTokenTree::Token(ref token, _)
if let TokenKind::Interpolated(nt) = &token.kind =>
{
panic!("Nonterminal should have been flattened at {:?}: {:?}", token.span, nt);
AttrTokenTree::Token(
Token {
kind:
TokenKind::NtIdent(..)
| TokenKind::NtLifetime(..)
| TokenKind::Interpolated(..),
..
},
_,
) => {
panic!("Nonterminal should have been flattened: {:?}", tree);
}
AttrTokenTree::Token(token, spacing) => {
Some(AttrTokenTree::Token(token, spacing)).into_iter()