Rewrite the new attribute argument parser
Fixes https://github.com/rust-lang/rust/issues/143940
This rewrites the parser, should improve performance and maintainability.
This can be reviewed commit by commit
rustc_expand: ensure stack in `InvocationCollector::visit_expr`
In Fedora, when we built rustc with PGO on ppc64le, we started failing
the test `issue-74564-if-expr-stack-overflow.rs`. This could also be
reproduced on other arches by setting a smaller `RUST_MIN_STACK`, so
it's probably just unlucky that ppc64le PGO created a large stack frame
somewhere in this recursion path. Adding an `ensure_sufficient_stack`
solves the stack overflow.
Historically, that test and its fix were added in rust-lang/rust#74708,
which was also an `ensure_sufficient_stack` in this area of code at the
time. However, the refactor in rust-lang/rust#92573 basically left that
to the general `MutVisitor`, and then rust-lang/rust#142240 removed even
that ensure call. It may be luck that our tier-1 tested targets did not
regress the original issue across those refactors.
Prevent impossible combinations in `ast::ModKind`.
`ModKind::Loaded` has an `inline` field and a `had_parse_error` field. If the `inline` field is `Inline::Yes` then `had_parse_error` must be `Ok(())`.
This commit moves the `had_parse_error` field into the `Inline::No` variant. This makes it impossible to create the nonsensical combination of `inline == Inline::Yes` and `had_parse_error = Err(_)`.
r? ```@Urgau```
`ModKind::Loaded` has an `inline` field and a `had_parse_error` field.
If the `inline` field is `Inline::Yes` then `had_parse_error` must be
`Ok(())`.
This commit moves the `had_parse_error` field into the `Inline::No`
variant. This makes it impossible to create the nonsensical combination
of `inline == Inline::Yes` and `had_parse_error = Err(_)`.
In Fedora, when we built rustc with PGO on ppc64le, we started failing
the test `issue-74564-if-expr-stack-overflow.rs`. This could also be
reproduced on other arches by setting a smaller `RUST_MIN_STACK`, so
it's probably just unlucky that ppc64le PGO created a large stack frame
somewhere in this recursion path. Adding an `ensure_sufficient_stack`
solves the stack overflow.
Historically, that test and its fix were added in rust-lang/rust#74708,
which was also an `ensure_sufficient_stack` in this area of code at the
time. However, the refactor in rust-lang/rust#92573 basically left that
to the general `MutVisitor`, and then rust-lang/rust#142240 removed even
that ensure call. It may be luck that our tier-1 tested targets did not
regress the original issue across those refactors.
Add infrastructure to apply a derive macro to arguments, consuming and
returning a `TokenTree` only.
Handle `SyntaxExtensionKind::MacroRules` when expanding a derive, if the
macro's kinds support derive.
Add tests covering various cases of `macro_rules` derives.
Note that due to a pre-existing FIXME in `expand.rs`, derives are
re-queued and some errors get emitted twice. Duplicate diagnostic
suppression makes them not visible, but the FIXME should still get
fixed.
Handle macros with multiple kinds, and improve errors
(I recommend reviewing this commit-by-commit.)
Switch to a bitflags `MacroKinds` to support macros with more than one kind
Review everything that uses `MacroKind`, and switch anything that could refer to more than one kind to use `MacroKinds`.
Add a new `SyntaxExtensionKind::MacroRules` for `macro_rules!` macros, using the concrete `MacroRulesMacroExpander` type, and have it track which kinds it can handle. Eliminate the separate optional `attr_ext`, now that a `SyntaxExtension` can handle multiple macro kinds.
This also avoids the need to downcast when calling methods on `MacroRulesMacroExpander`, such as `get_unused_rule`.
Integrate macro kind checking into name resolution's `sub_namespace_match`, so that we only find a macro if it's the right type, and eliminate the special-case hack for attributes.
This allows detecting and report macro kind mismatches early, and more precisely, improving various error messages. In particular, this eliminates the case in `failed_to_match_macro` to check for a function-like invocation of a macro with no function-like rules.
Instead, macro kind mismatches now result in an unresolved macro, and we detect this case in `unresolved_macro_suggestions`, which now carefully distinguishes between a kind mismatch and other errors.
This also handles cases of forward-referenced attributes and cyclic attributes.
----
In this PR, I've minimally fixed up `rustdoc` so that it compiles and passes tests. This is just the minimal necessary fixes to handle the switch to `MacroKinds`, and it only works for macros that don't actually have multiple kinds. This will panic (with a `todo!`) if it encounters a macro with multiple kinds.
rustdoc needs further fixes to handle macros with multiple kinds, and to handle attributes and derive macros that aren't proc macros. I'd appreciate some help from a rustdoc expert on that.
----
r? ````````@petrochenkov````````
Review everything that uses `MacroKind`, and switch anything that could
refer to more than one kind to use `MacroKinds`.
Add a new `SyntaxExtensionKind::MacroRules` for `macro_rules!` macros,
using the concrete `MacroRulesMacroExpander` type, and have it track
which kinds it can handle. Eliminate the separate optional `attr_ext`,
now that a `SyntaxExtension` can handle multiple macro kinds.
This also avoids the need to downcast when calling methods on
`MacroRulesMacroExpander`, such as `get_unused_rule`.
Integrate macro kind checking into name resolution's
`sub_namespace_match`, so that we only find a macro if it's the right
type, and eliminate the special-case hack for attributes.
Use a visitor to collect *all* items (including those nested) that were stripped behind a `cfg` condition.
```
error[E0425]: cannot find function `f` in this scope
--> $DIR/nested-cfg-attrs.rs:4:13
|
LL | fn main() { f() }
| ^ not found in this scope
|
note: found an item that was configured out
--> $DIR/nested-cfg-attrs.rs:2:4
|
LL | fn f() {}
| ^
note: the item is gated here
--> $DIR/nested-cfg-attrs.rs:1:35
|
LL | #[cfg_attr(all(), cfg_attr(all(), cfg(FALSE)))]
| ^^^^^^^^^^
```
completely deduplicate `Visitor` and `MutVisitor`
r? oli-obk
This closesrust-lang/rust#127615.
### Discussion
> * Give every `MutVisitor::visit_*` method a corresponding `flat_map_*` method.
Not every AST node exists in a location where they can be mapped to multiple instances of themselves. Not every AST node exists in a location where they can be removed from existence (e.g. `filter_map_expr`). I don't think this is doable.
> * Give every `MutVisitor::visit_*` method a corresponding `Visitor` method and vice versa
The only three remaining method-level asymmetries after this PR are `visit_stmt` and `visit_nested_use_tree` (only on `Visitor`) and `visit_span` (only on `MutVisitor`).
`visit_stmt` doesn't seem applicable to `MutVisitor` because `walk_flat_map_stmt_kind` will ask `flat_map_item` / `filter_map_expr` to potentially turn a single `Stmt` to multiple based on what a visitor wants. So only using `flat_map_stmt` seems appropriate.
`visit_nested_use_tree` is used for `rustc_resolve` to track stuff. Not useful for `MutVisitor` for now.
`visit_span` is currently not used for `MutVisitor` already, it was just kept in case we want to revive rust-lang/rust#127241. cc `@cjgillot` maybe we could remove for now and re-insert later if we find a use-case? It does involve some extra effort to maintain.
* Remaining FIXMEs
`visit_lifetime` has an extra param for `Visitor` that's not in `MutVisitor`. This is again something only used by `rustc_resolve`. I think we can keep that symmetry for now.
It collects data about macro expansions and prints them in a table after
expansion finishes. It's very useful for detecting macro bloat,
especially for proc macros.
Details:
- It measures code snippets by pretty-printing them and then measuring
lines and bytes. This required a bunch of additional pretty-printing
plumbing, in `rustc_ast_pretty` and `rustc_expand`.
- The measurement is done in `MacroExpander::expand_invoc`.
- The measurements are stored in `ExtCtxt::macro_stats`.
remove `visit_clobber` and move `DummyAstNode` to `rustc_expand`
`visit_clobber` is not really useful except for one niche purpose
involving generic code. We should just use the replace logic where we
can.
Remove `token::{Open,Close}Delim`
By replacing them with `{Open,Close}{Param,Brace,Bracket,Invisible}`.
PR #137902 made `ast::TokenKind` more like `lexer::TokenKind` by
replacing the compound `BinOp{,Eq}(BinOpToken)` variants with fieldless
variants `Plus`, `Minus`, `Star`, etc. This commit does a similar thing
with delimiters. It also makes `ast::TokenKind` more similar to
`parser::TokenType`.
This requires a few new methods:
- `TokenKind::is_{,open_,close_}delim()` replace various kinds of
pattern matches.
- `Delimiter::as_{open,close}_token_kind` are used to convert
`Delimiter` values to `TokenKind`.
Despite these additions, it's a net reduction in lines of code. This is
because e.g. `token::OpenParen` is so much shorter than
`token::OpenDelim(Delimiter::Parenthesis)` that many multi-line forms
reduce to single line forms. And many places where the number of lines
doesn't change are still easier to read, just because the names are
shorter, e.g.:
```
- } else if self.token != token::CloseDelim(Delimiter::Brace) {
+ } else if self.token != token::CloseBrace {
```
r? `@petrochenkov`
By replacing them with `{Open,Close}{Param,Brace,Bracket,Invisible}`.
PR #137902 made `ast::TokenKind` more like `lexer::TokenKind` by
replacing the compound `BinOp{,Eq}(BinOpToken)` variants with fieldless
variants `Plus`, `Minus`, `Star`, etc. This commit does a similar thing
with delimiters. It also makes `ast::TokenKind` more similar to
`parser::TokenType`.
This requires a few new methods:
- `TokenKind::is_{,open_,close_}delim()` replace various kinds of
pattern matches.
- `Delimiter::as_{open,close}_token_kind` are used to convert
`Delimiter` values to `TokenKind`.
Despite these additions, it's a net reduction in lines of code. This is
because e.g. `token::OpenParen` is so much shorter than
`token::OpenDelim(Delimiter::Parenthesis)` that many multi-line forms
reduce to single line forms. And many places where the number of lines
doesn't change are still easier to read, just because the names are
shorter, e.g.:
```
- } else if self.token != token::CloseDelim(Delimiter::Brace) {
+ } else if self.token != token::CloseBrace {
```
I'm removing empty identifiers everywhere, because in practice they
always mean "no identifier" rather than "empty identifier". (An empty
identifier is impossible.) It's better to use `Option` to mean "no
identifier" because you then can't forget about the "no identifier"
possibility.
Some specifics:
- When testing an attribute for a single name, the commit uses the
`has_name` method.
- When testing an attribute for multiple names, the commit uses the new
`has_any_name` method.
- When using `match` on an attribute, the match arms now have `Some` on
them.
In the tests, we now avoid printing empty identifiers by not printing
the identifier in the `error:` line at all, instead letting the carets
point out the problem.
It bugs me when variables of type `Ident` are called `name`. It leads to
silly things like `name.name`. `Ident` variables should be called
`ident`, and `name` should be used for variables of type `Symbol`.
This commit improves things by by doing `s/name/ident/` on a bunch of
`Ident` variables. Not all of them, but a decent chunk.
`ast::Item` has an `ident` field.
- It's always non-empty for these item kinds: `ExternCrate`, `Static`,
`Const`, `Fn`, `Mod`, `TyAlias`, `Enum`, `Struct`, `Union`,
`Trait`, `TraitAlias`, `MacroDef`, `Delegation`.
- It's always empty for these item kinds: `Use`, `ForeignMod`,
`GlobalAsm`, `Impl`, `MacCall`, `DelegationMac`.
There is a similar story for `AssocItemKind` and `ForeignItemKind`.
Some sites that handle items check for an empty ident, some don't. This
is a very C-like way of doing things, but this is Rust, we have sum
types, we can do this properly and never forget to check for the
exceptional case and never YOLO possibly empty identifiers (or possibly
dummy spans) around and hope that things will work out.
The commit is large but it's mostly obvious plumbing work. Some notable
things.
- `ast::Item` got 8 bytes bigger. This could be avoided by boxing the
fields within some of the `ast::ItemKind` variants (specifically:
`Struct`, `Union`, `Enum`). I might do that in a follow-up; this
commit is big enough already.
- For the visitors: `FnKind` no longer needs an `ident` field because
the `Fn` within how has one.
- In the parser, the `ItemInfo` typedef is no longer needed. It was used
in various places to return an `Ident` alongside an `ItemKind`, but
now the `Ident` (if present) is within the `ItemKind`.
- In a few places I renamed identifier variables called `name` (or
`foo_name`) as `ident` (or `foo_ident`), to better match the type, and
because `name` is normally used for `Symbol`s. It's confusing to see
something like `foo_name.name`.
