Refactor the parser to consume token trees
This is groundwork for efficiently parsing attribute proc macro invocations, bang macro invocations, and `TokenStream`-based attributes and fragment matchers.
This improves parsing performance by 8-15% and expansion performance by 0-5% on a sampling of the compiler's crates.
r? @nrc
Implement `#[proc_macro_attribute]`
This implements `#[proc_macro_attribute]` as described in https://github.com/rust-lang/rfcs/pull/1566
The following major (hopefully non-breaking) changes are included:
* Refactor `proc_macro::TokenStream` to use `syntax::tokenstream::TokenStream`.
* `proc_macro::tokenstream::TokenStream` no longer emits newlines between items, this can be trivially restored if desired
* `proc_macro::TokenStream::from_str` does not try to parse an item anymore, moved to `impl MultiItemModifier for CustomDerive` with more informative error message
* Implement `#[proc_macro_attribute]`, which expects functions of the kind `fn(TokenStream, TokenStream) -> TokenStream`
* Reactivated `#![feature(proc_macro)]` and gated `#[proc_macro_attribute]` under it
* `#![feature(proc_macro)]` and `#![feature(custom_attribute)]` are mutually exclusive
* adding `#![feature(proc_macro)]` makes the expansion pass assume that any attributes that are not built-in, or introduced by existing syntax extensions, are proc-macro attributes
* Fix `feature_gate::find_lang_feature_issue()` to not use `unwrap()`
* This change wasn't necessary for this PR, but it helped debugging a problem where I was using the wrong feature string.
* Move "completed feature gate checking" pass to after "name resolution" pass
* This was necessary for proper feature-gating of `#[proc_macro_attribute]` invocations when the `proc_macro` feature flag isn't set.
Prototype/Litmus Test: [Implementation](https://github.com/abonander/anterofit/blob/proc_macro/service-attr/src/lib.rs#L13) -- [Usage](https://github.com/abonander/anterofit/blob/proc_macro/service-attr/examples/post_service.rs#L35)
* Add support for `#[proc_macro]`
* Reactivate `proc_macro` feature and gate `#[proc_macro_attribute]` under it
* Have `#![feature(proc_macro)]` imply `#![feature(use_extern_macros)]`,
error on legacy import of proc macros via `#[macro_use]`
This marks the pushpop_unsafe feature as removed inside the feature_gate.
It was added in commit 1829fa5199 and then
removed again in commit d399098fd8 .
Seems that the second commit forgot to mark it as removed in feature_gate.rs.
This enables us to remove another element from the whitelist of non gate
tested unstable lang features (issue #39059).
syntax: enable attributes and cfg on struct fields
This enables conditional compilation of field initializers in a struct literal, simplifying construction of structs whose fields are themselves conditionally present. For example, the intializer for the constant in the following becomes legal, and has the intuitive effect:
```rust
struct Foo {
#[cfg(unix)]
bar: (),
}
const FOO: Foo = Foo {
#[cfg(unix)]
bar: (),
};
```
It's not clear to me whether this calls for the full RFC process, but the implementation was simple enough that I figured I'd begin the conversation with code.
This commit stabilizes the `proc_macro` and `proc_macro_lib` features in the
compiler to stabilize the "Macros 1.1" feature of the language. Many more
details can be found on the tracking issue, #35900.
Closes#35900
Don't spin expanding stmt macros.
If we can't make progress when parsing a macro expansion as a statement then we should just bail.
This alleviates the symptoms shown in e.g. #37113 and #37234 but it doesn't fix the problem that parsing invalid enum bodies (and others) leaves the parser in a crappy state.
I'm not sold on this strategy (checking `tokens_consumed`), so if anyone has a better idea, I'm all ears!
If we can't make progress when parsing a macro expansion as a statement
then we should just bail.
This alleviates the symptoms shown in e.g. #37113 but it doesn't fix the
problem that parsing invalid enum bodies (and others) leaves the parser
in a crappy state.
This commit blanket renames the `rustc_macro` infrastructure to `proc_macro`,
which reflects the general consensus of #35900. A follow up PR to Cargo will be
required to purge the `rustc-macro` name as well.
Allow more non-inline modules in blocks
Currently, non-inline modules without a `#[path]` attribute are not allowed in blocks.
This PR allows non-inline modules that have an ancestor module with a `#[path]` attribute, provided there is not a nearer ancestor block.
For example,
```rust
fn main() {
#[path = "..."] mod foo {
mod bar; //< allowed by this PR
fn f() {
mod bar; //< still an error
}
}
}
```
Fixes#36772.
r? @nikomatsakis
