The parser repeatedly invokes the `parse` function, constructing a
one-entry vector, and assuming that the return value will be a one-entry
vector. Add a helper for that case. This will simplify adding additional
callers, and put all the logic in one place to allow potential future
simplification of the one-TT case.
setup typos check in CI
This allows to check typos in CI, currently for compiler only (to reduce commit size with fixes). With current setup, exclude list is quite short, so it worth trying?
Also includes commits with actual typo fixes.
MCP: https://github.com/rust-lang/compiler-team/issues/817
typos check currently turned for:
* ./compiler
* ./library
* ./src/bootstrap
* ./src/librustdoc
After merging, PRs which enables checks for other crates (tools) can be implemented too.
Found typos will **not break** other jobs immediately: (tests, building compiler for perf run). Job will be marked as red on completion in ~ 20 secs, so you will not forget to fix it whenever you want, before merging pr.
Check typos: `python x.py test tidy --extra-checks=spellcheck`
Apply typo fixes: `python x.py test tidy --extra-checks=spellcheck:fix` (in case if there only 1 suggestion of each typo)
Current fail in this pr is expected and shows how typo errors emitted. Commit with error will be removed after r+.
Rewrite `macro_rules!` parser to not use the MBE engine itself
The `macro_rules!` parser was written to match the series of rules using the macros-by-example (MBE) engine and a hand-written equivalent of the left-hand side of a MBE macro. This was complex to read, difficult to extend, and produced confusing error messages. Because it was using the MBE engine, any parse failure would be reported as if some macro was being applied to the `macro_rules!` invocation itself; for instance, errors would talk about "macro invocation", "macro arguments", and "macro call", when they were actually about the macro *definition*.
And in practice, the `macro_rules!` parser only used the MBE engine to extract the left-hand side and right-hand side of each rule as a token tree, and then parsed the rest using a separate parser.
Rewrite it to parse the series of rules using a simple loop, instead. This makes it more extensible in the future, and improves error messages. For instance, omitting a semicolon between rules will result in "expected `;`" and "unexpected token", rather than the confusing "no rules expected this token in macro call".
This work was greatly aided by pair programming with Vincenzo Palazzo (`@vincenzopalazzo)` and Eric Holk (`@eholk).`
For review, I recommend reading the two commits separately.
Move the `concat` implementation to a separate function so it is easier
to work on. Other metavariable expressions are already split this way.
This is a non-functional change.
More diagnostic structs related to metavariable expressions will be
introduced. Introduce the abbreviation "mve" which is reasonably
unambiguous (`rg Mve` and `rg '(\b|_|-)mve(\b|_|-)'` return no results
outside of a Thumb target feature) and use it for these diagnostic
types. A new module is also created.
It's like `Symbol` but for byte strings. The interner is now used for
both `Symbol` and `ByteSymbol`. E.g. if you intern `"dog"` and `b"dog"`
you'll get a `Symbol` and a `ByteSymbol` with the same index and the
characters will only be stored once.
The motivation for this is to eliminate the `Arc`s in `ast::LitKind`, to
make `ast::LitKind` impl `Copy`, and to avoid the need to arena-allocate
`ast::LitKind` in HIR. The latter change reduces peak memory by a
non-trivial amount on literal-heavy benchmarks such as `deep-vector` and
`tuple-stress`.
`Encoder`, `Decoder`, `SpanEncoder`, and `SpanDecoder` all get some
changes so that they can handle normal strings and byte strings.
This change does slow down compilation of programs that use
`include_bytes!` on large files, because the contents of those files are
now interned (hashed). This makes `include_bytes!` more similar to
`include_str!`, though `include_bytes!` contents still aren't escaped,
and hashing is still much cheaper than escaping.
The `macro_rules!` parser was written to match the series of rules using
the macros-by-example (MBE) engine and a hand-written equivalent of the
left-hand side of a MBE macro. This was complex to read, difficult to
extend, and produced confusing error messages. Because it was using the
MBE engine, any parse failure would be reported as if some macro was
being applied to the `macro_rules!` invocation itself; for instance,
errors would talk about "macro invocation", "macro arguments", and
"macro call", when they were actually about the macro *definition*.
And in practice, the `macro_rules!` parser only used the MBE engine to
extract the left-hand side and right-hand side of each rule as a token
tree, and then parsed the rest using a separate parser.
Rewrite it to parse the series of rules using a simple loop, instead.
This makes it more extensible in the future, and improves error
messages. For instance, omitting a semicolon between rules will result
in "expected `;`" and "unexpected token", rather than the confusing "no
rules expected this token in macro call".
This work was greatly aided by pair programming with Vincenzo Palazzo
and Eric Holk.
Tweak `-Zmacro-stats` measurement.
It currently reports net size, i.e. size(output) - size(input). After some use I think this is sub-optimal, and it's better to just report size(output). Because for derive macros the input size is always 1, and for attribute macros it's almost always 1.
r? ```@petrochenkov```
Don't include current rustc version string in feature removed help
The version string is difficult to properly normalize out, and removing it isn't a huge deal (the user can query version info easily through `rustc --version` or `cargo --version`).
The normalization options were all non-ideal (see https://github.com/rust-lang/rust/pull/142940#issuecomment-2998518450):
- Per-test version string normalization is nasty to maintain, and we need to maintain `n` copies of it. See rust-lang/rust#142930 where the regex wasn't robust against different release channels.
- Centralized compiletest normalization (with a directive opt-out) is also not ideal, because `cfg(version(..))` tests can't have those accidentally normalized out (and you'd have to remember to opt-out).
r? `@workingjubilee` (discussed in rust-lang/rust#142940)
mbe: Clean up code with non-optional `NonterminalKind`
Since [rust-lang/rust#128425], the fragment specifier is unconditionally required in all
editions. This means `NonTerminalKind` no longer needs to be optional,
as we can reject this code during the expansion of `macro_rules!` rather
than handling it throughout the code. Do this cleanup here.
[rust-lang/rust#128425]: https://github.com/rust-lang/rust/pull/128425
The version string is difficult to properly normalize out, and removing
it isn't a huge deal (the user can query version info easily through
`rustc --version` or `cargo --version`).
The normalization options were all non-ideal:
- Per-test version string normalization is nasty to maintain, and we
need to maintain `n` copies of it.
- Centralized compiletest normalization (with a directive opt-out) is
also not ideal, because `cfg(version(..))` tests can't have those
accidentally normalized out (and you'd have to remember to opt-out).
Since [1], the fragment specifier is unconditionally required in all
editions. This means `NonTerminalKind` no longer needs to be optional,
as we can reject this code during the expansion of `macro_rules!` rather
than handling it throughout the code. Do this cleanup here.
[1]: https://github.com/rust-lang/rust/pull/128425
It currently reports net size, i.e. size(output) - size(input). After
some use I think this is sub-optimal, and it's better to just report
size(output). Because for derive macros the input size is always 1, and
for attribute macros it's almost always 1.
completely deduplicate `Visitor` and `MutVisitor`
r? oli-obk
This closesrust-lang/rust#127615.
### Discussion
> * Give every `MutVisitor::visit_*` method a corresponding `flat_map_*` method.
Not every AST node exists in a location where they can be mapped to multiple instances of themselves. Not every AST node exists in a location where they can be removed from existence (e.g. `filter_map_expr`). I don't think this is doable.
> * Give every `MutVisitor::visit_*` method a corresponding `Visitor` method and vice versa
The only three remaining method-level asymmetries after this PR are `visit_stmt` and `visit_nested_use_tree` (only on `Visitor`) and `visit_span` (only on `MutVisitor`).
`visit_stmt` doesn't seem applicable to `MutVisitor` because `walk_flat_map_stmt_kind` will ask `flat_map_item` / `filter_map_expr` to potentially turn a single `Stmt` to multiple based on what a visitor wants. So only using `flat_map_stmt` seems appropriate.
`visit_nested_use_tree` is used for `rustc_resolve` to track stuff. Not useful for `MutVisitor` for now.
`visit_span` is currently not used for `MutVisitor` already, it was just kept in case we want to revive rust-lang/rust#127241. cc `@cjgillot` maybe we could remove for now and re-insert later if we find a use-case? It does involve some extra effort to maintain.
* Remaining FIXMEs
`visit_lifetime` has an extra param for `Visitor` that's not in `MutVisitor`. This is again something only used by `rustc_resolve`. I think we can keep that symmetry for now.
Introduce `MacroTcbCtx` that holds everything relevant to transcription.
This allows for the following changes:
* Split `transcribe_sequence` and `transcribe_metavar` out of the
heavily nested `transcribe`
* Split `metavar_expr_concat` out of `transcribe_metavar_expr`
This is a nonfunctional change.
It collects data about macro expansions and prints them in a table after
expansion finishes. It's very useful for detecting macro bloat,
especially for proc macros.
Details:
- It measures code snippets by pretty-printing them and then measuring
lines and bytes. This required a bunch of additional pretty-printing
plumbing, in `rustc_ast_pretty` and `rustc_expand`.
- The measurement is done in `MacroExpander::expand_invoc`.
- The measurements are stored in `ExtCtxt::macro_stats`.
remove `visit_clobber` and move `DummyAstNode` to `rustc_expand`
`visit_clobber` is not really useful except for one niche purpose
involving generic code. We should just use the replace logic where we
can.
