rustc: Stabilize options for pipelined compilation
This commit stabilizes options in the compiler necessary for Cargo to
enable "pipelined compilation" by default. The concept of pipelined
compilation, how it's implemented, and what it means for rustc are
documented in #60988. This PR is coupled with a PR against Cargo
(rust-lang/cargo#7143) which updates Cargo's support for pipelined
compliation to rustc, and also enables support by default in Cargo.
(note that the Cargo PR cannot land until this one against rustc lands).
The technical changes performed here were to stabilize the functionality
proposed in #60419 and #60987, the underlying pieces to enable pipelined
compilation support in Cargo. The issues have had some discussion during
stabilization, but the newly stabilized surface area here is:
* A new `--json` flag was added to the compiler.
* The `--json` flag can be passed multiple times.
* The value of the `--json` flag is a comma-separated list of
directives.
* The `--json` flag cannot be combined with `--color`
* The `--json` flag must be combined with `--error-format=json`
* The acceptable list of directives to `--json` are:
* `diagnostic-short` - the `rendered` field of diagnostics will have a
"short" rendering matching `--error-format=short`
* `diagnostic-rendered-ansi` - the `rendered` field of diagnostics
will be colorized with ansi color codes embedded in the string field
* `artifacts` - JSON blobs will be emitted for artifacts being emitted
by the compiler
The unstable `-Z emit-artifact-notifications` and `--json-rendered`
flags have also been removed during this commit as well.
Closes#60419Closes#60987Closes#60988
This commit stabilizes options in the compiler necessary for Cargo to
enable "pipelined compilation" by default. The concept of pipelined
compilation, how it's implemented, and what it means for rustc are
documented in #60988. This PR is coupled with a PR against Cargo
(rust-lang/cargo#7143) which updates Cargo's support for pipelined
compliation to rustc, and also enables support by default in Cargo.
(note that the Cargo PR cannot land until this one against rustc lands).
The technical changes performed here were to stabilize the functionality
proposed in #60419 and #60987, the underlying pieces to enable pipelined
compilation support in Cargo. The issues have had some discussion during
stabilization, but the newly stabilized surface area here is:
* A new `--json` flag was added to the compiler.
* The `--json` flag can be passed multiple times.
* The value of the `--json` flag is a comma-separated list of
directives.
* The `--json` flag cannot be combined with `--color`
* The `--json` flag must be combined with `--error-format=json`
* The acceptable list of directives to `--json` are:
* `diagnostic-short` - the `rendered` field of diagnostics will have a
"short" rendering matching `--error-format=short`
* `diagnostic-rendered-ansi` - the `rendered` field of diagnostics
will be colorized with ansi color codes embedded in the string field
* `artifacts` - JSON blobs will be emitted for artifacts being emitted
by the compiler
The unstable `-Z emit-artifact-notifications` and `--json-rendered`
flags have also been removed during this commit as well.
Closes#60419Closes#60987Closes#60988
Implement another internal lints
cc #49509
This adds ~~two~~ one internal lint~~s~~:
1. LINT_PASS_IMPL_WITHOUT_MACRO: Make sure, that the `{declare,impl}_lint_pass` macro is used to implement lint passes. cc #59669
2. ~~USAGE_OF_TYCTXT_AND_SPAN_ARGS: item 2 on the list in #49509~~
~~With 2. I wasn't sure, if this lint should be applied everywhere. That means a careful review of 0955835 would be great. Also 73fb9b4 allows this lint on some functions. Should I also apply this lint there?~~
TODO (not directly relevant for review):
- [ ] https://github.com/rust-lang/rust/pull/59316#discussion_r280186517 (not sure yet, if this works or how to query for `rustc_private`, since it's not in [`Features`](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/feature_gate/struct.Features.html) 🤔 cc @eddyb)
- [x] https://github.com/rust-lang/rust/pull/61735#discussion_r292389870
- [x] Check explicitly for the `{declare,impl}_lint_pass!` macros
r? @oli-obk
* Move fs::create_dir_all calls into DocFS to provide a clean
extension point if async extension there is needed.
* Convert callsites of create_dir_all to ensure_dir to reduce syscalls.
* Convert fs::write usage to DocFS.write
(which also removes a lot of try_err! usage for easier reading)
* Convert File::create calls to use Vec buffers and then DocFS.write
in order to consistently reduce syscalls as well, make
deferring to threads cleaner and avoid leaving dangling content if
writing to existing files....
* Convert OpenOptions usage similarly - I could find no discussion on
the use of create_new for that one output file vs all the other
files render creates, if link redirection attacks are a concern
DocFS will provide a good central point to introduce systematic
create_new usage. (fs::write/File::create is vulnerable to link
redirection attacks).
* DocFS::write defers to rayon for IO on Windows producing a modest
speedup: before this patch on my development workstation:
$ time cargo +mystg1 doc -p winapi:0.3.7
Documenting winapi v0.3.7
Finished dev [unoptimized + debuginfo] target(s) in 6m 11s
real 6m11.734s
Afterwards:
$ time cargo +mystg1 doc -p winapi:0.3.7
Compiling winapi v0.3.7
Documenting winapi v0.3.7
Finished dev [unoptimized + debuginfo] target(s) in 49.53s
real 0m49.643s
I haven't measured how much time is in the compilation logic vs in the
IO and outputting etc, but this takes it from frustating to tolerable
for me, at least for now.
rustdoc: add option to calculate "documentation coverage"
This PR adds a new flag to rustdoc, `--show-coverage`. When passed, this flag will make rustdoc count the number of items in a crate with documentation instead of generating docs. This count will be output as a table of each file in the crate, like this (when run on my crate `egg-mode`):
```
+-------------------------------------+------------+------------+------------+
| File | Documented | Total | Percentage |
+-------------------------------------+------------+------------+------------+
| src/auth.rs | 16 | 16 | 100.0% |
| src/common/mod.rs | 1 | 1 | 100.0% |
| src/common/response.rs | 9 | 9 | 100.0% |
| src/cursor.rs | 24 | 24 | 100.0% |
| src/direct/fun.rs | 6 | 6 | 100.0% |
| src/direct/mod.rs | 41 | 41 | 100.0% |
| src/entities.rs | 50 | 50 | 100.0% |
| src/error.rs | 27 | 27 | 100.0% |
| src/lib.rs | 1 | 1 | 100.0% |
| src/list/fun.rs | 19 | 19 | 100.0% |
| src/list/mod.rs | 22 | 22 | 100.0% |
| src/media/mod.rs | 27 | 27 | 100.0% |
| src/place/fun.rs | 8 | 8 | 100.0% |
| src/place/mod.rs | 35 | 35 | 100.0% |
| src/search.rs | 26 | 26 | 100.0% |
| src/service.rs | 74 | 74 | 100.0% |
| src/stream/mod.rs | 49 | 49 | 100.0% |
| src/tweet/fun.rs | 15 | 15 | 100.0% |
| src/tweet/mod.rs | 73 | 73 | 100.0% |
| src/user/fun.rs | 24 | 24 | 100.0% |
| src/user/mod.rs | 87 | 87 | 100.0% |
+-------------------------------------+------------+------------+------------+
| Total | 634 | 634 | 100.0% |
+-------------------------------------+------------+------------+------------+
```
Trait implementations are not counted because by default they "inherit" the docs from the trait, even though an impl can override those docs. Similarly, inherent impl blocks are not counted at all, because for the majority of cases such docs are not useful. (The usual pattern for inherent impl blocks is to throw all the methods on a type into a single impl block. Any docs you would put on that block would be better served on the type itself.)
In addition, `--show-coverage` can be combined with `--document-private-items` to get the coverage counts for everything in the crate, not just public items.
The coverage calculation is implemented as a late pass and two new sets of passes which strip out most of the work that rustdoc otherwise does when generating docs. The is because after the new pass is executed, rustdoc immediately closes instead of going on to generate documentation.
Many examples of coverage calculations have been included as `rustdoc-ui` tests.
r? @rust-lang/rustdoc
Rustdoc remove old style files
Reopening of #56577 (which I can't seem to reopen...).
I made the flag unstable so with this change, what was blocking the PR is now gone I assume.
rustdoc: overhaul code block lexing errors
Fixes#53919.
This PR moves the reporting of code block lexing errors from rendering time to an early pass, so we can use the compiler's error reporting mechanisms. This dramatically improves the diagnostics in this situation: we now de-emphasize the lexing errors as a note under a warning that has a span and suggestion instead of just emitting errors at the top level.
Additionally, this PR generalizes the markdown -> source span calculation function, which should allow other rustdoc warnings to use better spans in the future.
Last, the PR makes sure that the code block is always emitted in the docs, even if it fails to highlight correctly.
Of note:
- The new pass unfortunately adds another pass over the docs to gather the doc blocks for syntax-checking. I wonder if this could be combined with the pass that looks for testable blocks? I'm not familiar with that code, so I don't know how feasible that is.
- `pulldown_cmark` doesn't make it easy to find the spans of the code blocks, so the code that calculates the spans is a little nasty. It works for all the test cases I threw at it, but I wouldn't be surprised if an edge case would break it. Should have a thorough review.
- This PR worsens the state of #56885, since those certain fatal lexing errors are now emitted before docs get generated at all.
This commit completely removes usage of the `panictry!` macro from
outside libsyntax. The macro causes parse errors to be fatal, so using
it in libsyntax_ext caused parse failures *within* a syntax extension to
be fatal, which is probably not intended.
Furthermore, this commit adds spans to diagnostics emitted by empty
extensions if they were missing, à la #56491.
Add index page argument
@Mark-Simulacrum: I might need some help from you: in bootstrap, I want to add an argument (a new flag added into `rustdoc`) in order to generate the current index directly when `rustdoc` is documenting the `std` lib. However, my change in `bootstrap` didn't do it and I assume it must be moved inside the `Std` struct. But there, I don't see how to pass it to `rustdoc` through `cargo`. Did I miss anything?
r? @QuietMisdreavus
