Overhaul token collection.

This commit does the following.

- Renames `collect_tokens_trailing_token` as `collect_tokens`, because
  (a) it's annoying long, and (b) the `_trailing_token` bit is less
  accurate now that its types have changed.

- In `collect_tokens`, adds a `Option<CollectPos>` argument and a
  `UsePreAttrPos` in the return type of `f`. These are used in
  `parse_expr_force_collect` (for vanilla expressions) and in
  `parse_stmt_without_recovery` (for two different cases of expression
  statements). Together these ensure are enough to fix all the problems
  with token collection and assoc expressions. The changes to the
  `stringify.rs` test demonstrate some of these.

- Adds a new test. The code in this test was causing an assertion
  failure prior to this commit, due to an invalid `NodeRange`.

The extra complexity is annoying, but necessary to fix the existing
problems.

compiler/rustc_parse/src/parser/mod.rs

@@ -14,7 +14,7 @@ use std::assert_matches::debug_assert_matches;
 use std::ops::Range;
 use std::{fmt, mem, slice};
 
-use attr_wrapper::AttrWrapper;
+use attr_wrapper::{AttrWrapper, UsePreAttrPos};
 pub use diagnostics::AttemptLocalParseRecovery;
 pub(crate) use expr::ForbiddenLetReason;
 pub(crate) use item::FnParseMode;
@@ -254,7 +254,7 @@ enum Capturing {
     Yes,
 }
 
-// This state is used by `Parser::collect_tokens_trailing_token`.
+// This state is used by `Parser::collect_tokens`.
 #[derive(Clone, Debug)]
 struct CaptureState {
     capturing: Capturing,
@@ -466,8 +466,8 @@ impl<'a> Parser<'a> {
         parser.bump();
 
         // Change this from 1 back to 0 after the bump. This eases debugging of
-        // `Parser::collect_tokens_trailing_token` nicer because it makes the
-        // token positions 0-indexed which is nicer than 1-indexed.
+        // `Parser::collect_tokens` because 0-indexed token positions are nicer
+        // than 1-indexed token positions.
         parser.num_bump_calls = 0;
 
         parser
@@ -1553,11 +1553,9 @@ impl<'a> Parser<'a> {
     ) -> PResult<'a, R> {
         // The only reason to call `collect_tokens_no_attrs` is if you want tokens, so use
         // `ForceCollect::Yes`
-        self.collect_tokens_trailing_token(
-            AttrWrapper::empty(),
-            ForceCollect::Yes,
-            |this, _attrs| Ok((f(this)?, Trailing::No)),
-        )
+        self.collect_tokens(None, AttrWrapper::empty(), ForceCollect::Yes, |this, _attrs| {
+            Ok((f(this)?, Trailing::No, UsePreAttrPos::No))
+        })
     }
 
     /// `::{` or `::*`