Introduce rustc_lexer

The idea here is to make a reusable library out of the existing
rust-lexer, by separating out pure lexing and rustc-specific concerns,
like spans, error reporting an interning.

So, rustc_lexer operates directly on `&str`, produces simple tokens
which are a pair of type-tag and a bit of original text, and does not
report errors, instead storing them as flags on the token.
This commit is contained in:
Aleksey Kladov
2019-05-06 11:53:40 +03:00
parent 95b1fe560d
commit 395ee0b79f
15 changed files with 1337 additions and 1261 deletions

View File

@@ -53,10 +53,10 @@ LL | 0o;
| ^^
error: expected at least one digit in exponent
--> $DIR/lex-bad-numeric-literals.rs:12:8
--> $DIR/lex-bad-numeric-literals.rs:12:5
|
LL | 1e+;
| ^
| ^^^
error: hexadecimal float literal is not supported
--> $DIR/lex-bad-numeric-literals.rs:13:5

View File

@@ -1,8 +1,8 @@
error: unterminated raw string
--> $DIR/raw-byte-string-eof.rs:2:6
--> $DIR/raw-byte-string-eof.rs:2:5
|
LL | br##"a"#;
| ^ unterminated raw string
| ^ unterminated raw string
|
= note: this raw string should be terminated with `"##`

View File

@@ -11,10 +11,10 @@ LL | br"é";
LL | br"é";
| ^
error: found invalid character; only `#` is allowed in raw string delimitation: ~
error: found invalid character; only `#` is allowed in raw string delimitation: ~
--> $DIR/raw-byte-string-literals.rs:6:5
|
LL | br##~"a"~##;
LL | br##~"a"~##;
| ^^^^^
error: aborting due to 3 previous errors

View File

@@ -2,7 +2,7 @@ error: found invalid character; only `#` is allowed in raw string delimitation:
--> $DIR/raw-str-delim.rs:2:5
|
LL | r#~"#"~#
| ^^
| ^^^
error: aborting due to previous error