54a172f3b5
Add a dump-html-tokens utility that prints HTML tokenizer output in a stable text format, along with focused tokenizer input and expectation files. Include silent and repeated-iteration modes so the utility can also be used for local tokenizer throughput checks without changing the dumped token format.
10 lines
222 B
Text
10 lines
222 B
Text
Comment " simple comment " @0:0-0:23
|
|
Character U+000A @1:0
|
|
Comment "" @1:0-1:7
|
|
Character U+000A @2:0
|
|
Comment "-" @2:0-2:8
|
|
Character U+000A @3:0
|
|
Comment " multi
|
|
line comment " @3:0-4:16
|
|
Character U+000A @5:0
|
|
EndOfFile @5:0
|