Handles BOM on input files by removing.

Fixed for UTF-8 input.
Added D test.

1 files changed, 47 insertions, 0 deletions

diff --git a/test/examples/x.d.styled b/test/examples/x.d.styled
new file mode 100644
index 000000000..ef90ffb6b
--- /dev/null
+++ b/test/examples/x.d.styled
@@ -0,0 +1,47 @@
+{14}${0}
+{2}// /++ +/ doccomments are not yet supported
+{1}/* */{0}
+{3}/** */{0}
+{15}/// drdr
+{4}/+ /+ +/ +/{0}
+{2}//keyword test
+{6}keyword1{0}
+{7}keyword2{0}
+{9}keyword4{0}
+{8}keyword5{0}
+{20}keyword6{0}
+{21}keyword7{0}
+{2}//unicode identifier test
+{14}вапёasdÓΘΣαԷԸՑהכ拉麺とひシマイ단결을{0}
+{2}//strings test
+{11}'s
+'
+{14}w{12}'s'{14}w{0}
+{10}"multiline
+			string"w{0}
+{14}e{10}"zz"{14}e{0}
+{19}r"asd\"{14}e{0}
+{19}r"multiline
+			string"c{0}
+{14}r{18}`asd\`{14}e{0}
+{18}`multiline
+			string`d{0}
+{19}x"023 abc"{14}e{0}
+{19}x"023
+	abc"w{0}
+{2}//numbers test
+{14}a{13}[{5}3{13}..{5}4{13}]={5}3{0}
+{5}2.stringof{0}
+{5}2.0{13}.{14}stringof{0}
+{5}2.{0}
+{5}2.2e+2{0}
+{5}2.2e-2{0}
+{5}.2e+2{0}
+{5}.2{0}
+{5}2e+2{0}
+{5}0x2e{13}+{5}2{0}
+{5}0x2ep+10{0}
+{13},{5}.2{13}.{14}stringof{13},{0}
+
+{14}end{0}
+