<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=koi8-r">
<style type="text/css" style="display:none"><!-- p { margin-top: 0px; margin-bottom: 0px; }--></style>
</head>
<body dir="ltr" style="font-size:12pt;color:#000000;background-color:#FFFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;">
<p><span style="color: rgb(33, 33, 33); font-size: 12pt;">>Our tokenizer recognize</span></p>
<div dir="ltr" style="font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<div>
<div style="color: rgb(33, 33, 33);">
<div>
<div dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;"><font face="monospace, monospace">>
</font></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;">> <font face="monospace, monospace">[A-Za-z0-9_.$/\\~=+[]*?\-:!<>]+</font></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;">> </div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;">>as a token. gold uses more complex rules to tokenize. I don't think we need that much complex rules, but there seems to be >room to improve our tokenizer. In particular, I believe we can parse
the Linux's linker script by changing the tokenizer rules as >follows.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;">> </div>
<div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;"><font face="monospace, monospace">> [A-Za-z_.$/\\~=+[]*?\-:!<>][A-Za-z0-9_.$/\\~=+[]*?\-:!<>]*</font></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;">> </div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;">>or</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;">> </div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;"><font face="monospace, monospace">> [0-9]+</font></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;"><font face="monospace, monospace"><br>
</font></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;"><font face="monospace, monospace">After more investigation, that seems will not work so simple.</font></div>
<div><font face="monospace, monospace">Next are possible examples where it will be broken:</font></div>
<div><font face="monospace, monospace">. = 0x1000; (gives tokens "0, x1000")</font></div>
<div><font face="monospace, monospace">. = A*10; (gives "A*10")</font></div>
<div><font face="monospace, monospace">. = 10k; (gives "10, k")</font></div>
<div><font face="monospace, monospace">. = 10*5; (gives "10, *5"</font></div>
<div><font face="monospace, monospace"><br>
</font></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;"><font face="monospace, monospace">"[0-9]+" could be "[0-9][kmhKMHx0-9]*"</font></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;"><font face="monospace, monospace">but for "10*5" that anyways gives "10" and "*5" tokens.</font></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;"><font face="monospace, monospace">And I do not think we can involve some handling of operators,</font></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;"><font face="monospace, monospace">as <span style="color: rgb(33, 33, 33); font-family: monospace, monospace; font-size: 16px; background-color: rgb(255, 255, 255);">its hard to assume some context</span>
on tokenizing step.</font></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;"><font face="monospace, monospace">We do not know if that a file name we are parsing or a math expression.</font></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;"><font face="monospace, monospace"><br>
</font></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;"><font face="monospace, monospace">May be worth trying to handle this on higher level, during evaluation of</font></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;"><font face="monospace, monospace">expressions ?</font></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;"><font face="monospace, monospace"><br>
</font></div>
</div>
<div class="gmail_extra" style="font-family: Calibri, Arial, Helvetica, sans-serif;">
George.<br>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>