<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=koi8-r">
<style type="text/css" style="display:none"><!-- p { margin-top: 0px; margin-bottom: 0px; }--></style>
</head>
<body dir="ltr" style="font-size:12pt;color:#000000;background-color:#FFFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;">
<p><span style="color: rgb(33, 33, 33); font-size: 12pt;">>Our tokenizer recognize</span><br>
</p>
<div style="color: rgb(33, 33, 33);">
<div>
<div dir="ltr">
<div><font face="monospace, monospace">> </font></div>
<div>> <font face="monospace, monospace">[A-Za-z0-9_.$/\\~=+[]*?\-:!<>]+</font></div>
<div>> </div>
<div>>as a token. gold uses more complex rules to tokenize. I don't think we need that much complex rules, but there seems to be >room to improve our tokenizer. In particular, I believe we can parse the Linux's linker script by changing the tokenizer rules
as >follows.</div>
<div>> </div>
<div>
<div><font face="monospace, monospace">> [A-Za-z_.$/\\~=+[]*?\-:!<>][A-Za-z0-9_.$/\\~=+[]*?\-:!<>]*</font></div>
<div>> </div>
<div>>or</div>
<div>> </div>
<div><font face="monospace, monospace">> [0-9]+</font></div>
</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">That probably should help a bit,<br>
</div>
<div class="gmail_extra">but does not solve a problem in general.<br>
</div>
<div class="gmail_extra">I think it will not work for expressions like<br>
</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">. = z5*4;<br>
</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">as it will read "z5*4" as a single token I think.<br>
</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">I was thinking about entering some special parser state for </div>
<div class="gmail_extra">extracting sub tokens from tokens transparently when<br>
</div>
<div class="gmail_extra">we are inside code that evaluates the expression.<br>
</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">We can start from your suggestion first I think and see how it works<br>
</div>
<div class="gmail_extra">and if we really face scripts writtel like above in real life.<br>
</div>
<div class="gmail_extra">At least it is not harmfull and should help to kernel.<br>
</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">I'll try to prepare a patch if you do not mind.<br>
</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">George.<br>
</div>
</div>
</div>
</div>
</body>
</html>