<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jan 24, 2017 at 7:57 AM, George Rimar <span dir="ltr"><<a href="mailto:grimar@accesssoftek.com" target="_blank">grimar@accesssoftek.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr" style="font-size:12pt;color:#000000;background-color:#ffffff;font-family:Calibri,Arial,Helvetica,sans-serif">
<p><span style="color:rgb(33,33,33);font-size:12pt">>Our tokenizer recognize</span></p>
<div dir="ltr" style="font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<div>
<div style="color:rgb(33,33,33)">
<div>
<div dir="ltr"><span>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif"><font face="monospace, monospace">>
</font></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif">> <font face="monospace, monospace">[A-Za-z0-9_.$/\\~=+[]*?\-:!<><wbr>]+</font></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif">> </div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif">>as a token. gold uses more complex rules to tokenize. I don't think we need that much complex rules, but there seems to be >room to improve our tokenizer. In particular, I believe we can parse
the Linux's linker script by changing the tokenizer rules as >follows.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif">> </div>
</span><div><span>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif"><font face="monospace, monospace">> [A-Za-z_.$/\\~=+[]*?\-:!<>][A<wbr>-Za-z0-9_.$/\\~=+[]*?\-:!<>]*</font></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif">> </div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif">>or</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif">> </div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif"><font face="monospace, monospace">> [0-9]+</font></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif"><font face="monospace, monospace"><br>
</font></div>
</span><div style="font-family:Calibri,Arial,Helvetica,sans-serif"><font face="monospace, monospace">After more investigation, that seems will not work so simple.</font></div>
<div><font face="monospace, monospace">Next are possible examples where it will be broken:</font></div>
<div><font face="monospace, monospace">. = 0x1000; (gives tokens "0, x1000")</font></div>
<div><font face="monospace, monospace">. = A*10; (gives "A*10")</font></div>
<div><font face="monospace, monospace">. = 10k; (gives "10, k")</font></div>
<div><font face="monospace, monospace">. = 10*5; (gives "10, *5"</font></div>
<div><font face="monospace, monospace"><br>
</font></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif"><font face="monospace, monospace">"[0-9]+" could be "[0-9][kmhKMHx0-9]*"</font></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif"><font face="monospace, monospace">but for "10*5" that anyways gives "10" and "*5" tokens.</font></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif"><font face="monospace, monospace">And I do not think we can involve some handling of operators,</font></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif"><font face="monospace, monospace">as <span style="color:rgb(33,33,33);font-family:monospace,monospace;font-size:16px;background-color:rgb(255,255,255)">its hard to assume some context</span>
on tokenizing step.</font></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif"><font face="monospace, monospace">We do not know if that a file name we are parsing or a math expression.</font></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif"><font face="monospace, monospace"><br>
</font></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif"><font face="monospace, monospace">May be worth trying to handle this on higher level, during evaluation of</font></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif"><font face="monospace, monospace">expressions ?</font></div></div></div></div></div></div></div></div></blockquote><div><br></div><div>The lexical format of linker scripts requires a context-sensitive lexer.</div><div><br></div><div>Look at how gold does it. IIRC there are 3 cases that are something like: one is for file-name like things, one is for numbers and stuff, and the last category is for numbers and stuff but numbers can also include things like `10k` (I think; would need to look at the code to remember for sure). It's done in a very elegant way in gold (passing a callback "can continue" that says which characters can continue the token). Which token regex to use is dependent on the grammar production (hence context sensitive). If you look at the other message I sent in this thread just now, ScriptParserBase is essentially a lexer interface and can be pretty easily converted to a more standard on-the-fly character-scanning implementation of a lexer. Once that is done adding a new method to scan a different kind of token for certain parts of the parser.</div><div><br></div><div>-- Sean Silva</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" style="font-size:12pt;color:#000000;background-color:#ffffff;font-family:Calibri,Arial,Helvetica,sans-serif"><div dir="ltr" style="font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"><div><div style="color:rgb(33,33,33)"><div><div dir="ltr"><div><span class="m_-4427920441883855883HOEnZb"><font color="#888888">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif"><font face="monospace, monospace"><br>
</font></div>
</font></span></div><span class="m_-4427920441883855883HOEnZb"><font color="#888888">
<div class="gmail_extra" style="font-family:Calibri,Arial,Helvetica,sans-serif">
George.<br>
</div>
</font></span></div>
</div>
</div>
</div>
</div>
</div>
</blockquote></div><br></div></div>