<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Sep 11, 2017 at 11:52 AM, Justin Bogner <span dir="ltr"><<a href="mailto:mail@justinbogner.com" target="_blank">mail@justinbogner.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="gmail-m_-902285400842551481gmail-">Kostya Serebryany <<a href="mailto:kcc@google.com" target="_blank">kcc@google.com</a>> writes:<br>
> Justin,<br>
<br>
> Calling appendToUsed has horrible complexity and if we call it in<br>
> every function clang consumes tons of memory (6Gb when compiling one<br>
> of the clang's source files). This killed my machine today :)<br>
><br>
> The solution is to call appendToUsed once per module, instead of once<br>
> per function.<br>
<br>
</span>Oh right, updating lists in metadata is O(n), so doing it per function<br>
is quadratic. This slipped my mind - sorry!<br></blockquote><div><br></div><div>Yea. Very unexpected (although I've stumbled on it a few times, I still don't remember about it)</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<span class="gmail-m_-902285400842551481gmail-"><br>
> Also, since this does not seem to be required for linux, I've put this<br>
> under if TargetTriple.isOSBinFormatMach<wbr>O Submitted r312855, I'll see<br>
> if this breaks Mac (there seem to be no llvm tests for this, only<br>
> compiler-rt tests) but please also check if this looks ok.<br>
<br>
</span>This looks fine, though I'd rather if we just did it on all platforms<br>
for consistency / clear semantic intent. Running appendToUsed once<br>
should be cheap, and we do it unguarded in our other instrumentation<br>
(like InstrProfiling.cpp).<br></blockquote><div><br></div><div>It's unclear if it's going to hurt things on linux. </div><div>If I see it doesn't -- I'll remove the if TargetTriple.isOSBinFormatMach<wbr>O</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<span class="gmail-m_-902285400842551481gmail-"><br>
> But this all still sounds bad on linux at least:<br>
> * with the old bfd linker and -ffunction-sections -Wl,-gc-sections these<br>
> arrays get removed (as discussed here)<br>
<br>
</span>This is sad, and I don't think we have any particularly good ideas to<br>
fix this.<br>
<span class="gmail-m_-902285400842551481gmail-"><br>
> * with newer linkers the sanitizer coverage essentially disables<br>
> gc-sections<br>
<br>
</span>I'm not sure I follow. We're making sure the global arrays /<br>
instrumentation data doesn't get dead stripped here, but dead stripping<br>
of functions should work as normal.<br></blockquote><div><br></div><div>Ah, that's now my confusion. </div><div>inline-8bit-counters and trace-pc-guard seems to work just fine. <br></div><div>pc-table actually does break gc-sections as there is no a reference to every function in the pc-table section. <br></div><div><br></div><div>+eugenis, who dealt with a similar issue in asan (although it's probably a separate topic now). </div><div><br></div><div>--kcc </div><div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div class="gmail-m_-902285400842551481gmail-HOEnZb"><div class="gmail-m_-902285400842551481gmail-h5"><br>
> --kcc<br>
><br>
><br>
> On Thu, Aug 24, 2017 at 6:43 PM, Peter Collingbourne <<a href="mailto:peter@pcc.me.uk" target="_blank">peter@pcc.me.uk</a>><br>
> wrote:<br>
><br>
>><br>
>><br>
>> On Thu, Aug 24, 2017 at 6:30 PM, Justin Bogner <<a href="mailto:mail@justinbogner.com" target="_blank">mail@justinbogner.com</a>><br>
>> wrote:<br>
>><br>
>>> Peter Collingbourne <<a href="mailto:peter@pcc.me.uk" target="_blank">peter@pcc.me.uk</a>> writes:<br>
>>> > On Thu, Aug 24, 2017 at 3:38 PM, Kostya Serebryany <<a href="mailto:kcc@google.com" target="_blank">kcc@google.com</a>><br>
>>> wrote:<br>
>>> ><br>
>>> >><br>
>>> >><br>
>>> >> On Thu, Aug 24, 2017 at 3:35 PM, Peter Collingbourne <<a href="mailto:peter@pcc.me.uk" target="_blank">peter@pcc.me.uk</a>><br>
>>> >> wrote:<br>
>>> >><br>
>>> >>> On Thu, Aug 24, 2017 at 3:21 PM, Kostya Serebryany via llvm-dev <<br>
>>> >>> <a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>> wrote:<br>
>>> >>><br>
>>> >>>><br>
>>> >>>><br>
>>> >>>> On Thu, Aug 24, 2017 at 3:20 PM, Justin Bogner <<br>
>>> <a href="mailto:mail@justinbogner.com" target="_blank">mail@justinbogner.com</a>><br>
>>> >>>> wrote:<br>
>>> >>>><br>
>>> >>>>> I think the simplest fix is something like this:<br>
>>> >>>>><br>
>>> >>>>> diff --git a/lib/Transforms/Instrumentati<wbr>on/SanitizerCoverage.cpp<br>
>>> >>>>> b/lib/Transforms/Instrumentati<wbr>on/SanitizerCoverage.cpp<br>
>>> >>>>> index c6f0d17f8fe..e81957ab80a 100644<br>
>>> >>>>> --- a/lib/Transforms/Instrumentati<wbr>on/SanitizerCoverage.cpp<br>
>>> >>>>> +++ b/lib/Transforms/Instrumentati<wbr>on/SanitizerCoverage.cpp<br>
>>> >>>>> @@ -256,6 +256,7 @@ SanitizerCoverageModule::Creat<br>
>>> eSecStartEnd(Module<br>
>>> >>>>> &M, const char *Section,<br>
>>> >>>>> new GlobalVariable(M, Ty, false,<br>
>>> GlobalVariable::ExternalLinkag<br>
>>> >>>>> e,<br>
>>> >>>>> nullptr, getSectionEnd(Section));<br>
>>> >>>>> SecEnd->setVisibility(GlobalVa<wbr>lue::HiddenVisibility);<br>
>>> >>>>> + appendToUsed(M, {SecStart, SecEnd});<br>
>>> >>>>><br>
>>> >>>>> return std::make_pair(SecStart, SecEnd);<br>
>>> >>>>> }<br>
>>> >>>>><br>
>>> >>>>> I'm trying it out now.<br>
>>> >>>>><br>
>>> >>>><br>
>>> >>>> LGTM (if this works), thanks!<br>
>>> >>>><br>
>>> >>><br>
>>> >>> I wouldn't expect that to work because for ELF targets llvm.used has<br>
>>> no<br>
>>> >>> effect on the object file (only on the optimizer).<br>
>>> >>><br>
>>> >>> Is there a simple way to reproduce the link failure?<br>
>>> >>><br>
>>> >><br>
>>> >><br>
>>> >> ninja compiler-rt<br>
>>> >> echo 'extern "C" int LLVMFuzzerTestOneInput(const unsigned char *a,<br>
>>> >> unsigned long b){return 0; } ' > test.cc<br>
>>> >> clang -O3 test.cc -fsanitize=fuzzer # works<br>
>>> >> clang -O3 test.cc -Wl,-gc-sections -fsanitize=fuzzer # fails<br>
>>> >><br>
>>> ><br>
>>> > It seems that the issue is that older versions of ld.bfd have a bug<br>
>>> which<br>
>>> > causes it not to define __start_ and __stop_ symbols if the only<br>
>>> reference<br>
>>> > to those symbols is from a constructor.<br>
>>><br>
>>> It looks like this is a different problem from the one on macOS (and I<br>
>>> wasn't able to reproduce it with any bfd ld I had available, they were<br>
>>> all too new)<br>
>>><br>
>>> I've gone ahead and fixed the issue on macOS in r311742.<br>
>>><br>
>>> > If I add an artificial reference to the start symbol from libfuzzer's<br>
>>> main<br>
>>> > function, the program links correctly.<br>
>>> ><br>
>>> > diff --git a/compiler-rt/lib/fuzzer/Fuzze<wbr>rMain.cpp<br>
>>> > b/compiler-rt/lib/fuzzer/Fuzze<wbr>rMain.cpp<br>
>>> > index af8657200be2..c41e28e012db 100644<br>
>>> > --- a/compiler-rt/lib/fuzzer/Fuzze<wbr>rMain.cpp<br>
>>> > +++ b/compiler-rt/lib/fuzzer/Fuzze<wbr>rMain.cpp<br>
>>> > @@ -16,6 +16,10 @@ extern "C" {<br>
>>> > int LLVMFuzzerTestOneInput(const uint8_t *Data, size_t Size);<br>
>>> > } // extern "C"<br>
>>> ><br>
>>> > +__attribute__((weak)) void nop(void *p) {}<br>
>>> > +extern void *__start___sancov_pcs;<br>
>>> > +<br>
>>> > int main(int argc, char **argv) {<br>
>>> > + nop(__start___sancov_pcs);<br>
>>> > return fuzzer::FuzzerDriver(&argc, &argv, LLVMFuzzerTestOneInput);<br>
>>> > }<br>
>>><br>
>>> If we were to do this, we'd have to guard it appropriately - not all<br>
>>> platforms name the __start symbols like this.<br>
>>><br>
>><br>
>> Of course. There's also the issue of how to keep the symbols alive in DSOs.<br>
>><br>
>> > The problem also goes away if I use "GNU ld (GNU Binutils)<br>
>>> > 2.28.51.20170105".<br>
>>><br>
>>> 2.27 also doesn't have the issue. I don't know what our minimum version<br>
>>> of binutils is, and I'm under the impression most people use gold or lld<br>
>>> to link LLVM these days, so it isn't clear to me how big of a problem<br>
>>> this is.<br>
>>><br>
>><br>
>> For the record, the problem reproduces under 2.24, which is shipped by<br>
>> Ubuntu 14.04 LTS, which isn't that old. My view is that if we can find an<br>
>> unintrusive enough workaround, we should deploy it (with a comment to<br>
>> remove it after N years).<br>
>><br>
>> Peter<br>
>><br>
>><br>
>>> > Peter<br>
>>> ><br>
>>> ><br>
>>> ><br>
>>> >><br>
>>> >><br>
>>> >><br>
>>> >><br>
>>> >><br>
>>> >>><br>
>>> >>> Peter<br>
>>> >>><br>
>>> >>><br>
>>> >>>><br>
>>> >>>>><br>
>>> >>>>> Kostya Serebryany <<a href="mailto:kcc@google.com" target="_blank">kcc@google.com</a>> writes:<br>
>>> >>>>> > With -Wl,-gc-sections I get this:<br>
>>> >>>>> > SimpleTest.cpp:(.text.sancov.m<wbr>odule_ctor[sancov.module_ctor]<br>
>>> +0x1b):<br>
>>> >>>>> > undefined reference to `__start___sancov_pcs'<br>
>>> >>>>> > SimpleTest.cpp:(.text.sancov.m<wbr>odule_ctor[sancov.module_ctor]<br>
>>> +0x20):<br>
>>> >>>>> > undefined reference to `__stop___sancov_pcs'<br>
>>> >>>>> ><br>
>>> >>>>> ><br>
>>> >>>>> ><br>
>>> >>>>> > On Thu, Aug 24, 2017 at 3:07 PM, George Karpenkov <<br>
>>> >>>>> <a href="mailto:ekarpenkov@apple.com" target="_blank">ekarpenkov@apple.com</a>><br>
>>> >>>>> > wrote:<br>
>>> >>>>> ><br>
>>> >>>>> >><br>
>>> >>>>> >> On Aug 24, 2017, at 2:55 PM, Kostya Serebryany <<a href="mailto:kcc@google.com" target="_blank">kcc@google.com</a>><br>
>>> >>>>> wrote:<br>
>>> >>>>> >><br>
>>> >>>>> >> Interesting.<br>
>>> >>>>> >> This is a relatively new addition (fsanitize-coverage=pc-tables,<br>
>>> >>>>> which is<br>
>>> >>>>> >> now a part of -fsanitize=fuzzer).<br>
>>> >>>>> >> The tests worked (did they? On Mac?) so I thought everything is<br>
>>> ok.<br>
>>> >>>>> >><br>
>>> >>>>> >><br>
>>> >>>>> >> For tests we never compile the tested target with -O3 (and that<br>
>>> >>>>> wouldn’t<br>
>>> >>>>> >> be sufficient),<br>
>>> >>>>> >> and for testing fuzzers I was always building them in debug<br>
>>> >>>>> >><br>
>>> >>>>> >> Yea, we need to make sure the pc-tables are not stripped (this<br>
>>> is a<br>
>>> >>>>> >> separate section with globals).<br>
>>> >>>>> >> (I still haven't documented pc-tables, will do soon)<br>
>>> >>>>> >><br>
>>> >>>>> >><br>
>>> >>>>> >> Do you know what's the analog of Wl,-dead_strip on Linux?<br>
>>> >>>>> >><br>
>>> >>>>> >><br>
>>> >>>>> >> Apparently -Wl,—gc-sections.<br>
>>> >>>>> >> For some reason LLVM does not do it for gold, even though it<br>
>>> seems to<br>
>>> >>>>> >> support this flag as well.<br>
>>> >>>>> >> (that could be another reason why you don’t see the failure on<br>
>>> Linux)<br>
>>> >>>>> >><br>
>>> >>>>> >> 1 *if*(NOT LLVM_NO_DEAD_STRIP)<br>
>>> >>>>> >> 2 *if*(${CMAKE_SYSTEM_NAME} MATCHES "Darwin")<br>
>>> >>>>> >> 3 # ld64's implementation of -dead_strip breaks tools that<br>
>>> use<br>
>>> >>>>> >> plugins.<br>
>>> >>>>> >> 4 set_property(TARGET ${target_name} APPEND_STRING PROPERTY<br>
>>> >>>>> >> 5 LINK_FLAGS " -Wl,-dead_strip")<br>
>>> >>>>> >> 6 *elseif*(${CMAKE_SYSTEM_NAME} MATCHES "SunOS")<br>
>>> >>>>> >> 7 set_property(TARGET ${target_name} APPEND_STRING PROPERTY<br>
>>> >>>>> >> 8 LINK_FLAGS " -Wl,-z<br>
>>> -Wl,discard-unused=sections")<br>
>>> >>>>> >> 9 *elseif*(NOT WIN32 AND NOT LLVM_LINKER_IS_GOLD)<br>
>>> >>>>> >> 10 # Object files are compiled with -ffunction-data-sections.<br>
>>> >>>>> >> 11 # Versions of bfd ld < 2.23.1 have a bug in --gc-sections<br>
>>> that<br>
>>> >>>>> >> breaks<br>
>>> >>>>> >> 12 # tools that use plugins. Always pass --gc-sections once<br>
>>> we<br>
>>> >>>>> require<br>
>>> >>>>> >> 13 # a newer linker.<br>
>>> >>>>> >> 14 set_property(TARGET ${target_name} APPEND_STRING PROPERTY<br>
>>> >>>>> >> 15 LINK_FLAGS " -Wl,--gc-sections")<br>
>>> >>>>> >> 16 *endif*()<br>
>>> >>>>> >> 17 *endif*()<br>
>>> >>>>> >><br>
>>> >>>>> >><br>
>>> >>>>> >><br>
>>> >>>>> >> --kcc<br>
>>> >>>>> >><br>
>>> >>>>> >><br>
>>> >>>>> >><br>
>>> >>>>> >> On Thu, Aug 24, 2017 at 2:49 PM, Justin Bogner <<br>
>>> >>>>> <a href="mailto:mail@justinbogner.com" target="_blank">mail@justinbogner.com</a>><br>
>>> >>>>> >> wrote:<br>
>>> >>>>> >><br>
>>> >>>>> >>> George Karpenkov <<a href="mailto:ekarpenkov@apple.com" target="_blank">ekarpenkov@apple.com</a>> writes:<br>
>>> >>>>> >>> > OK so with Kuba’s help I’ve found the error: with<br>
>>> optimization,<br>
>>> >>>>> dead<br>
>>> >>>>> >>> > stripping of produced libraries is enabled,<br>
>>> >>>>> >>> > which removes coverage instrumentation.<br>
>>> >>>>> >>> ><br>
>>> >>>>> >>> > However, this has nothing to do with the move to compiler-rt,<br>
>>> so<br>
>>> >>>>> I’m<br>
>>> >>>>> >>> > quite skeptical on whether it has worked<br>
>>> >>>>> >>> > beforehand.<br>
>>> >>>>> >>> ><br>
>>> >>>>> >>> > A trivial fix is to do:<br>
>>> >>>>> >>> ><br>
>>> >>>>> >>> > diff --git a/cmake/modules/HandleLLVMOpti<wbr>ons.cmake<br>
>>> >>>>> >>> b/cmake/modules/HandleLLVMOpti<wbr>ons.cmake<br>
>>> >>>>> >>> > index 04596a6ff63..5465d8d95ba 100644<br>
>>> >>>>> >>> > --- a/cmake/modules/HandleLLVMOpti<wbr>ons.cmake<br>
>>> >>>>> >>> > +++ b/cmake/modules/HandleLLVMOpti<wbr>ons.cmake<br>
>>> >>>>> >>> > @@ -665,6 +665,9 @@ if(LLVM_USE_SANITIZER)<br>
>>> >>>>> >>> > endif()<br>
>>> >>>>> >>> > if (LLVM_USE_SANITIZE_COVERAGE)<br>
>>> >>>>> >>> > append("-fsanitize=fuzzer-no-l<wbr>ink" CMAKE_C_FLAGS<br>
>>> >>>>> CMAKE_CXX_FLAGS)<br>
>>> >>>>> >>> > +<br>
>>> >>>>> >>> > + # Dead stripping messes up coverage instrumentation.<br>
>>> >>>>> >>> > + set(LLVM_NO_DEAD_STRIP ON)<br>
>>> >>>>> >>> > endif()<br>
>>> >>>>> >>> > endif()<br>
>>> >>>>> >>> ><br>
>>> >>>>> >>> > Any arguments against that?<br>
>>> >>>>> >>><br>
>>> >>>>> >>> We shouldn't do this. We really only want to prevent dead<br>
>>> stripping<br>
>>> >>>>> of<br>
>>> >>>>> >>> the counters themselves - disabling it completely isn't very<br>
>>> nice.<br>
>>> >>>>> >>><br>
>>> >>>>> >>> > Apparently, a better way is to follow ASAN instrumentation<br>
>>> pass,<br>
>>> >>>>> >>> > which uses some magic to protect against dead-stripping.<br>
>>> >>>>> >>><br>
>>> >>>>> >>> I thought this was already being done - how else did it work<br>
>>> before?<br>
>>> >>>>> >>><br>
>>> >>>>> >>> >> On Aug 24, 2017, at 11:29 AM, Justin Bogner <<br>
>>> >>>>> <a href="mailto:mail@justinbogner.com" target="_blank">mail@justinbogner.com</a>><br>
>>> >>>>> >>> wrote:<br>
>>> >>>>> >>> >><br>
>>> >>>>> >>> >> (kcc, george: sorry for the re-send, the first was from a<br>
>>> >>>>> non-list<br>
>>> >>>>> >>> email<br>
>>> >>>>> >>> >> address)<br>
>>> >>>>> >>> >><br>
>>> >>>>> >>> >> My configuration for building the fuzzers in the LLVM tree<br>
>>> >>>>> doesn't<br>
>>> >>>>> >>> seem to<br>
>>> >>>>> >>> >> work any more (possibly as of moving libFuzzer to<br>
>>> compiler-rt,<br>
>>> >>>>> but<br>
>>> >>>>> >>> there<br>
>>> >>>>> >>> >> have been a few other changes in the last week or so that<br>
>>> may be<br>
>>> >>>>> >>> related).<br>
>>> >>>>> >>> >><br>
>>> >>>>> >>> >> I'm building with a fresh top-of-tree clang and setting<br>
>>> >>>>> >>> >> -DLLVM_USE_SANITIZER=Address and<br>
>>> -DLLVM_USE_SANITIZE_COVERAGE=O<br>
>>> >>>>> n,<br>
>>> >>>>> >>> which<br>
>>> >>>>> >>> >> was working before:<br>
>>> >>>>> >>> >><br>
>>> >>>>> >>> >> % cmake -GNinja \<br>
>>> >>>>> >>> >> -DCMAKE_BUILD_TYPE=Release<br>
>>> -DLLVM_ENABLE_ASSERTIONS=On \<br>
>>> >>>>> >>> >> -DLLVM_ENABLE_WERROR=On \<br>
>>> >>>>> >>> >> -DLLVM_USE_SANITIZER=Address<br>
>>> >>>>> -DLLVM_USE_SANITIZE_COVERAGE=O<wbr>n<br>
>>> >>>>> >>> \<br>
>>> >>>>> >>> >> -DCMAKE_C_COMPILER=$HOME/llvm-<wbr>lkgc/bin/clang \<br>
>>> >>>>> >>> >> $HOME/code/llvm-src<br>
>>> >>>>> >>> >><br>
>>> >>>>> >>> >> But when I run any of the fuzzers, it looks like the<br>
>>> sanitizer<br>
>>> >>>>> coverage<br>
>>> >>>>> >>> >> hasn't been set up correctly:<br>
>>> >>>>> >>> >><br>
>>> >>>>> >>> >> % ./bin/llvm-as-fuzzer<br>
>>> >>>>> >>> 2017-08-24 11:14:33<br>
>>> >>>>> >>> >> INFO: Seed: 4089166883 <(408)%20916-6883><br>
>>> >>>>> >>> >> INFO: Loaded 1 modules (50607 guards): 50607 [0x10e14ef80,<br>
>>> >>>>> >>> 0x10e18063c),<br>
>>> >>>>> >>> >> INFO: Loaded 1 PC tables (0 PCs): 0<br>
>>> [0x10e2870a8,0x10e2870a8),<br>
>>> >>>>> >>> >> ERROR: The size of coverage PC tables does not match the<br>
>>> number<br>
>>> >>>>> of<br>
>>> >>>>> >>> instrumented PCs. This might be a bug in the compiler, please<br>
>>> >>>>> contact the<br>
>>> >>>>> >>> libFuzzer developers.<br>
>>> >>>>> >>> >><br>
>>> >>>>> >>> >> From the build logs, it looks like we're now building objects<br>
>>> >>>>> with<br>
>>> >>>>> >>> these<br>
>>> >>>>> >>> >> sanitizer flags:<br>
>>> >>>>> >>> >><br>
>>> >>>>> >>> >> -fsanitize=address<br>
>>> >>>>> >>> >> -fsanitize-address-use-after-s<wbr>cope<br>
>>> >>>>> >>> >> -fsanitize=fuzzer-no-link<br>
>>> >>>>> >>> >><br>
>>> >>>>> >>> >> We're then linking the fuzzer binaries with these:<br>
>>> >>>>> >>> >><br>
>>> >>>>> >>> >> -fsanitize=address<br>
>>> >>>>> >>> >> -fsanitize-address-use-after-s<wbr>cope<br>
>>> >>>>> >>> >> -fsanitize=fuzzer-no-link<br>
>>> >>>>> >>> >> -fsanitize=fuzzer<br>
>>> >>>>> >>> >><br>
>>> >>>>> >>> >> Any idea what's wrong or where to start looking?<br>
>>> >>>>> >>><br>
>>> >>>>> >><br>
>>> >>>>> >><br>
>>> >>>>> >><br>
>>> >>>>><br>
>>> >>>><br>
>>> >>>><br>
>>> >>>> ______________________________<wbr>_________________<br>
>>> >>>> LLVM Developers mailing list<br>
>>> >>>> <a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
>>> >>>> <a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/<wbr>mailman/listinfo/llvm-dev</a><br>
>>> >>>><br>
>>> >>>><br>
>>> >>><br>
>>> >>><br>
>>> >>> --<br>
>>> >>> --<br>
>>> >>> Peter<br>
>>> >>><br>
>>> >><br>
>>> >><br>
>>> ><br>
>>> ><br>
>>> > --<br>
>>><br>
>><br>
>><br>
>><br>
>> --<br>
>> --<br>
>> Peter<br>
>><br>
</div></div></blockquote></div><br></div></div>