<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:p="urn:schemas-microsoft-com:office:powerpoint" xmlns:a="urn:schemas-microsoft-com:office:access" xmlns:dt="uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns:s="uuid:BDC6E3F0-6DA3-11d1-A2A3-00AA00C14882" xmlns:rs="urn:schemas-microsoft-com:rowset" xmlns:z="#RowsetSchema" xmlns:b="urn:schemas-microsoft-com:office:publisher" xmlns:ss="urn:schemas-microsoft-com:office:spreadsheet" xmlns:c="urn:schemas-microsoft-com:office:component:spreadsheet" xmlns:odc="urn:schemas-microsoft-com:office:odc" xmlns:oa="urn:schemas-microsoft-com:office:activation" xmlns:html="http://www.w3.org/TR/REC-html40" xmlns:q="http://schemas.xmlsoap.org/soap/envelope/" xmlns:rtc="http://microsoft.com/officenet/conferencing" xmlns:D="DAV:" xmlns:Repl="http://schemas.microsoft.com/repl/" xmlns:mt="http://schemas.microsoft.com/sharepoint/soap/meetings/" xmlns:x2="http://schemas.microsoft.com/office/excel/2003/xml" xmlns:ppda="http://www.passport.com/NameSpace.xsd" xmlns:ois="http://schemas.microsoft.com/sharepoint/soap/ois/" xmlns:dir="http://schemas.microsoft.com/sharepoint/soap/directory/" xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:dsp="http://schemas.microsoft.com/sharepoint/dsp" xmlns:udc="http://schemas.microsoft.com/data/udc" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:sub="http://schemas.microsoft.com/sharepoint/soap/2002/1/alerts/" xmlns:ec="http://www.w3.org/2001/04/xmlenc#" xmlns:sp="http://schemas.microsoft.com/sharepoint/" xmlns:sps="http://schemas.microsoft.com/sharepoint/soap/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:udcs="http://schemas.microsoft.com/data/udc/soap" xmlns:udcxf="http://schemas.microsoft.com/data/udc/xmlfile" xmlns:udcp2p="http://schemas.microsoft.com/data/udc/parttopart" xmlns:wf="http://schemas.microsoft.com/sharepoint/soap/workflow/" xmlns:dsss="http://schemas.microsoft.com/office/2006/digsig-setup" xmlns:dssi="http://schemas.microsoft.com/office/2006/digsig" xmlns:mdssi="http://schemas.openxmlformats.org/package/2006/digital-signature" xmlns:mver="http://schemas.openxmlformats.org/markup-compatibility/2006" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns:mrels="http://schemas.openxmlformats.org/package/2006/relationships" xmlns:spwp="http://microsoft.com/sharepoint/webpartpages" xmlns:ex12t="http://schemas.microsoft.com/exchange/services/2006/types" xmlns:ex12m="http://schemas.microsoft.com/exchange/services/2006/messages" xmlns:pptsl="http://schemas.microsoft.com/sharepoint/soap/SlideLibrary/" xmlns:spsl="http://microsoft.com/webservices/SharePointPortalServer/PublishedLinksService" xmlns:Z="urn:schemas-microsoft-com:" xmlns:st="" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
{page:WordSection1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=DE link=blue vlink=purple>
<div class=WordSection1>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Konstantin,<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>To the best of my knowledge we do not have any SW test routines to
measure these latencies. You may want to ask in the MARC forum (<a
href="http://communities.intel.com/community/marc">http://communities.intel.com/community/marc</a>),
though.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>I consider the numbers as highly trustworthy as we got them through
simulating the logic we built. If software is used to derive these results one
has to take uncertainties in program execution into account. Even in a
BareMetal environment and with this simple in-order core I doubt that you can
measure times with single clock cycle precision because there is jitter among
pairs of RDTSC if there are outstanding memory operations in the pipeline.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Regarding your latency-related questions, please note that all
times in the latency table are measured from the output of the core, i.e. this
implies a L1 miss. For memory accesses with the MPBT attribute bit set the L2
cache is transparent and as far I can see in the implementation it does not
matter whether the L2 is enabled or not. So the times for your red and orange
scenarios are the ones from the table, i.e. either 15 core or 45 core + 8 mesh clock
cycles. I never thought about the impact on access latencies if either paging in
the MMU or caching in L1 was disabled. Such time savings occur inside the P54C
core, i.e. they do not affect the latencies listed in the table.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Best regards,<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Werner<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<div style='border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt'>
<div>
<div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'>
<p class=MsoNormal><b><span lang=EN-US style='font-size:10.0pt;font-family:
"Tahoma","sans-serif"'>From:</span></b><span lang=EN-US style='font-size:10.0pt;
font-family:"Tahoma","sans-serif"'> Konstantin Zertsekel
[mailto:zertsekel@gmail.com] <br>
<b>Sent:</b> Wednesday, March 02, 2011 9:43 AM<br>
<b>To:</b> Haas, Werner<br>
<b>Cc:</b> barrelfish-users@lists.inf.ethz.ch; Dan Tsafrir; Roei; Ido Shamay;
Avi Mendelson; Prof. Assaf Schuster<br>
<b>Subject:</b> Re: [Barrelfish-users] Intel SCC latency measurements for MPB
operations<o:p></o:p></span></p>
</div>
</div>
<p class=MsoNormal><o:p> </o:p></p>
<div>
<p class=MsoNormal style='margin-bottom:12.0pt'>Werner, thanks for the answer.<br>
Assuming we won't use bypass mode to access the local MPB, is there any
software that tests the latency of accessing the local MPB. Our first-step goal
is to measure the latency in this simple case and see that it is 45 core clock
+ 8 mesh clock as is stated in the graph or at least could be measured in
clocks, not microseconds.<br>
Can you please relate to this picture: [<a
href="https://docs.google.com/drawings/edit?id=1X-U10YjKvFQ22sdsKcNFnpHKVLyhR8_6YhJv6tiySUo&hl=en">https://docs.google.com/drawings/edit?id=1X-U10YjKvFQ22sdsKcNFnpHKVLyhR8_6YhJv6tiySUo&hl=en</a>]?
Did anyone measured latency of those paths?<br>
Thanks again, KostaZ.<o:p></o:p></p>
<div>
<p class=MsoNormal>On Tue, Mar 1, 2011 at 5:53 PM, Haas, Werner <<a
href="mailto:werner.haas@intel.com">werner.haas@intel.com</a>> wrote:<o:p></o:p></p>
<div>
<div>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span
lang=EN-US style='font-size:11.0pt;color:#1F497D'>Konstantin,</span><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span
lang=EN-US style='font-size:11.0pt;color:#1F497D'> </span><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span
lang=EN-US style='font-size:11.0pt;color:#1F497D'>I work at Intel Labs so let
me try answering the RCCE-related part: The latency table reflects the numbers
from looking at the actual hardware, i.e. without taking software operation
into account. The RCCE round-trip times, however, were measured by running an
actual application, i.e. they rather reflect the efficiency of one particular
communication algorithm than hardware properties. I do not know the precise
number but there are actually several MPB accesses involved in passing data via
RCCE. </span><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span
lang=EN-US style='font-size:11.0pt;color:#1F497D'> </span><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span
lang=EN-US style='font-size:11.0pt;color:#1F497D'>Please also note that the
bypass mode should _<i>not</i>_ be used as we have a hardware bug which can
lead to reading incorrect data. Unfortunately this greatly reduces the benefit
of using the on-die SRAM vs. off-die DDR3.</span><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span
lang=EN-US style='font-size:11.0pt;color:#1F497D'> </span><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span
lang=EN-US style='font-size:11.0pt;color:#1F497D'>Best regards,</span><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span
lang=EN-US style='font-size:11.0pt;color:#1F497D'>Werner</span><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><o:p> </o:p></p>
</div>
</div>
</div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
</div>
</div>
<font face="monospace">--------------------------------------------------------------------------------------<br>
Intel GmbH<br>
Dornacher Strasse 1<br>
85622 Feldkirchen/Muenchen, Deutschland <br>
Sitz der Gesellschaft: Feldkirchen bei Muenchen<br>
Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer<br>
Registergericht: Muenchen HRB 47456 <br>
Ust.-IdNr./VAT Registration No.: DE129385895<br>
Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052</font></body>
</html>