tag:blogger.com,1999:blog-135963392024-02-03T05:34:19.222-05:00ccg.tech...everything, all at once...ccghttp://www.blogger.com/profile/01309764084115413592noreply@blogger.comBlogger27125tag:blogger.com,1999:blog-13596339.post-73296757991665954292012-01-17T23:41:00.000-05:002012-01-17T23:41:26.148-05:00CENSORED<div class="separator" style="clear: both; text-align: center;"><a href="https://encrypted-tbn3.google.com/images?q=tbn:ANd9GcT4SHcRfsH7DHQdxa_WDKX-t3OP_bK1dr3JPJkMQdOwm9xnNvHVTA" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="348" src="https://encrypted-tbn3.google.com/images?q=tbn:ANd9GcT4SHcRfsH7DHQdxa_WDKX-t3OP_bK1dr3JPJkMQdOwm9xnNvHVTA" width="640" /></a></div><div class="separator" style="clear: both; text-align: center;"><br />
</div><div class="separator" style="clear: both; text-align: center;"><br />
</div><div class="separator" style="clear: both; text-align: center;"><br />
</div><div class="separator" style="clear: both; text-align: center;">For the next 24 hours this blog is censored. This is what blogs look like on SOPA (and PIPA). Read up and make your own decision: <a href="http://www.blogger.com/%C2%A0http://projects.propublica.org/sopa/"> http://projects.propublica.org/sopa/ </a>. And if you agree, join Wikipedia, Reddit, and others by "going dark" for 24 hours. You can support the protest via Facebook, Twitter, and Google+ by changing your profile picture: <a href="http://www.blackoutsopa.org/">http://www.blackoutsopa.org/</a></div><div class="separator" style="clear: both; text-align: center;"><br />
</div><br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://encrypted-tbn0.google.com/images?q=tbn:ANd9GcQmpHGeFNrUnY9Qvd1hlNx64RAqfzxjO44AbQaZQGT0tCNVMtqlqA" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="380" src="https://encrypted-tbn0.google.com/images?q=tbn:ANd9GcQmpHGeFNrUnY9Qvd1hlNx64RAqfzxjO44AbQaZQGT0tCNVMtqlqA" width="400" /></a></div><br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://encrypted-tbn3.google.com/images?q=tbn:ANd9GcQqMq2JZwWmLSkg6iW4AXIP1nN-q8mZM2W5LKywYpXdX8aGXEBwCA" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="512" src="https://encrypted-tbn3.google.com/images?q=tbn:ANd9GcQqMq2JZwWmLSkg6iW4AXIP1nN-q8mZM2W5LKywYpXdX8aGXEBwCA" width="640" /></a></div>ccghttp://www.blogger.com/profile/01309764084115413592noreply@blogger.com0tag:blogger.com,1999:blog-13596339.post-24628693628409600542010-02-10T15:13:00.000-05:002010-02-10T15:13:00.564-05:00Hadoop HDFS: Deceived by Xciever<span class="Apple-style-span"><span class="Apple-style-span" style="font-family: arial;"><span class="Apple-style-span" style="font-size: x-large;">It</span><span class="Apple-style-span" style="font-size: small;">'s undocumented. It's misspelled. And without it your (insert euphemism for "really big" here) <a href="http://hadoop.apache.org/">Hadoop</a> grid won't be able to crunch those multi-hundred gigabyte input files. It's the "xciever" setting for <a href="http://hadoop.apache.org/hdfs/">HDFS</a> <a href="http://hadoop.apache.org/common/docs/r0.20.0/hdfs_design.html#NameNode+and+DataNodes">DataNode</a> daemons, and the absence of it in your configuration is waiting to bite you in the butt.</span></span></span><br />
<div><span class="Apple-style-span" style="font-family: arial;"><span class="Apple-style-span" style="font-size: small;"><br />
</span> </span></div><div><span class="Apple-style-span" style="font-family: arial;"><span class="Apple-style-span" style="font-size: small;">The software components that make up HDFS include the NameNode, the SecondaryNameNode, and DataNode daemons. Typical installations place NameNode and SecondaryNameNode on a single "master" node. Installations following best practices go the extra mile and place the NameNode and SecondaryNameNode daemons on separate nodes. DataNode daemons are placed on participating slaves nodes. At the highest level the NameNode/SecondaryNameNode daemons manage HDFS metadata (the data used to map file names to server/block lists, etc.), while the DataNode daemons handle the mechanics of read/writing to disk, and serving up blocks of data to processes on the node or to requesting processes on other nodes.</span></span></div><div><span class="Apple-style-span" style="font-family: arial;"><span class="Apple-style-span" style="font-size: small;"><br />
</span> </span></div><div><span class="Apple-style-span" style="font-family: arial;"><span class="Apple-style-span" style="font-size: small;">There are many settings that can be use to configure and tune HDFS (as well as the <a href="http://hadoop.apache.org/mapreduce/">MapReduce</a> engine and other Hadoop components). The </span></span><a href="http://hadoop.apache.org/common/docs/r0.20.0/hdfs_user_guide.html"><span class="Apple-style-span" style="font-family: arial;"><span class="Apple-style-span" style="font-size: small;">HDFS documentation</span></span></a><span class="Apple-style-span" style="font-family: arial;"><span class="Apple-style-span" style="font-size: small;"> lists </span></span><a href="http://hadoop.apache.org/common/docs/current/hdfs-default.html"><span class="Apple-style-span" style="font-family: arial;"><span class="Apple-style-span" style="font-size: small;">45 of them</span></span></a><span class="Apple-style-span" style="font-family: arial;"><span class="Apple-style-span" style="font-size: small;"> (along with their default values) last time I checked. These configuration settings are a somewhat disorganized mix of elements used to configure the NameNode and DataNodes.</span> </span><br />
<span class="Apple-style-span" style="font-family: arial;"><span class="Apple-style-span" style="font-size: small;"><br />
</span></span></div><div><span class="Apple-style-span"><span class="Apple-style-span" style="font-family: arial;"><span class="Apple-style-span" style="font-size: small;">The DataNode daemons have an upper limit on the number of threads they can support, which is set to the absurdly small (IMHO) value of 256. If you have a large job (or even a moderately-sized job) that has many files open you can exhaust this limit. When you do exceed this limit there are a variety of failure modes that can arise. Some of the exception traces point to this value being exhausted, while others do not. In either case, understanding what has gone wrong is difficult. For example, one failure mode observed <br />
raised</span><span class="Apple-style-span" style="font-size: small;"> </span></span></span><span class="Apple-style-span" style="white-space: pre;"><span class="Apple-style-span"><span class="Apple-style-span" style="font-family: 'courier new';"><span class="Apple-style-span" style="font-size: small;">java.io.EOFException</span></span><span class="Apple-style-span" style="font-family: arial;">, <span class="Apple-style-span" style="font-size: small;">with a traceback flowing down into <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">DFSClient</span>.</span></span></span></span></div><div><span class="Apple-style-span" style="white-space: pre;"><span class="Apple-style-span" style="white-space: normal;"><span class="Apple-style-span"><span class="Apple-style-span" style="font-family: arial;"><br />
</span></span></span></span></div><div><span class="Apple-style-span" style="white-space: pre;"><span class="Apple-style-span" style="white-space: normal;"><span class="Apple-style-span"><span class="Apple-style-span" style="font-family: arial;"><span class="Apple-style-span" style="font-size: small;">The solution to all this is to configure the Xcievers setting to raise the maximum limit on the number of threads the DataNode daemons are willing to manage. For modern Hadoop releases this should be done in <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">conf/hdfs-site.xml</span>. Here is an example:<br />
</span></span></span></span></span><br />
<span class="Apple-style-span" style="font-family: arial;"><span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="background-color: #cccccc;">< property > </span></span></span><br />
<span class="Apple-style-span" style="font-family: arial;"><span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="background-color: #cccccc;"> < name >dfs.datanode.max.xcievers< /name > </span></span></span><br />
<span class="Apple-style-span" style="font-family: arial;"><span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="background-color: #cccccc;"> < value >4096< /value > </span></span></span><br />
<span class="Apple-style-span" style="font-family: arial;"><span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="background-color: #cccccc;">< /property > </span></span></span><br />
<div><span class="Apple-style-span" style="font-family: arial;"><span class="Apple-style-span" style="font-size: small;"><br />
</span></span></div><div><span class="Apple-style-span" style="font-family: arial;"><span class="Apple-style-span" style="font-size: small;">[Notez-bien: I apologize for the lame formatting above. I really need to improve my template]</span></span></div><div><span class="Apple-style-span" style="font-family: arial;"><span class="Apple-style-span" style="font-size: small;"><br />
</span></span></div><div><span class="Apple-style-span" style="font-family: arial;"><span class="Apple-style-span" style="font-size: small;">Interestingly, <a href="http://hadoop.apache.org/hbase/">HBase</a> users seem to have more <a href="http://wiki.apache.org/hadoop/Hbase/Troubleshooting#A5">conversations</a> about this setting and related issues than do predominantly MapReduce users.</span></span></div><div><span class="Apple-style-span" style="font-family: arial;"><span class="Apple-style-span" style="font-size: small;"><br />
</span></span></div></div><div><span class="Apple-style-span" style="white-space: pre;"><span class="Apple-style-span" style="white-space: normal;"><span class="Apple-style-span" style="font-family: arial;"><span class="Apple-style-span" style="font-family: Georgia, serif;"><span class="Apple-style-span" style="font-size: small;"> <blah blah=""></blah></span></span></span></span></span></div><div><span class="Apple-style-span" style="white-space: pre;"><span class="Apple-style-span" style="white-space: normal;"><span class="Apple-style-span" style="font-family: arial;"><span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="font-family: Georgia, serif;"></span></span></span></span></span></div><div><span class="Apple-style-span" style="font-family: arial;"><span class="Apple-style-span" style="font-size: small;">The central reason for this posting is to point out this setting as it might be helpful to others. We encountered this problem while porting from an older version of Hadoop to 0.20 (via the </span><a href="http://cloudera.com/"><span class="Apple-style-span" style="font-size: small;">Cloudera</span></a><span class="Apple-style-span" style="font-size: small;"> </span><a href="http://www.cloudera.com/developers/downloads/"><span class="Apple-style-span" style="font-size: small;">distribution</span></a><span class="Apple-style-span" style="font-size: small;">, which I highly recommend). The failure mode was not immediately evident and we wasted a lot of time debugging and chasing assorted theories before we recognized what was happening and changed the configuration. Ironically the<a href="http://issues.apache.org/jira/browse/HDFS-162?focusedCommentId=12636150&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12636150"> bread crumbs</a> which led us to change this setting came from a <a href="http://issues.apache.org/jira/browse/HDFS-162">JIRA I opened</a> regarding a scalability issue in 2008. As I recall the setting was not supported by the version of Hadoop we were using at that time.</span></span></div><div><span class="Apple-style-span" style="font-family: arial;"><span class="Apple-style-span" style="font-size: small;"><br />
</span> </span></div><div><span class="Apple-style-span" style="font-family: arial;"><span class="Apple-style-span" style="font-size: small;">A secondary reason for this posting is to point out the crazy name associated with this setting. The term "xceiver" ("i" before "e" except after "c" blah blah blah...) is short-hand for transceiver, which means a device which transmits and receives. But does the phrase "transceiver" really describe "maximum threads a DataNode can support"? At the very least the spelling of this setting should be corrected and the underlying code should recognize both spellings. What would be even better would be add a setting called "transceivers" or even more explicitly "concurrent_threads" like this:</span></span></div><div><span class="Apple-style-span" style="font-family: arial;"><span class="Apple-style-span" style="font-size: small;"><br />
</span> </span></div><div><div><span class="Apple-style-span" style="font-family: 'courier new';"><span class="Apple-style-span" style="font-size: small;">dfs.datanode.max.transceivers </span></span><span class="Apple-style-span" style="font-family: arial;"><span class="Apple-style-span" style="font-size: small;"> </span></span></div><div><span class="Apple-style-span" style="font-family: arial;"><span class="Apple-style-span" style="font-size: small;">or</span></span></div><div><div><span class="Apple-style-span" style="font-family: 'courier new';"><span class="Apple-style-span" style="font-size: small;">dfs.datanode.max.concurrent_threads</span></span></div><div><span class="Apple-style-span" style="font-family: arial;"> </span><br />
<span class="Apple-style-span" style="font-family: arial;"><span class="Apple-style-span" style="font-size: small;">Finally, why is the default value for this setting so low? Why not have it default to a more reasonable value like 2048 or 4096 out of the box? Memory and CPU is cheap, chasing infrastructure issues is expensive.</span></span><br />
<span class="Apple-style-span" style="font-family: arial;"><span class="Apple-style-span" style="font-size: small;"><br />
</span></span></div><div><span class="Apple-style-span" style="font-family: arial;"><span class="Apple-style-span" style="font-size: small;">So summing it all up, lack of documentation and poor spelling of this setting caused us to lose hours of productivity. If you're deploying a big grid, be sure to configure this properly. If you're seeing strange failures related to I/O within your Hadoop infrastructure be sure to check for this value, and potentially increase it.</span></span></div><div><span class="Apple-style-span"><span class="Apple-style-span" style="font-family: arial;"><br />
</span></span></div></div></div><div><span class="Apple-style-span" style="white-space: pre;"><span class="Apple-style-span"><span class="Apple-style-span" style="font-family: arial;"> </span></span></span></div>ccghttp://www.blogger.com/profile/01309764084115413592noreply@blogger.com2tag:blogger.com,1999:blog-13596339.post-83428555815808465722008-12-31T02:54:00.009-05:002008-12-31T03:22:24.436-05:00Down Time == Productive Time<span class="Apple-style-span" style="font-family:arial;"><span class="Apple-style-span" style="font-size:small;">Back in my college days - long enough ago that mainframe computing was still the rage - I discovered the pleasure of The Holiday Break. Classes were over, everybody went home, the university was largely empty, and as a result you could grab all the computing time you wanted, and you could work for hours undisturbed. The Christmas holiday was always the best. </span></span><div><span class="Apple-style-span" style="font-size:small;"></span><span class="Apple-style-span" style="font-family:arial;"><br /></span><div><span class="Apple-style-span" style="font-family:arial;"><span class="Apple-style-span" style="font-size:small;">Fast forward to present day, and I continue to find the time from just before Christmas until just after the New Year a sort of magic time to get things done. Many offices are shut down for the holiday, so my unpredictable 45-minutes-or-maybe-3-hours commute becomes predictable. The shock jocks and on-air "personalities" on drive time FM radio are all on vacation, and so the radio actually plays music. Things in the office are quiet - folks don't normally schedule releases, death marches, etc., around the holidays. The time of year puts people in a mellow mood. All-in-all, it's a great time of year to grab some quiet time.</span></span></div><div><span class="Apple-style-span" style="font-family: arial;"><br /></span></div><div><span class="Apple-style-span" style="font-family: arial; ">Quiet time becomes productive time in unexpected ways. Since I'm not in the throes of a major release crunch I actually have time to catch up on some reading. Today, for example, I read about 20-30 pages from some of the books shown on my current reading list at the top of the blog. I also read a few great blog posts about a number of things technical. So great, I'm reading and surfing...but is it productive?</span><br /></div><div><span class="Apple-style-span" style="font-family:arial;"><br /></span></div><div><span class="Apple-style-span" style="font-family:arial;"><span class="Apple-style-span" style="font-size:small;">Oh yeah, it sure is. I concocted a new way to visualize how well or poorly a very complicated n-tier application is performing, in part based on some blog reading. Very cool stuff, for which I cannot take all the credit, but about which I will blog more in the near future.</span></span></div><div><span class="Apple-style-span" style="font-family:arial;"><br /></span></div><div><span class="Apple-style-span" style="font-family:arial;"><span class="Apple-style-span" style="font-size:small;">I also got a couple hours to write up some much needed documentation, and spent time catching up with colleagues who actually have time to talk about what they are working on and what sorts of challenges they are facing. This naturally leads to more ideas, and the ball starts rolling.</span></span></div><div><span class="Apple-style-span" style="font-family:arial;"><br /></span></div><div><span class="Apple-style-span" style="font-family:arial;"><span class="Apple-style-span" style="font-size:small;">People often talk about the December holidays as down-time for business, and I'm sure that it is. But the flip-side of the coin is that it can be a great time to open up the mind, solve some problems, and come up with good stuff to tackle in the upcoming year. Down time offers the chance to productively daydream without (usually) getting off schedule. So for engineers, down time can be super productive time. And maybe that's why when the weather gets cold, and the Christmas bell-ringers appear on every street corner, my mind turns to thoughts of software design, architecture, and coding. </span></span></div><div><br /></div></div>ccghttp://www.blogger.com/profile/01309764084115413592noreply@blogger.com0tag:blogger.com,1999:blog-13596339.post-67301694564847602622008-06-03T23:06:00.004-04:002008-06-04T00:06:41.124-04:00Visible Measures wins 2008 MITX Technology Award!<span style="font-family:Arial;font-size:85%;"><a href="http://www.visiblemeasures.com/">Visible Measures </a>(my current gig) won an award tonight at the <a href="http://www.mitx.org/">Massachusetts Innovation and Technology Exchange </a>(MITX) <a href="http://www.mitxawards.org/technologyawards/finalists_winners.aspx?year=2008">2008 What's Next Forum and Technology Awards</a>. We were recognized in the Analytics and Business Intelligence category - the same category that <a href="http://www.compete.com/">Compete</a> (my old company) was entered in last year.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">A lot of great companies were finalists in our category, including Progress Software, salary.com, SiteSpect, and Lexalytics. This was tough competition, which made winning this award all the more sweet. A big shout out to <a href="http://www.v2comms.com/">Version 2 Communications</a> as well - we were their guests at the awards.</span><span style="font-family:Arial;font-size:85%;"><br /></span><br /><span style="font-family:Arial;font-size:85%;">Visible Measures is an awesome company, with an extremely hard-working highly motvated team. I am extremely proud and humbled to be part of this company.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">The MITX event was very nice. There were plenty of opportunities to network with a lot of interesting people doing a lot of cool stuff. It was great to listen to Larry Weber (Chairman of the Board for MITX and founder of W2 Group) host the awards and dispense free advice ("...with 37 offices worldwide - that's too much overhead..."). MITX honored Amar Bose, who gave a very interesting talk. Bose is legenday - at least in the New England high tech community and particularly within MIT, so hearing him speak live is a privilege.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">The only downside to the evening was the fire alarm going off mid-way through the ceremony. This lead to a rather awkward pause in the action while the fire department made sure nothing was wrong. </span><br /><span style="font-family:Arial;font-size:85%;"></span>ccghttp://www.blogger.com/profile/01309764084115413592noreply@blogger.com0tag:blogger.com,1999:blog-13596339.post-53887092575497337512008-04-22T16:40:00.003-04:002008-04-22T16:45:09.855-04:00Hadoop Summit Slides<span style="font-family:arial;font-size:85%;">A few weeks ago I went to California for the Hadoop Summit. I posted a bunch of notes in real-time during the summit until the network connection became too flakey to continue. </span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">The Yahoo folks have come to the rescue however. The slides from the presentations, which are tons better than my notes, are freely available on-line <a href="http://research.yahoo.com/node/2104">here</a>. There are also slides from the Data Intensive Computing Symposium which was held the next day. </span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">I wish I had know about the Data Intensive symposium as it looks really really interesting (not to mention an excuse to stay in Califorinia one more day...).</span>ccghttp://www.blogger.com/profile/01309764084115413592noreply@blogger.com0tag:blogger.com,1999:blog-13596339.post-11065227619461948672008-04-10T01:57:00.002-04:002008-04-10T02:37:28.718-04:00Infrant/NetGear ReadyNAS NV+<span style="font-family:Arial;font-size:85%;">Just picked up one of these last week. The plan is to use the box as a shared storage resource to back up family data (pictures, etc.), and to back up other systems, and the grid machines in the rack.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">I was originally going to build a box to handle the task, but a friend of mine recommended the ReadyNAS server as a cost effective (and less labor intensive) alternative. This box is basically plug-and-play...the operating system is delivered in firmware, and you configure and operate the box via a web interface and with a program called RAIDar. </span><span style="font-family:Arial;font-size:85%;">The box speaks a variety of protocols and can talk to Windows, Linux, Macs, and streaming media players so it should get along well with all the servers, workstations, etc. </span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">I bought a diskless version, and populated it with 2x500G Western Digital drives. Initially nothing worked and for a brief time I thought the server was DOA. After a bunch of trial and error I concluded that one of the WD drives was DOA. I brought the box up on 1 drive, configured things, and it just worked. NewEgg RMA'd the bad drive (and even gave me freebie shipping label to send the bad device back...good stuff). </span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">I've got 2 more 500G drives arriving tomorrow - the box is hot-pluggable so in theory installation is simple. It should be interesting to get the box up to 2T with X-RAID and do some performance testing.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Product reviews of the ReadyNAS have been widely varied, but so far all things look positive. I'll post more about the box once I get my bad drive issues sorted out...</span><br /><span style="font-family:Arial;font-size:85%;"></span>ccghttp://www.blogger.com/profile/01309764084115413592noreply@blogger.com2tag:blogger.com,1999:blog-13596339.post-17582414116696040362008-04-05T23:18:00.003-04:002008-04-05T23:36:52.242-04:00United Economy Section not so good for laptops users...<span style="font-family:Arial;font-size:85%;">Thankfully I don't have to travel too much on business - there are plenty of things to do right here in the office most of the time. However, I do get a chance to escape the office now and again, and the Hadoop Summit in California a couple weeks ago was one such opportunity.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">The summit was awesome, you can see some of the notes I took in earlier postings here. I talked to a lot of people, heard a lot of good presentations, and got tons of good information about the Hadoop roadmap, future directions, etc. All very good stuff.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">The point of this post, however, is not to heap praises on the Yahoo! folks for such a great meeting, it's to chat a bit about Economy Class on United Airlines. When I did the on-line check-in thing for the flight from BOS to SFO, United offered me the chance to upgrade to "Economy Plus" for an extra $60 to get more legroom. "No thanks", I clicked and thought nothing further about it. Then I got on the airplane (wasn't it George Carlin who said "Let Evil Kneivel get ON the plane, I am getting IN the plane...", but I digress...). As we reached the altitude "at which it is safe for portable electronic devices to be used", I reach for my laptop...just as the guy ahead of me reclined for what turned out to be a 6 hour snooze across the country. Hmmm, uhhhhh......not enough room between seats to uhhh open the lid on my Lenovo T60. Ugh. So I broke out the pad of paper and pen to jot down some notes and uhhhhhhh not enough room to even write comfortably. </span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Fortunately, the guy sitting next to me turned out to be an interesting fellow starting up a hedge fund, and so I spent the rest of the trip happilly chatting away about everything from technology to politics. </span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">On the return flight I'm nobody's fool. I really really want to get a good 5 hours of work done on the plane (or should that be "in the plane"...) and so I pony up the extra $60 for the Economy Plus seat. It was a night-and-day difference. I flipped open the laptop, stretched out, and used up both batteries writing code. </span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Now I'm not gonna gripe too much about the Welcome-To-Economy-Sorry-About-The-Laptop seat on the flight out...the ticket was cheap enough, particulary for a non-stop flight ($339, BOS->SFO), but I would like to:</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">A. Recommend the hell out of Economy Plus if you have to fly United and you want to work</span><br /><span style="font-family:Arial;font-size:85%;">B. Encourage the airlines to think about the impact such narrow seating has on business travelers</span><br /><span style="font-family:Arial;font-size:85%;">C. Remind myself to get to the gym a little more often...maybe if I skinny up enough the Econony section won't be so bad...</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Given the State of the Airline Industry I am pessimistic about anything happening with respect to (B) above, but I sure do feel better getting this rant off my chest....</span>ccghttp://www.blogger.com/profile/01309764084115413592noreply@blogger.com0tag:blogger.com,1999:blog-13596339.post-36140346991865701292008-03-25T18:23:00.002-04:002008-03-25T23:50:53.317-04:00Hadoop Summit: Internet connectivity..<span style="font-family:arial;font-size:85%;">The conference is rolling along - a lot of great information and good presentations all around. Unfortunately, there has been some network flakiness particularly during the afternoon...so I've stopped trying to blog each talk. </span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">I'll try to summarize some of the more interesting points later...</span>ccghttp://www.blogger.com/profile/01309764084115413592noreply@blogger.com0tag:blogger.com,1999:blog-13596339.post-66224422057794318452008-03-25T16:48:00.001-04:002008-03-25T23:50:53.320-04:00Hadoop Summit: Bryan Duxberry and HBase<span style="font-family:arial;font-size:85%;">Rapleaf is a people search, profile aggregation, and Data API</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Application: custom Ruby web crawler - indexes structured data from profiles. Currently, index page once and then gone forever:</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;"> Internet -> Web Crawler ->Index Processing -> MySQL</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Using HBase to store pages, HBase via REST servlet. Allows: reindexing at a later date, better factored workflow, easier debugging:</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;"> Internet -> Web Crawler -> HBase -> Index Processing -> MySQL Database</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Data similar to webtable: keyed on person no URL, webtable less structured, rapleaf table not focused on links.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Schema: 2 column families: content (stores search and profile pages), meta (stores info about the retrieval [when, how long, who it's about]). Keys are tuples of (person ID, site ID)</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Cluster specs: HDFS/HBase cluster of 16 machines, 2TB disk, 64 cores, 64G of memory</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Load pattern: Approx 3.6T per month (830G compressed), average row size 64K, 14K gzipped</span><br /><span style="font-family:Arial;font-size:85%;">.</span><br /><span style="font-family:Arial;font-size:85%;">Performance: average write time 31 seconds (yikes! accounts for ruby time, base64, sending to servlet, writing to HBase, etc.)...median write time is 0.201 seconds. Max write time 359 seconds. Reads not used much at the moment. Note some of these perf. issues are due to Ruby on GreenThreads instead of native threads, etc....haven't profiled the REST servlet either.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">General Observations: Record compression hurt performance on testing; compressing in client gives a big boost; possible addition to standard HBase client. HBase write logs stored in HDFS which don't exist until closed, which means HBase has durability issues. (this will be resolved by Hadoop-1700).</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;"></span>ccghttp://www.blogger.com/profile/01309764084115413592noreply@blogger.com0tag:blogger.com,1999:blog-13596339.post-89892726287989772222008-03-25T16:17:00.003-04:002008-03-25T23:50:53.321-04:00Hadoop Summit: Michael Stack and HBase<span style="font-family:arial;font-size:85%;">HBase: Distributed database modelled on B igTable (Google)</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Runs on top of Hadoop core</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Column-store: Wide tables cost only the data sotred, NULLs in rows are "free", columns compress well</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Not a SQL database: no joins, no transactions, no column typing, no sophisticated query engine, no SQL, ODBC, etc.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Use HBase: Scale is large; access can be basic and mostly table scans.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">The canonical use case is web-table: table of web crawls keyed by URL, columns of webpage, parse, attributes, etc.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Data Model: Table of rows X columns; timestamp is 3rd dimension</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Cel uninterpretted byte arrays</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Rows can be any Text value, e.g. a URL: rows are ordered lexicographically, row writes are atomic</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Columns grouped into column families: have prefix and a qualifier (attribute: mimetype, attribute: language)</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Members of column family group have similar charcter/access</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Implementation: Client, a Master, and one or more region servers (analagous to slaves in Hadoop).</span><br /><span style="font-family:Arial;font-size:85%;">Cluster carries 0->N Tables</span><br /><span style="font-family:Arial;font-size:85%;">Master assigns table regions to region servers</span><br /><span style="font-family:Arial;font-size:85%;">region server vcarries 0->N regions: region server keeps write-ahead log of every update; used recovering lost regionservers; edits first go to WAL, then to Region</span><br /><span style="font-family:Arial;font-size:85%;">Each region is made of MemCache and Stores</span><br /><span style="font-family:Arial;font-size:85%;">Row CRUD: client initially goes to Master for row loaction; client caches and goes direct to region server therafter; on fault returns to master to freshen cache</span><br /><span style="font-family:Arial;font-size:85%;">When regions get too big they are split: regionserver manages split, master is informed parent is off-lined, new daughters deployed.</span><br /><span style="font-family:Arial;font-size:85%;">All data persisted to HDFS</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Connecting: java is first -class client; Non-Java clients: thrift server hosting hbase client instance (ruby, c++, java). Also REST server hosts hbase client (ruby gem, Active Record via RESt).; SQL-like shell (HQL); TableInput/OutputFormat for map/reduce</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">History: 2006 Powerset interest in Bigtable; 02/2007 Mike Cafarella provides initial code drop - cleaned up by powerset and added as Hadoop contrib; First usable HBase in Hadoop 0.15.0; 01/2008 HBase subprojects of of Hadoop; Hadoop 0.16.0 incorporates into code base</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">HBase 0.1.0 release candidate is effectively Hadoop 0.16.0 with logs of fixes...HBase now stands outside of Hadoop contrib.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Focus on developing use/developer base: 3 committers, tech support (other people's clusters via VPN), 2 User Group meetings - more to follow; working on documentation and ease-of-use.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Performance: Performance good and improving</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Known users: powerset and rapleaf, worldlingo, wikia</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Near Future: release 0.2.9 to follow release of Hadoop 017.0, theme robustness and scalability - rebalancing of regions of cluster, replace HQL with jirb, jython, or beanshell, repair and debugging tools.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">HBase and HDFS: No appends in HDFS: data loss...WAL is useless without it, Hadoop-1700 making progress</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">HBase usage pattern is not same as Map/Reduce - random reading and keep files open raises "too many open files" issues</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">HBase or HDFS errors: Hard to figure without debug logging enabled</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Visit hbase.org: mailing lists, source, etc.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><br /><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;"></span>ccghttp://www.blogger.com/profile/01309764084115413592noreply@blogger.com0tag:blogger.com,1999:blog-13596339.post-69345728062975201532008-03-25T14:50:00.002-04:002008-03-25T23:50:53.321-04:00Hadoop Summit: Ben Reed and Zookeeper<span style="font-family:Arial;font-size:85%;">Zookeeper: General, robust coordination service...</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Observations: Distributed systems always need some form of coordination. Programmers cannot use locks correctly (note that google uses "chubby" lock service): you can learn locks in 5 minutes, but spend a lifetime learning how to use them properly; Message based coodination can be hard to use in some apps.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Wants: Simple, robust, good performane; tuned for read-dominant workloads; familiar models and interfaces, wait-free: failed client wil not interfere with requests of a fast client; need to be able to wait efficiently.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Design starting point: start with File API and strip out what we don't need: partial writes/reads, name. Add what is needed: ordered upates and strong persistance guarantees, conditional updates, watched for data changes, ephemeral nodes (client create files, if client goes away the file goes away), generated file names (i.e. mktemp()).</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Data Model: Hierarchical namespace, each znode has data and children, data is read and written in its entirety</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Zookeeper API: create(), delete), setData(), getData(), exists(), getChildren(), sync()...also some security APIs ("but nobody cares about security"...). getChildren() is like "ls".</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Create flags: epheeral - the znode gets deleted with the session that created it times out; sequence the path name will have a monotonically increasing counter relative to the parent appended.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">How Zookeeper works: Service made up of a set of machines. Each server stores a full copy of data in-memory....gives very low latency and high throughput. A leader is elected at startup, the rest of the servers become followers. Followers service clients, all updates go through leader. Update responses are sent when a majority of servers agree.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Example of Zookeeper service: Hadoop on Demand. I got distracted by something during this part of the presentation and didn't get all the basic pieces here, but this uses Zookeeper to track interaction with Torque, and handles graceful shutdown, etc.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Status: Project started in 2006, prototyped in fall 2006, initial implementation March 2007, code moved to zookeeper.sourceforge.net and Apache License November 2007. Everything is pure Java: quorum version and standalone version. Clients are Java and C clients.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><br /><span style="font-family:Arial;font-size:85%;"></span><br /><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;"></span>ccghttp://www.blogger.com/profile/01309764084115413592noreply@blogger.com0tag:blogger.com,1999:blog-13596339.post-42352089996432087822008-03-25T14:19:00.004-04:002008-03-25T23:50:53.321-04:00Hadoop Summit: Andy Konwinski & X-trace<span style="font-family:arial;font-size:85%;">Monitoring Hadoop using X-trace: Andy Konwinski is from UC Berkeley RAD Lab</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Motivation: Hadoop style processing masks failures and performance problems</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Objectives: Help Hadoop developers debug and profile Hadoop; help operators monitor and optimize Map/Reduce jobs</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">[CCG: This is cool, we REALLY need this stuf NOW]</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">RAD Lab: Reliable Adaptive Distributed Systems </span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Approach: Instrument Hadoop using X-Trace framework. Trace Analysis: virtualizatioin via web-based UI; statistical analysis and anomaly detection</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">So what's X-Trace: Path-based tracing framework; generate event graph to capture causality of events across network (RPC, HTTP, etc.). Annotate message with trace metadata carried along execution path (instrument protocol APIs and RPC libraries). Within Hadop the RCPC library has been instrumented.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">DFS Write Message Flow Example: client -> DataNode1 -> DataNode2 -> DataNode3</span><br /><span style="font-family:Arial;font-size:85%;">Report Graph Node: Report Label, Trace ID#, Report ID3, Hostname, timestamp <key></span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">builds a graph which can then be walked.</span><br /><br /><span style="font-family:Arial;font-size:85%;"><begin></span><span style="font-family:Arial;font-size:85%;">[CCG: again you need to see the picture, but you get the idea right?]</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Andy showed some cool graphs representing map/reduce operations in flight...</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Properties of Path Tracing: Deterministic causality and concurrence; control over which events get traced; cross-layer; low-overhead; modest modification of the app source code ( <><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Architecture: Xtrace front ends on each Hadoop node, communicates with Xtrace backend via TCP/IP, backend stores data using BDB, trace analysis web UI communicats with Xtrace backend. Also cool fault detection programs can be run - interact with backend via HTTP.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Trace Analysis UI: "we have pretty pictures which can tell you a lot": perf stats, graphcs of utilization, critical path analysis...</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Applications of Trace Analysis: Examined perf of Apache nutch web indexing engine oin a Wikipedia cral. Time to create an inverted link index of a 50G crawl - with default configuration, ran in 2 hours.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Then by using trace analysis, was able to make changes to run same workload in 7 minutes. Used workload analysis to determine that one single reduce task which actually fails several times at the beginning: 3 ten minute timeouts. Bumped max reducers to 2 per node, and dropped execution time to 7 minutes.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Behold the power of pretty pictures!</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Off-line machine learning: faulty machine detection, buggy software detection. Current work oin graph processing and analysis.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Future work: Tracing more production map/reduce applications. More advanced trace processing tools, migrating code into Hadoop codebase.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><br /></span><span style="font-family:Arial;font-size:85%;"></span><span style="font-family:Arial;font-size:85%;"></span>ccghttp://www.blogger.com/profile/01309764084115413592noreply@blogger.com0tag:blogger.com,1999:blog-13596339.post-25472134913905831972008-03-25T13:47:00.002-04:002008-03-25T23:50:53.322-04:00Hadoop Summit: Michael Isard and DryadLINQ<span style="font-family:arial;font-size:85%;">Michael Isard is from Microsoft Research:</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">"Dryad: Beyond Map-Reduce"</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">What is Map-Reduce: An implementation, a computational model, a programming model.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Implementation Performance: Literally map, reduce, and that's it...reducers write to replicated storage. Complex jobs require pipeline multipl stages....no fault tolerance between stages. Output of Reduce: 2 network copies, 3 disks</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Computational Model: Join combines inputs of diff types, "split" produced outputs of different types. This can be done with map-reduce, but leads to ugly programs. Hard to avoid performance penalty described above. Some merge-joins are very expensive. Finally, baking in more operators adds complexity.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Dryad Middleware Layer: Address flexibility and performance issues, more generalized than map-reduce, interface is more complex.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Computational Model: Job is a DAG, each node takes any number of inputs and produces any number of outputs (you need to see the picture). </span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">DAG Abstraction Layer: Scheduler handles arbitrary graphs independent of vertext semantics, simple uniform state machine for scheduling and fault-tolerance. Higher levels build plan from application code: Layers isolated, many optimizations are simple graph manipulations, graph can be modified at runtime.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">MapReduce Programming Model: opaque <key,value> pairs flexible. Front-ends like sawzall and pig help, but domain specific simplifications limit some applications. </span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Moving beyond simple data-mining to machine learning, etc.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">LINQ: Extensions to .Net in Visual Studio in 2008...general purpose data-parallel programming constructs. Data elements are arbitrary .NET types, combined in generalized framework</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">DryadLINQ: Automagically distribute a LINQ program; some Dryad-specific extensions: same source program runs on single-core to multi-core to cluster. Execution model depends on data source.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">LINQ designed to be extensible, LINQ+C# provides parsing, thype checking. LINQ builds expressioin tree. Root provider class called on evaluation (has access to entire tree, reflection allows very powerful operations). Can add custom operators.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">PLINQ - Running Queries on Multi-Core Processors - parallel implementation of LINQ.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">SQL server to LINQ: cluster computers run run SQL Server. Partitioned tables in local SQLServer DBs. DryadLINQ process can use "SQL to LINQ" provider - "best of both worlds".</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Continuing research:L Applicatioinlevel research (what can we do), system level research (how can we improve performance), LINQDB?</span>ccghttp://www.blogger.com/profile/01309764084115413592noreply@blogger.com0tag:blogger.com,1999:blog-13596339.post-89653708850967448712008-03-25T12:59:00.002-04:002008-03-25T23:50:53.322-04:00Hadoop Summit: Kevin Beyer and JAQL...<span style="font-family:Arial;font-size:85%;">Kevin Beyer is from IBM Almaden Research Center</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">JAQL (pronounced "jackel") - Query Language for JSON</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;"> JSON: JavaScript Object Notation: simple, self-describing and designed for data</span><br /><span style="font-family:Arial;font-size:85%;"> Why: want complete entity in one place, support schema that vary or evolve over time; standard: used in web 2.0 applications, bindings available for many languages, etc.; not XML - XML designed for doc markup, not for data (hooray, thanks for saying THAT).</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">JAQL processes any data that can be interpreted as JSON (JSON text files, binary, CSV, etc.). Internally, JAQL processees binary data-structures.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">JAQL similar to Pig Latin, goal is to get JAQL accepted anywhere JSON might be used (document analytics, doc management [couchDB], ETL, Mashups,....</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Immediate Goal: Hide grungy details of writin map-reduce jobs for ETL and analysis workloads; compile JAQL queries into map/reduce jobs.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">JAQL Goals: designed for JSON data, functional query language (few side effects, not a scripting language - set-oriented, highly transformed) , composable expressions, draws on other languages, operator plan -> query (rewrites are transforms within the language, any plan is representabke in the language itself).</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Core operations: iterations, grouping, joining, combining, sorting, projection, constructors for arrays, records, atomic values, "unnesting", function definition/evaluation.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Some good examples were presented that are too long to type in...wait for the presentations to appear on-line I guess...sorry. Good stuff though, I am liking the language presented more than Pig Latin.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">ImplementationL JAQL input/output designed for extensibility...basically reads/writes JSON values. Examples: Hadoop InputFormat and OutputFormat. </span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Roadmap: Another release is imminent, next release this summer (open the source, indexing support, native callout, server implementation with a REST interface).</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><br /><span style="font-family:Arial;font-size:85%;"></span>ccghttp://www.blogger.com/profile/01309764084115413592noreply@blogger.com0tag:blogger.com,1999:blog-13596339.post-89633616783394510502008-03-25T12:33:00.003-04:002008-03-25T23:50:53.323-04:00Hadoop Summit - Christopher Olsten and PIG<span style="font-family:arial;font-size:85%;">I saw Chris talk about Pig at MIT a few weeks ago...this looks like the same presentation..</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Example: Tracking users who visit good pages (pagerank type thing). Typical operations: loading data, canonicalizing, database JOIN-type operation, database GROUP BY operation, leading to a something which computes the pagerank. </span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">note: the drawings within the presentation make the above very clear. Striping visits and pages across multipe servers, highly parallel processing..fairly straightforward approach.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">But using just map/reduce: Write the join yourself (ccg - been there done that, thanks Joydeep Sen Sharma for getting me started). Hadoop users tend to share code among themselves for doing JOINs, and how best to do the join operation, etc. In short, things get ugly in a hurry - gluing map/reduce jobs together, etc. You have to do a lot of low-level operations by hand, etc. It's potentially hard to understand and maintain code.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">So: A data flow language could easily synthesize map/reduce sequences</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">PIG accomplishes this, using a dataflow language called Pig Latin. Essentially a terse language where each step is loading data, doing some sort of filtering/canonical operation, or doing custom work (via map/reduce). Operators includ: FILTER, FOREACH, GENERATE, GROUP, JOIN, COGROUP, UNION. Also support for sorting, splitting, etc. Goal is a very simple language to do powerful things.</span><br /><span style="font-family:Arial;font-size:85%;"><br />Related languages:<br /> SQL - declarative language (i.e. what, not how).<br /> Pig Latin: Sequence of simple steps - close to imperative/procedural programming, semantic order of operations is obvious, incremental construction, debug by viewing intermediate results, etc.<br /><br /> Map/Reduce: welds together primitives (process records - > create groups -> process groups)<br /> Pig Latin: Map/Reduce is basically a special case of Pig, Pig adds built-in primitives for most-used transformations.<br /><br />So: Is Pig+Hadoop a database system ? Not really.....<br /><br />Workload: DBMS does tons of stuff, P+H does bulk reads/writes only...just sequential scans<br />Data Representation: DBMS controls format...must predeclare schema, etc, Pigs eat anything :-)<br />Programming Style: DMBS - system of constraints, P+H: sequence of steps<br />Custom Processing: DBMS: functions second class to logic expressions, P+H: Easy to extend<br /><br />Coming Soon to Pig: Streaming (external executables), static type checking, error handling (partial evaluation), development environment (Eclipse).<br /><br /><br /> </span><br /></span><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;"></span>ccghttp://www.blogger.com/profile/01309764084115413592noreply@blogger.com0tag:blogger.com,1999:blog-13596339.post-51890194368044567702008-03-25T11:58:00.005-04:002008-03-26T00:55:15.124-04:00Hadoop Summit: Doug Cutting and Eric Baldeschweiler<span style="font-family:arial;font-size:85%;"><em>Hadoop Overview - Doug Cutting and Eric Baldeschwieler</em></span><br /><span style="font-family:Arial;font-size:85%;"><em></em></span><br /><span style="font-family:arial;font-size:85%;">Doug Cutting - pretty much the father of Hadoop gave an overview of Hadoop history. Interesting comment was that Hadoop has achieved web-scale in early 2008...</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Eric14 (Eric B...): Grid computing at Yahoo. 500M unique users per month, billions of interesting events per day. 'Data analysis is the inner loop" at Yahoo. </span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Y's vision and focus: On-demand shared access to vast pool of resources, support for massively parallel execution. , Data Intensive Super Computer (DISC), centrally provisioned and managed, service oriented...Y's focus is not grid computing in in terms of Globus, etc., not focused on external usage ala Amazon EC2/S3. Biggest grid is about 2,000 nodes.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Open Source Stack: Commitment to open source developent, Yahoo is an Apache Platinum Sponsor</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Tools used to implement Yahoo's grid vision: Hadoop, Pig, Zookeeper (high avail directory and config sevices), Simon (cluster and app monitoring).</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Simon: Very early days, internal to Yahoo right now...similar to Ganglia "but more configurable". Highly configurable aggregation system - gathering data from various nodes to produce (customizable?) reports. </span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">HOD - Hadoop On Demand. Current Hadoop scheduler currently FIFO - jobs will run in parallel to the extent that the previous job doesn't saturate the node. HOG is built on Torque (<a href="http://www.clusterresources.com/">http://www.clusterresources.com/</a>) to build virtual clusters, separate file systems, etc. Yahoo has taken development about as far as they want...cleaning up code, etc. Future direction for Yahoo is to invest more heavily in the Hadoop scheduler. Does HOG disrupt data locality - yup, it does. Good news: Future Hadoop work will improve rack locality handling significantly.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Hadoop, HOD, Pig all part of Apache today,</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Multiple grids inside Yahoo: tens of thousands of nodes, hundreds of thousands of cores, TBs of memory, PBs of disk...ingests TBs of data daily. </span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">M45 Project: Open Academic Cluster in collaboration with CMU: 500 nodes, 3TB RAM, 1.5P disk, high bandwith located conveniently in a semi-truck trailer</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Open source project and Apache: Goal is for Hadoop to remain a viable open source project. Yahoo has invested heavily...very excited to see additional contributors and commiters. "Yahoo is very proud of what we've done with Hadoop over the past few years." Interesting metric: Megawatts of Hadoop</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Hadoop Interest Growing: 400 people expressed interest in today's conference, 28 organizations registered their Hadoop usage/cluster, in use in universities on multple continents, Y is now started hiring employees with Hadoop experience.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">GET INVOLVED: Fix a bug, submit a test case, write some docs, help out!</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Random notes: More than 50% conf attendees running Hadoop, many with grids more than 20 nodes, and several with grids > 100 nodes.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Yahoo just announced collaboration with Computational Research Labs (CRL) in India to "jointly support cloud computing research"...CRL runs EKA - the 4th fastest supercomputer on the planet.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;"></span>ccghttp://www.blogger.com/profile/01309764084115413592noreply@blogger.com0tag:blogger.com,1999:blog-13596339.post-71120287984991276352008-03-25T11:40:00.004-04:002008-03-25T23:50:53.324-04:00Hadoop Summit - 25-March-2008 8:45AM<span style="font-family:arial;font-size:85%;">Huge crowd, various Yahoo celebrities like JeremyZ and EricB, Doug Cutting floating around...it's fun to put faces to names with the various mailing list participants. Looks like good network access and there are even plug strips lying around everywhere to plug in laptops. Looking forward to hearing some excellent presentations...</span>ccghttp://www.blogger.com/profile/01309764084115413592noreply@blogger.com0tag:blogger.com,1999:blog-13596339.post-26009645492633853522008-03-22T00:34:00.002-04:002008-03-25T23:50:53.324-04:00Hadoop Summit 25-March-2008<span style="font-family:Arial;font-size:85%;">I'm packing my bags for the 6-1/2 hour flight from right coast to left for the <a href="http://upcoming.yahoo.com/event/436226/">Hadoop Summit </a>next week. This promises to be a great event with a lot of good material on the agenda, plus the opportunity to chat with Hadoop contributors and application developers as well. I just wish it was longer than 1 day...</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">There are 215 registered attendees - which seems to be a larger number than Yahoo anticipated when the event was initially announced - </span><span style="font-family:Arial;font-size:85%;">so there is evidently significant interest in Hadoop despite map/reduce technology being declared <a href="http://www.databasecolumn.com/2008/01/mapreduce-a-major-step-back.html">"a major step backwards"</a> by <a href="http://www.databasecolumn.com/2007/09/stonebraker.html">Mike Stonebraker </a>and <a href="http://www.databasecolumn.com/2007/09/dewitt.html">Dave DeWitt</a>. Can't we all just get along?</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">I anticipate having network access at the summit, so I'll try to blog about the presentations and discussions throughout the day.</span>ccghttp://www.blogger.com/profile/01309764084115413592noreply@blogger.com0tag:blogger.com,1999:blog-13596339.post-41576233030067977332008-03-16T00:51:00.003-04:002008-03-16T01:26:59.088-04:00Compete acquired by Taylor Nelson Sofres<span style="font-family:arial;font-size:85%;"><a href="http://www.compete.com/">Compete</a>, a company I worked for as Chief Software Architect, was just recently acquired by <a href="http://www.tnsglobal.com/">Taylor Nelson Sofres</a>. This is a remarkable achievement for Compete and a great deal for both Compete and TNS. If you're interested, you can <a href="http://blog.compete.com/2008/03/03/tns-acquires-compete/">read the official postings </a>about the acquisition. Congratulations to everyone at Compete and TNS! </span><br /><span style="font-family:arial;font-size:85%;"><br />Learning about this acquisition made me feel very proud and quite nostalgic. It was back in April 2001 that I got involved with Compete. I had wrapped up a long contract at Oracle, spent a month in India with my wife visiting her extended family, and was back in Nashua, NH contemplating the irrational exuberance of what we would all soon call "dot bomb." I had been contracting for a while, and while that was good fun I felt it was time to do something a bit more daring/challenging.<br /><br />One thing led to another and I found myself in Boston, on oh-so-chic Newbury Street of all places, sitting in a literally transparent office on the upper floors of a converted church building, discussing a company called Compete with its CTO (and now my good friend) <a href="http://davidcancel.com/">David Cancel</a>. I wound up joining Compete a few weeks later as its Chief Software Architect, a position that I held for almost 5 years. I think I was the 11th or 12th person Compete hired. </span><span style="font-family:arial;font-size:85%;">Compete rode the waves of dot com and dot bomb, and eventually found its way to become the strong player it is today, thanks to its excellent <a href="http://www.competeinc.com/about_compete/management/">management team</a> and a lot of hard work by everybody.</span><br /><span style="font-family:arial;font-size:85%;"><br />Fast forward 7 years and Compete is now a recognized industry force - an established player with a work force of close to 100 extremely talented and dedicated individuals. Expect more great things from this company in the future!</span><br /><span style="font-family:Arial;font-size:85%;"></span>ccghttp://www.blogger.com/profile/01309764084115413592noreply@blogger.com0tag:blogger.com,1999:blog-13596339.post-88897137199758741322007-01-14T02:35:00.000-05:002007-01-14T03:12:55.494-05:00The Grid in My Basement, part 3: That Sinking Feeling<span style="font-family:trebuchet ms;font-size:85%;">Size matters. At least when you are building rackmount machines. Of course were I not suffering from sleep deprivation when I made my hardware purchasing decisions I would have realized that you can't put a MASSIVE heat sink into a tiny space, but such is life.</span><br /><span style="font-family:Trebuchet MS;font-size:85%;"></span><br /><span style="font-family:Trebuchet MS;font-size:85%;">Anyway, the <a href="http://www.newegg.com/product/product.asp?item=N82E16835106069">very spiffy Blue Orb II CPU cooler </a>is never ever gonna fit in the 2U case I bought. That was evident by inspection before I even unpacked the coolers. Had I done my homework on the motherboard and case dimensions I would have realized that a package with a combined fan + heatsink height of 90.3mm would never fit. Not only that, but the heatsink has length and width dimensions of 140x140 which means it might not fit the motherboard at all. There's a huge row of capacitors next to the retention module base, and the DIMM sockets are proximate on the other side. This is all badness from the perspective of installing a massive heatsink.</span><br /><span style="font-family:Trebuchet MS;font-size:85%;"></span><br /><span style="font-family:Trebuchet MS;font-size:85%;">So with a heavy sigh I file for my first RMA from Newegg and package the Orbs for shipment back. Bummer drag, they looked so cool too. So I start looking for an appropriate K8 heatsink for my new nodes, and the fun really begins.</span><br /><span style="font-family:Trebuchet MS;font-size:85%;"></span><br /><span style="font-family:Trebuchet MS;font-size:85%;">First, you may be wondering why I didn't use the cooler that came with the CPU when I bought it. Well, in order to save money I bought the OEM version of everthing I could. That eliminates a lot of unnecssary packaging, instruction manuals, and in some cases features - like the CPU cooler on my AMD X2s. So I need to buy a cooler on the aftermarket. </span><br /><span style="font-family:Trebuchet MS;font-size:85%;"></span><br /><span style="font-family:Trebuchet MS;font-size:85%;">The assumption that manufacturers make is that you WILL overclock if you are buying an aftermarket cooler. Therefore, the heatsinks reflect this assumption and most are massive. Looking at heatsinks in close up is sorta like looking at big scary machinery. Pipes and tubes run in all directions, massive banks of fins jut out at weird angles and rise up toward to sky towering over the motherboard. None of these devices are particularly well suited for the tight space of a 2U (or heaven help you a 1U) case. </span><br /><span style="font-family:Trebuchet MS;font-size:85%;"></span><br /><span style="font-family:Trebuchet MS;font-size:85%;">I start shopping around for low-profile CPU coolers for 2U cases and run into several problems. First, there aren't too many cooler vendors out there that make this stuff. Second, the ones that do aren't terribly interested in Socket 939 applications. Third, the low-profile stuff tends to be crazy expensive - $95 for a low-profile heat sink and fan? No thanks...</span><br /><span style="font-family:Trebuchet MS;font-size:85%;"></span><br /><span style="font-family:Trebuchet MS;font-size:85%;">So I pick up a ruler, open up the case, and start measuring. And measuring. After a good deal of plotting, I calculate that my heat sink can be no more than 70 x 70 x 65mm. And then I start shopping. And shopping.</span><br /><br /><span style="font-family:Trebuchet MS;font-size:85%;">Finally, after literally 2 evenings wasted googling around, I hit on a cooler/heatsink sold by ASUS - the same manufacturer that makes the motherboard I am using. I look at the height dimension and am psyched - the combined total of both devices is only 55mm tall! The bad news is that the heatsink runs 77 x 68 x 40mm - meaning that it's too big potentially. I look on the ASUS website (sidebar your honor: remind me to rant and rant later about web sites that provide everything BUT the information you need) and find nothing helpful regarding compatibility with their own motherboards. </span><br /><br /><span style="font-family:Trebuchet MS;font-size:85%;">So I reason as follows: The height dimension will fit just fine; the heatsink will probably fit an ASUS motherboard since ASUS makes both; the absence of a compatibility list means it's compatible with all their offerings or somebody is just lazy. So I bite the bullet and order up the <a href="http://www.newegg.com/Product/Product.asp?Item=N82E16835101202">ASUS Crux K8 MH7S 70mm Hydraulic CPU Cooling Fan with Heatsink</a> and hope for the best. </span><br /><span style="font-family:Trebuchet MS;font-size:85%;"></span><br /><span style="font-family:Trebuchet MS;font-size:85%;">2 days later I get the parts, and a couple days after that I open up the build-in-progress machine and install the heatsink. Have I mentioned how stressful putting a heatsink in can be? I mean there you are with all this expensive hardware that looks pretty darn fragile, and you are pushing down on it with no small amount of force trying to more-or-less permanently mate the CPU to the heatsink. Every time I do this I expect the motherboard to crack or something equally as awful.</span><br /><br /><span style="font-family:Trebuchet MS;font-size:85%;">Good news! The new cooler fits perfectly. It clears the lid of the case beautifully, and the dimensions of both heatsink and cooler are within the perimter of the retention module.</span>ccghttp://www.blogger.com/profile/01309764084115413592noreply@blogger.com0tag:blogger.com,1999:blog-13596339.post-71968906249121801462007-01-14T01:23:00.000-05:002007-01-14T02:30:12.149-05:00The Grid in my Basement, part 2: Parts is Parts...<span style="font-family:Trebuchet MS;font-size:85%;">When I began this project the original goal was "reasonable performance for $500 per machine". That is turning out to be a bit of a challenge, especially since I decided not to cut corners on the rackmount chassis. There is nothing like working in a case for an hour and emerging with half a dozen cuts to your hands from rough edges to cost justify a clean well-made chassis. Further challenging the $500 bottom line was the desire to run either a dual core or dual CPU configuration.</span><br /><span style="font-family:Trebuchet MS;font-size:85%;"></span><br /><em><span style="font-family:Trebuchet MS;font-size:85%;"><strong>Form Factor, Topology, etc.</strong></span><br /></em><span style="font-family:Trebuchet MS;font-size:85%;">Socket 939 is fading away, and my research showed me that prices for 939 gear were fading similarly. So as a money saving technique, I decided to actively seek out socket 939 hardware for this project. I also decided to focus on a good quality motherboard while not necessarily using a server motherboard...this may turn out to be a poor decision - we'll see once things are up and running. After reviewing the data sheets and specs on a number of motherboards I decided to use an ATX form factor. </span><br /><span style="font-family:Trebuchet MS;font-size:85%;"></span><br /><strong><em><span style="font-family:Trebuchet MS;font-size:85%;">Performance and Cost Considerations</span><br /><span style="font-family:Trebuchet MS;font-size:85%;"></span></em></strong><br /><span style="font-family:Trebuchet MS;font-size:85%;">I want good performance without breaking the bank. While a sweet dual core/dual core system with tons of memory and a massive SCSI array would make me smile, it would put the project way beyond budget. So here are the tradeoffs I made:</span><br /><br /><ul><li><span style="font-family:Trebuchet MS;font-size:85%;">Running a single dual-core CPU instead of dual CPUs with dual cores. This means that each box will only be 2-way instead of 4-way, then again with 4 nodes running a clustering tool like <a href="http://openmosix.sourceforge.net">openMosix</a> I will have an 8-way box which is still pretty cool.<br /></span></li><li><span style="font-family:Trebuchet MS;font-size:85%;">Have you noticed how pricey memory is lately? I'll start out with 1G per node, but make sure that my motherboard can support at least 4G for future expansion. Note to self: hoard memory later when it's cheap and make a killing on eBay when prices go up again.<br /></span></li><li><span style="font-family:Trebuchet MS;font-size:85%;">SAS or SCSI gives killer I/O performance but at a price. I'll build these machines with SATA-II devices in the 250-320G range, perhaps spending a little more for a larger on-device cache.<br /></span></li><li><span style="font-family:Trebuchet MS;font-size:85%;">My original plan was to build a blade server system, using DIY parts from a vendor like <a href="http://atxblade.com/Rackmt/ATXBladeS100.htm">ATXBlade</a>. But in analyzing the cost - $550 for blade storage unit, $325 for each blade chassis - I decided that I didn't really need to build a dense server farm. After all, I have a full height rack and will probably not build out enough systems to exceed the capacity of the rack.</span></li></ul><span style="font-family:Trebuchet MS;font-size:85%;"><strong><em>Parts Manifest</em></strong></span><br /><span style="font-family:Trebuchet MS;font-size:85%;">After a significant amount of consideration, here is the parts manifest for each node:</span><br /><br /><ul><li><span style="font-family:Trebuchet MS;font-size:85%;">Motherboard: <a href="http://www.newegg.com/product/product.asp?item=N82E16813131569">ASUS A8N5X Socket 939 NVIDIA nForce4 ATX AMD motherboard </a>- retail packaging</span></li><li><span style="font-family:Trebuchet MS;font-size:85%;">CPU: AMD Athlon 64 X2 4400+ (Toledo code) 2.2GHz processor - OEM packaging</span></li><li><span style="font-family:Trebuchet MS;font-size:85%;">Memory: <a href="http://www.newegg.com/product/product.asp?item=N82E16820141307">Kingston ValueRAM 1G 184 pin (PC3200) DDR400 memory </a>- retail packaging</span></li><li><span style="font-family:Trebuchet MS;font-size:85%;">Disk: <a href="http://www.newegg.com/product/product.asp?item=N82E16822136003">Western Digital Caviar SE16 320G 7200RPM SATA 3.0Gb/s hard drive </a>- OEM packaging</span></li><li><span style="font-family:Trebuchet MS;font-size:85%;">CPU cooler: <a href="http://www.newegg.com/product/product.asp?item=N82E16835106069">Thermaltake CL-P0257 "Blue Orb II" CPU cool for K8 </a>- retail packaging</span></li></ul><span style="font-family:Trebuchet MS;font-size:85%;">Here is parts manifest for the chassis:</span><br /><br /><ul><li><span style="font-family:Trebuchet MS;font-size:85%;">Case: <a href="http://www.newegg.com/product/product.asp?item=N82E16856999304">iStarUSA Storm Series D-200 Black 2U rackmount case </a>- retail packaging</span></li><li><span style="font-family:Trebuchet MS;font-size:85%;">Rails: iStarUSA TC-RSL-20 sliding rail kit for rackmount chassis - retail packaging</span></li></ul><span style="font-family:Trebuchet MS;font-size:85%;">The links above are all to Newegg.com product pages, because that's where I bought everything. The CPU I spec above is presently not available, but <a href="http://www.newegg.com/Product/Product.asp?Item=N82E16819103543">this CPU </a>has very similar specs.</span><br /><br /><span style="font-family:trebuchet ms;font-size:85%;">Motherboard selection was driven by the following selection criteria:</span><br /><ul><li><span style="font-family:Trebuchet MS;font-size:85%;">Socket 939, ATX form factor</span></li><li><span style="font-family:Trebuchet MS;font-size:85%;">Able to support AMD Athlon X2</span></li><li><span style="font-family:Trebuchet MS;font-size:85%;">At least 4GB memory</span></li><li><span style="font-family:Trebuchet MS;font-size:85%;">Support for at least 4 SATA-II devices</span></li><li><span style="font-family:Trebuchet MS;font-size:85%;">On-board RAID support for RAID-0, RAID-1, RAID 0+1, and JBOD</span></li><li><span style="font-family:Trebuchet MS;font-size:85%;">On-board video support</span></li><li><span style="font-family:Trebuchet MS;font-size:85%;">Front Side Bus speed of at least 1000MgHz</span></li><li><span style="font-family:Trebuchet MS;font-size:85%;">On-board gigabit network support</span></li></ul><p><span style="font-family:Trebuchet MS;font-size:85%;">The up-to-speed reader will note that the motherboard I chose does not come with on-board video support. I noticed that too - AFTER I had ordered the motherboards. There is a whole story behind this that I'll write down later. There is also a question about SATA performance - some spec sheets state the motherboard is SATA-I (1.5Gb/sec) while other spec sheets state it's SATA-II (3.0Gb/sec). I think the board was rev'd at some point and this may have been part of the rev. At any rate, if it turns out to be SATA-I the I can still do some benchmarking and perhaps install a SATA-II card later.</span></p><p></p>ccghttp://www.blogger.com/profile/01309764084115413592noreply@blogger.com0tag:blogger.com,1999:blog-13596339.post-58487471508357882472007-01-04T01:47:00.000-05:002007-01-14T02:29:50.599-05:00The Grid in My Basement, part 1<span style="font-family:Trebuchet MS;font-size:85%;">The name of the game is parallelism...in short: take apart a problem, break it up into independent pieces, and run as many of those independent pieces at once on separate computers (well at least on separate CPUs). This is nothing new...parallel computing has been around since Cro-Magnon Geek solved problems by dropping boxes of punch cards bearing almighty FORTRAN into card reading machines and then lurking impatiently in the line printer room for pages of greenbar while multi-gazillion dollar <a href="http://www.oref.co.uk/images/mainframe.jpg">CPUs the size of refrigerators </a>cogitated about his fast fourier transforms and what not. What is compelling in THIS century is that you can do it cost effectively. Ok, not even cost effectively - downright cheap.</span><span style="font-family:arial;font-size:85%;"><br /></span><span style="font-family:arial;font-size:85%;"><br /></span><span style="font-family:Trebuchet MS;font-size:85%;">In cheapest form, parallel computing has become nearly free. Witness the <a href="http://www.amazon.com/gp/browse.html?node=201590011">Elastic Compute Cloud </a>over at Amazon (buzz kill: ECC is in beta and they aren't accepting new users at the moment; double plus bad: the largest number of machines you can have is something like 20) where you can rent time on virtual machines for cents per hour. In more expensive forms parallel computing is, well, still pricey. If you're a high energy physicist or financial type with a big <a href="http://en.wikipedia.org/wiki/Grid_computing">grid</a> and you are running name brand hardware you are putting up some mighty big dollars.</span><span style="font-family:arial;font-size:85%;"><br /></span><span style="font-family:arial;font-size:85%;"><br /></span><span style="font-family:Trebuchet MS;font-size:85%;">So I find myself - as an employed practitioner wanting to test my <a href="http://www.vertica.com">employer's</a> new software, and as an entreprenuer designing and trying new concepts in search of the Next Big Geeky Thing - wanting to have access to my very own grid. Thus is born the idea of the Grid in my Basement, or as I prefer to call it: The Data Basement.</span><span style="font-family:arial;font-size:85%;"><br /></span><span style="font-family:arial;font-size:85%;"><br /></span><span style="font-family:Trebuchet MS;font-size:85%;">My plan is to build and deploy a basic grid capable of doing real work in a cost effective manner. I want to accomplish this with some fairly real-world parameters, so I need real computing horsepower. So after hours and hours of combing through CPU specs, motherboard specs, CPU cooler specs (naw, I would NEVER overclock...), power supply specs and the like I now have boxes and boxes of cool stuff en route from my good friends at <a href="http://newegg.com/">NewEgg </a>(what self-respecting techno-weenie doesn't love newegg?). Over the next few weeks I'll write about the Data Basement as it gets built out and evolves into something useful. With any luck I'll also publish a tutorial with photos about rolling your rackables.</span><span style="font-family:arial;font-size:85%;"><br /></span><span style="font-family:arial;font-size:85%;"><br /></span><span style="font-family:arial;font-size:85%;"><strong><span style="font-family:Trebuchet MS;">Budgetary Issues and Physical Plant</span><br /></strong></span><span style="font-family:arial;font-size:85%;"><br /></span><span style="font-family:Trebuchet MS;font-size:85%;">I need to build my Data Basement on a budget. After all, I still need to pay the mortgage, feed the family, buy Hoosiers for the racecar, and pay for all the electricity my spiffy new grid will need. I randomly chose a budget in the range of $3,000 - $3,500. The grid will live in the basement, and I want to save space there, so I will use a single standard rack that I picked up in the past. The rack will sit on a wooden platform I'll build from scrap wood (cheap protection in case of minor flooding). Since my basement has reasonably high humidity, I'll put a dehumidifier plumbed to a wasteline to keep humidity levels under control. The grid will pull between around 1.4Kw (about 13 amps @ 110VAC), so there will not be a need for any special electrical work. I want UPS support eventually on the grid, but initially I'll just use some spike filters on each AC line. I already have broadband via DSL with a wired/wireless router, but I'll need a gig switch with several ports so that the grid nodes can communicate with each other at speed.</span><br /><span style="font-family:Trebuchet MS;font-size:85%;"></span><br /><span style="font-family:Trebuchet MS;font-size:85%;">The next posting will discuss parts selection for the individual compute nodes.</span>ccghttp://www.blogger.com/profile/01309764084115413592noreply@blogger.com0tag:blogger.com,1999:blog-13596339.post-1165302165105103932006-12-05T02:01:00.000-05:002007-01-14T23:13:24.979-05:0065 nanometer chip from AMD: The war continues...<span style="font-family:trebuchet ms;font-size:85%;">If you're a fan of AMD, and it's a poorly kept secret that I am one, then you'll be pleased to read the latest blurb in CNET regarding new 65 nanometer chips from AMD:<br /><br /></span><a href="http://news.com.com/2100-1006_3-6140764.html?part=rss&tag=2547-1_3-0-5&subj=news"><span style="font-family:trebuchet ms;font-size:85%;">http://news.com.com/2100-1006_3-6140764.html?part=rss&tag=2547-1_3-0-5&subj=news</span></a><br /><span style="font-family:trebuchet ms;font-size:85%;"><br />This is good stuff, and helps AMD to catch up with Intel who rolled out 65nm process over a year ago.<br /><br />Netting out all the process changed, the new chips will draw about 30% less power and will be cheaper to produce. This means AMD can compete more heavily against Intel. All this, of course, is great news to chip purchasers.<br /><br />More performance, less power, less money...gotta love it!</span>ccghttp://www.blogger.com/profile/01309764084115413592noreply@blogger.com0tag:blogger.com,1999:blog-13596339.post-1160372451343768872006-10-09T01:39:00.000-04:002006-10-09T22:14:17.580-04:00More good reading<span style="font-family:arial;font-size:85%;">Interesting research from MIT about column store databases. Mike Stonebraker and several other database luminaries - and a bunch of smart students - wrote this paper for VLDB.</span><br /><br /><a href="http://db.csail.mit.edu/projects/cstore/vldb.pdf"><span style="font-family:arial;font-size:85%;">http://db.csail.mit.edu/projects/cstore/vldb.pdf</span></a><br /><br /><p><span style="font-family:arial;font-size:85%;">Mike is also the founding executive for both Streambase and Vertica...hmmm.....</span></p>ccghttp://www.blogger.com/profile/01309764084115413592noreply@blogger.com0tag:blogger.com,1999:blog-13596339.post-1120179239293156432005-06-30T20:36:00.000-04:002005-06-30T20:53:59.300-04:00Good reading...<span style="font-family:arial;font-size:85%;">My interests these days are around searching, indexing, and processing large data sets...datamining on steroids to some extent. Think terabytes not gigabytes, think hundreds of thousands of files instead of thousands, and that's getting to the scale that interests me. Parallel programming, grid processing, and all those new (old) technologies are a big part of making that sort of processing scalable and affordable, especially for small companies. Since I've been a research/learn/academic state of mind this summer, here's part of my current reading list:</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">Hariri, Parashar et. al., <em>"Tools and Environments for Parallel and Distributed Computing"</em> (Wiley, 2004)</span><br /><span style="font-family:Arial;font-size:85%;">Chakrabarti, Soumen, <em>"Mining the web: Discovering Knowledge from Hypertext Data"</em> (Morgan Kaufmann, 2003)</span><br /><span style="font-family:Arial;font-size:85%;">Morse, H. Stephen, <em>"Practical Parallel Computing"</em> (Academic Press, 1994)</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">The "Tool and Environments" book is a solid overview/review. The Chakrabarti book is fascinating...I can nit a bit about it being dated (alta vista was the BIG search engine when the book was written), but the presentation of algorithms, and the depth of detail is impressive...a real "page turner" in a geeky sort of way. I bought "Practical Parallel Computing" back in 1994 when I was prepping for an interview with Thinking Machines (remember them?). The material is a bit dated, but if you view "Tools and Environments" and "Practical" as a combined skim-through/review it's worth the time.</span><br /><span style="font-family:Arial;font-size:85%;"></span><br /><span style="font-family:Arial;font-size:85%;">I really have enjoyed reading Chakrabarti thus far...not yet sure what to do with all the ideas coming out of the reading, but I've got some interesting thoughts.</span>ccghttp://www.blogger.com/profile/01309764084115413592noreply@blogger.com0