world’s largest Hadoop application producing data used by the Yahoo! Search Webmap

Yahoo announced it is running the world’s largest Hadoop application, a 10,000 core Linux cluster producing data used by the Yahoo! Search Webmap.”. the numbers are really impressive:

• number of links between pages in the index: roughly 1 trillion links
• size of output: over 300 TB, compressed!
• number of cores used to run a single map-reduce job: over 10,000
• raw disk used in the production cluster: over 5 petabytes

No Comments so far
Leave a comment



Leave a comment
Line and paragraph breaks automatic, e-mail address never displayed, HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Name (required)

E-mail (required)

URI

Your Comment