Skip to main content

Apache Hadoop

  • Apache Hadoop is an open-source software framework written in Java for distributed storage and distributed processing of very large data sets on computer clusters built from commodity hardware.
  •  All the modules in Hadoop are designed with a fundamental assumption that hardware failures (of individual machines, or racks of machines) are commonplace and thus should be automatically handled in software by the framework. 
  • The core of Apache Hadoop consists of a storage part (Hadoop Distributed File System (HDFS)) and a processing part (MapReduce). 
  •   To process the data, Hadoop MapReduce transfers packaged code for nodes to process in parallel, based on the data each node needs to process.  

The base Apache Hadoop framework is composed of the following modules:
    • Hadoop Common – contains libraries and utilities needed by other Hadoop modules;
    • Hadoop Distributed File System (HDFS) – a distributed file-system that stores data on commodity machines, providing very high aggregate bandwidth across the cluster;
    • Hadoop YARN – a resource-management platform responsible for managing computing resources in clusters and using them for scheduling of users' applications
    • Hadoop MapReduce – a programming model for large scale data processing.

    The Hadoop framework itself is mostly written in the Java programming language, with some native code in C and command line utilities written as Shell script. For end-users, though MapReduce Java code is common, any programming language can be used with "Hadoop Streaming" to implement the "map" and "reduce" parts of the user's program. Other related projects expose other higher-level user interfaces.

    Labels

    Labels:

    Comments

    Post a Comment

    Popular posts from this blog

    Shortcut key to align code in eclipse

    While learning java or working with java, we may need to copy a code from other source like internet or other files. When we do that the code may looks messy like before. which is not in standard and difficult to work on complex logics. Below key is useful to align the messy code:       Ctrl + Shift + F:   Formats a selected block of code or a whole source file.  Format messy code to Java-standard code.  If a code block is not selected, Eclipse applies formatting for the whole file. #shortcut key to align code in eclipse #shortcut_key_to_align_code_in_eclipse #shortcut_key_to_format_code_in_eclipse #shortCut key to format code in eclipse #shortcut_key_to_code_ alignment_in_eclipse #Auto- Alignment Shortcut Key  in  Eclips e
    Post a Comment
    Read more

    10 WhatsApp Tips and Tricks Everyone Should Know

    For most of us, WhatsApp has become the default go-to messenger, because everyone's on it, and it's on nearly every platform. WhatsApp has over a billion downloads on Google Play alone, the only other messenger to boast that is Facebook Messenger. The app has made itself indispensable by adding a browser version and voice calling to its platform. For those who have to deal with the app on a daily basis, we've lined up some interesting tricks and tweaks that will improve your user-experience. 1. Mute group chats and notifications Group chat notifications can get noisy, they're best turned off when you have to focus on a task at hand. On Android On a group chat, tap the Menu button, and hit Mute. Then, choose the time-frame for which you would like the group to be muted. You can also disable notifications for the specified period. This feature is now available on the web browser too. On iPhone Open Group Chat, tap the group subject to reveal the Gr...
    1 comment
    Read more

    The First 10 People Who Sign up On Facebook

    The First 10 People Who Sign up On Facebook 10. Zach Bercu sphotos-b.xx.fbcdn.net “The past eight years have been extraordinary,” Bercu said. A graduate of Emory’s medical school, Bercu spent a year in Israel, where he became fluent in Hebrew. He completed his residency in New York, part of the last intern class at St. Vincent’s, whose “hospital infrastructure crumbled around me,” he remembered of the facility, which closed in 2010. Now a resident at Mount Sinai in radiology, Bercu plans to complete a fellowship in interventional radiology, a form of “micro-surgery.” From his undergraduate years, “whether through Facebook or in person,” Bercu says he “took with me some of the greatest friendships one could have.” 9. Manuel Antonio Aguilar publicogt.com Aguilar calls himself a social entrepreneur “focused on the base o...
    Post a Comment
    Read more