Skip to main content

Apache Hadoop

  • What it is
    • The original open-source framework for distributed storage and batch processing of very large datasets across commodity machines.
  • The three pieces
    • HDFS — distributed file system that splits and replicates blocks across nodes.
    • YARN — cluster resource manager and job scheduler.
    • MapReduce — the batch programming model: map to transform, reduce to aggregate.
  • On AWS
    • Amazon EMR is managed Hadoop — AWS provisions and operates the cluster; EMR clusters usually read from Amazon S3 (via EMRFS) rather than HDFS, which decouples storage from compute.
  • Its place today
  • Exam signal
    • The word “Hadoop” in a question is nearly always pointing at EMR.

Linked from