• Create a Seven-Node Hadoop Container on Docker
• Introduction
• Prerequisites
• Files Used
• Step 1: Prepare the Docker Environment
• Step 2: Build and Start the Cluster
• Step 3: Verify the Setup
• Conclusion
• Config reference
• Reference
• Create a Single-Node Hadoop Container on Docker
• Quick Steps for the Busy People
  • Steps to build the container using ready-to-use files
  • Check the setup
• Detailed steps - building the setup, file by file
  • Details of the files used
  • Building, Running, and Testing the Setup
  • How to test the setup?
  • Setup details
• Appendix
• Common Errors and Their Solutions

Create a Seven-Node Hadoop Container on Docker¶

Introduction¶

In this guide, I'll walk through the process of setting up an Apache Hadoop cluster using Docker containers. This setup is ideal for development and testing purposes on your local machine. I'll cover the configuration files, container setup, and how to verify that the cluster is functioning correctly.

For busy people:

• Download and unzip the file to a folder
• CD and run the following commands

      docker-compose build
      docker-compose up -d
  

• You will have a full-fledged Hadoop setup

Prerequisites¶

• Docker installed on your machine.
• Basic knowledge of Docker and Hadoop.

Files Used¶

1. Dockerfile: Defines the environment and how the Hadoop services will be run inside the Docker containers.
2. docker-compose.yml: Manages the multi-container application, ensuring all necessary Hadoop services are launched and networked correctly.
3. entrypoint.sh: A script to start the appropriate Hadoop service based on the container's role (e.g., NameNode, DataNode).

Important: If you create the entrypoint.sh file on Windows, you must convert it to Unix format using a tool like Toolslick DOS to Unix Converter before using it in your Docker environment.

Step 1: Prepare the Docker Environment¶

1. Dockerfile: The Dockerfile sets up the Java runtime and Hadoop environment. Here's the Dockerfile used:

# Use Java 8 runtime as base image
FROM openjdk:8-jdk

# Set environment variables
ENV HADOOP_VERSION=2.7.7
ENV HADOOP_HOME=/usr/local/hadoop
ENV HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
ENV PATH=$PATH:$HADOOP_HOME/bin

# Install Hadoop
RUN wget https://archive.apache.org/dist/hadoop/core/hadoop-$HADOOP_VERSION/hadoop-$HADOOP_VERSION.tar.gz && \
    tar -xzvf hadoop-$HADOOP_VERSION.tar.gz && \
    mv hadoop-$HADOOP_VERSION $HADOOP_HOME && \
    rm hadoop-$HADOOP_VERSION.tar.gz

# Copy and set entrypoint script
# Note: If entrypoint.sh is created on Windows, convert it to Unix format using dos2unix website.
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh

# Expose Hadoop ports
EXPOSE 50070 8088 9000 9864 9870 9866 9867

# Set entrypoint
ENTRYPOINT ["/entrypoint.sh"]

1. docker-compose.yml: The docker-compose.yml file orchestrates the Hadoop cluster by defining services like NameNode, DataNode, and ResourceManager.

version: '3.8'

services:
  namenode:
    build: .
    container_name: namenode
    hostname: namenode
    environment:
      - CLUSTER_NAME=my-hadoop-cluster
    volumes:
      - namenode_data:/hadoop/dfs/name
    ports:
      - "29070:50070"  # HDFS NameNode Web UI on port 29070
      - "29870:9870"   # NameNode Web UI port on port 29870
      - "29000:9000"   # HDFS port on port 29000
    command: namenode
    networks:
      - dasnet

  secondarynamenode:
    build: .
    container_name: secondarynamenode
    hostname: secondarynamenode
    volumes:
      - secondarynamenode_data:/hadoop/dfs/secondary
    command: secondarynamenode
    networks:
      - dasnet

  datanode1:
    build: .
    container_name: datanode1
    hostname: datanode1
    volumes:
      - datanode1_data:/hadoop/dfs/data
    command: datanode
    networks:
      - dasnet

  datanode2:
    build: .
    container_name: datanode2
    hostname: datanode2
    volumes:
      - datanode2_data:/hadoop/dfs/data
    command: datanode
    networks:
      - dasnet

  resourcemanager:
    build: .
    container_name: resourcemanager
    hostname: resourcemanager
    ports:
      - "28088:8088"  # ResourceManager Web UI on port 28088
    command: resourcemanager
    networks:
      - dasnet

  nodemanager1:
    build: .
    container_name: nodemanager1
    hostname: nodemanager1
    ports:
      - "29864:9864"  # NodeManager Web UI on port 29864
    command: nodemanager
    networks:
      - dasnet

  historyserver:
    build: .
    container_name: historyserver
    hostname: historyserver
    ports:
      - "29866:9866"  # HistoryServer Web UI on port 29866
      - "29867:9867"  # Additional service on port 29867
    command: historyserver
    networks:
      - dasnet

volumes:
  namenode_data:
  secondarynamenode_data:
  datanode1_data:
  datanode2_data:

networks:
  dasnet:
    external: true

1. entrypoint.sh: This script starts the appropriate Hadoop service based on the container’s role. Below is the script:

#!/bin/bash
# Format namenode if necessary
if [ "$1" == "namenode" ]; then
  $HADOOP_HOME/bin/hdfs namenode -format -force -nonInteractive
fi
# Start SSH service
service ssh start
# Start Hadoop service based on the role
if [ "$1" == "namenode" ]; then
  $HADOOP_HOME/sbin/hadoop-daemon.sh start namenode
elif [ "$1" == "datanode" ]; then
  $HADOOP_HOME/sbin/hadoop-daemon.sh start datanode
elif [ "$1" == "secondarynamenode" ]; then
  $HADOOP_HOME/sbin/hadoop-daemon.sh start secondarynamenode
elif [ "$1" == "resourcemanager" ]; then
  $HADOOP_HOME/sbin/yarn-daemon.sh start resourcemanager
elif [ "$1" == "nodemanager" ]; then
  $HADOOP_HOME/sbin/yarn-daemon.sh start nodemanager
elif [ "$1" == "historyserver" ]; then
  $HADOOP_HOME/sbin/mr-jobhistory-daemon.sh start historyserver
fi
# Keep the container running
tail -f /dev/null

Note: Convert entrypoint.sh to Unix format if it's created on Windows using Toolslick DOS to Unix Converter.

Step 2: Build and Start the Cluster¶

1. Build the Docker Images:
2. Navigate to the directory containing the Dockerfile, docker-compose.yml, and entrypoint.sh files.

3. Run the following command to build the Docker images:

   docker-compose build
   

4. Start the Cluster:

5. Start the cluster using the following command:

   docker-compose up -d
   

Step 3: Verify the Setup¶

1. Access Hadoop Web UIs:
2. NameNode Web UI: http://localhost:29870
3. HDFS NameNode Web UI: http://localhost:29070
4. ResourceManager Web UI: http://localhost:28088
5. NodeManager Web UI: http://localhost:29864
6. HistoryServer Web UI: http://localhost:29866

These interfaces will allow you to monitor the status of your Hadoop cluster and the jobs running on it.

1. Run a Test Job:

2. Create Input Directory in HDFS:

   docker exec -it namenode /bin/bash
   hdfs dfs -mkdir -p /input
   echo "Hello Hadoop" > /tmp/sample.txt
   hdfs dfs -put /tmp/sample.txt /input/
   

3. Run the WordCount Job:

   hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar wordcount /input /output
   

4. Check the Output:

   hdfs dfs -cat /output/part-r-00000
   

   Expected output:

   Hadoop  1
   Hello   1
   

Conclusion¶

Remember to convert any scripts created on Windows to Unix format before using them in your Docker containers to avoid potential issues. Happy coding!

Config reference¶

Reference¶

Official Hadoop Link

Create a Single-Node Hadoop Container on Docker¶

Setting up Hadoop can be quite confusing, especially if you're new to it. The best ready-to-use setups used to be provided by Cloudera, their current docker image(as on Aug 2024) has many issues. In this guide, I’ll help you create a pure Hadoop container with just one node. This guide is divided into two sections: one for quick steps and another for a detailed setup. The container setup has been thoroughly tested and can be created easily.

Quick Steps for the Busy People¶

If you’re short on time and don’t want to go through the entire process of setting everything up manually, this section is for you. Just follow the instructions here, use the ready-to-use files, and your container will be up and running in a few minutes!

Steps to build the container using ready-to-use files¶

Note: The `hadoop-3.4.0.tar.gz` file is not included in the zip because of its large size. I’ve set up the Dockerfile to download this file automatically from the Apache website if it’s not found in the folder. However, sometimes these download links change. If you encounter any issues with the automatic download, you can manually download the `hadoop-3.4.0.tar.gz` file from this link. If this link doesn’t work, simply search online for the file, download it, and place it in the folder. You don’t need to change anything in the Dockerfile; it will work as long as the file is named `hadoop-3.4.0.tar.gz`.

If you create a .sh file or any file on Windows and plan to use it in Linux, make sure you convert it to a Linux-friendly format using this site. Other methods may not work as well. It's important to convert the file, or else it might cause many unexpected errors.

• Download the hadoop-singlenode.zip file.
• Unzip it to a folder on your laptop.

• Open command prompt/terminal and cd to the folder where you unziped the files

  cd path_to_unzipped_folder
  

  - Build the doccker image from the Dockerfile. There is a dot

  docker build -t hadoop-singlenode .
  

  - Run the container from the built image

  docker run -it --name hadoop-singlenode --network dasnet -p 9870:9870 -p 8088:8088 -v namenode-data:/hadoop/dfs/namenode  -v datanode-data:/hadoop/dfs/datanode -v secondarynamenode-data:/hadoop/dfs/secondarynamenode  hadoop-singlenode
  

  - From inside the running container, start the Hadoop services:

  sudo service ssh start
  $HADOOP_HOME/sbin/start-dfs.sh
  $HADOOP_HOME/sbin/start-yarn.sh
  

  Check the setup¶

  • Once the services are up you can access the Hadoop links:

  • HDFS NameNode Web UI: http://localhost:9870

  • YARN ResourceManager Web UI: http://localhost:8088

Detailed steps - building the setup, file by file¶

If you want to understand how to build the Docker container from scratch or make custom modifications, follow these steps.

Details of the files used¶

The setup uses 6 files, Dockerfile, core-site.xml, hdfs-site.xml, mapred-site.xml, yarn-site.xml and hadoop-env.sh

• Dockerfile: This is the main file. We only use Dockerfile for our setup and no docker-compose.yml. But, you can add a docker-compose if you need. The filename is Dockerfile(no extension) and has the following content:

  # Use a base image with Java 8
  FROM openjdk:8-jdk
  
  # Set environment variables for Hadoop
  ENV HADOOP_VERSION=3.4.0
  ENV HADOOP_HOME=/usr/local/hadoop
  ENV JAVA_HOME=/usr/local/openjdk-8
  ENV PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
  ENV HDFS_NAMENODE_USER=dwdas
  ENV HDFS_DATANODE_USER=dwdas
  ENV HDFS_SECONDARYNAMENODE_USER=dwdas
  ENV YARN_RESOURCEMANAGER_USER=dwdas
  ENV YARN_NODEMANAGER_USER=dwdas
  
  # Install necessary packages
  RUN apt-get update && \
      apt-get install -y ssh rsync sudo wget && \
      apt-get clean
  
  # Set root password
  RUN echo "root:Passw0rd" | chpasswd
  
  # Create a user 'dwdas' with sudo privileges
  RUN useradd -m -s /bin/bash dwdas && \
      echo "dwdas:Passw0rd" | chpasswd && \
      usermod -aG sudo dwdas && \
      echo "dwdas ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers
  
  # Optional: Download and extract Hadoop tarball if not already provided
  COPY hadoop-${HADOOP_VERSION}.tar.gz /tmp/
  RUN if [ ! -f /tmp/hadoop-${HADOOP_VERSION}.tar.gz ]; then \
          wget https://archive.apache.org/dist/hadoop/common/hadoop-${HADOOP_VERSION}/hadoop-${HADOOP_VERSION}.tar.gz -O /tmp/hadoop-${HADOOP_VERSION}.tar.gz; \
      fi && \
      tar -xzf /tmp/hadoop-${HADOOP_VERSION}.tar.gz -C /usr/local && \
      mv /usr/local/hadoop-${HADOOP_VERSION} $HADOOP_HOME && \
      rm /tmp/hadoop-${HADOOP_VERSION}.tar.gz
  
  # Configure Hadoop - Create necessary directories on Docker volumes
  RUN mkdir -p /hadoop/dfs/namenode && \
      mkdir -p /hadoop/dfs/datanode && \
      mkdir -p /hadoop/dfs/secondarynamenode
  
  # Copy configuration files
  COPY core-site.xml $HADOOP_HOME/etc/hadoop/
  COPY hdfs-site.xml $HADOOP_HOME/etc/hadoop/
  COPY mapred-site.xml $HADOOP_HOME/etc/hadoop/
  COPY yarn-site.xml $HADOOP_HOME/etc/hadoop/
  # Copy and configure hadoop-env.sh. JAVA_HOME MUST be set here also.
  COPY hadoop-env.sh $HADOOP_HOME/etc/hadoop/hadoop-env.sh
  
  # Set ownership for Hadoop directories and volumes to 'dwdas'
  RUN chown -R dwdas:dwdas $HADOOP_HOME /hadoop/dfs/namenode /hadoop/dfs/datanode /hadoop/dfs/secondarynamenode
  
  # Switch to the dwdas user for all subsequent operations
  USER dwdas
  
  # Create the .ssh directory and set permissions
  RUN mkdir -p /home/dwdas/.ssh && \
      chmod 700 /home/dwdas/.ssh
  
  # Generate SSH keys for passwordless SSH login and configure SSH
  RUN ssh-keygen -t rsa -P '' -f /home/dwdas/.ssh/id_rsa && \
      cat /home/dwdas/.ssh/id_rsa.pub >> /home/dwdas/.ssh/authorized_keys && \
      chmod 600 /home/dwdas/.ssh/authorized_keys && \
      echo "Host localhost" >> /home/dwdas/.ssh/config && \
      echo "   StrictHostKeyChecking no" >> /home/dwdas/.ssh/config && \
      chmod 600 /home/dwdas/.ssh/config
  
  # Format HDFS as 'dwdas' user
  RUN $HADOOP_HOME/bin/hdfs namenode -format
  
  # Expose the necessary ports for Hadoop services
  EXPOSE 9870 8088 19888
  
  # Set the container to start in the dwdas user's home directory
  WORKDIR /home/dwdas
  
  # Set the container to start with a bash shell
  CMD ["bash"]
  

  - core-site.xml: Configures the default filesystem and permissions.

  <configuration>
      <property>
          <name>fs.defaultFS</name>
          <value>hdfs://localhost:9000</value>
      </property>
      <property>
          <name>dfs.permissions</name>
          <value>false</value>
      </property>
  </configuration>
  

• hdfs-site.xml: Configures the NameNode and DataNode directories.

  <configuration>
      <property>
          <name>dfs.namenode.name.dir</name>
          <value>file:///usr/local/hadoop/tmp/hdfs/namenode</value>
      </property>
      <property>
          <name>dfs.datanode.data.dir</name>
          <value>file:///usr/local/hadoop/tmp/hdfs/datanode</value>
      </property>
      <property>
          <name>dfs.replication</name>
          <value>1</value>
      </property>
      <property>
          <name>dfs.permissions.superusergroup</name>
          <value>dwdas</value>
      </property>
      <property>
          <name>dfs.cluster.administrators</name>
          <value>dwdas</value>
      </property>
  </configuration>
  

• mapred-site.xml: Configures the MapReduce framework to use YARN.

  <configuration>
      <property>
          <name>mapreduce.framework.name</name>
          <value>yarn</value>
      </property>
      <property>
          <name>mapred.job.tracker</name>
          <value>hadoop-master:9001</value>
      </property>
  </configuration>
  

• yarn-site.xml: Configures YARN services.

  <configuration>
      <property>
          <name>yarn.nodemanager.aux-services</name>
          <value>mapreduce_shuffle</value>
      </property>
      <property>
          <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
          <value>org.apache.hadoop.mapred.ShuffleHandler</value>
      </property>
  </configuration>
  

• hadoop-env.sh: I haven’t included this entire file because it’s large, and there’s only one change you need to make. Refer to the hadoop-env.sh file in the zip folder. The only modification required is shown below. Note: This step is crucial. Without this, Hadoop will give an error saying it can’t find JAVA_HOME, even if you’ve already set it as an environment variable. This change follows Apache's instructions.

  export JAVA_HOME=/usr/local/openjdk-8
  

Building, Running, and Testing the Setup¶

The process for building, running, and testing the setup is the same as described in the Quick Steps section. Simply navigate to the folder, build the container, and then run it as before.

How to test the setup?¶

The table below shows how various components and functionalities could be tested:

Setup details¶

Appendix¶

• Alternate command - Start the Namenode

  $HADOOP_HOME/sbin/hadoop-daemon.sh start namenode
  

• Alternate command - Start the Datanode

  $HADOOP_HOME/sbin/hadoop-daemon.sh start datanode
  

• Enter jps to see all the hadoop services

  

• Command to get a report on the Hadoop setup

  hdfs dfsadmin -report
  

  

Common Errors and Their Solutions¶

• Port Binding Error:
• Error: Ports are not available: exposing port TCP 0.0.0.0:50070 -> 0.0.0.0:0: listen tcp 0.0.0.0:50070: bind: An attempt was made to access a socket in a way forbidden by its access permissions.

• Solution: Make sure that the ports you’re using aren’t already in use by other services. You can check and stop any process using these ports with these commands:

  netstat -aon | findstr :50070
  taskkill /PID <PID> /F
  

  Or you can change the port numbers in the Dockerfile and docker run command.

• Permission Denied Error:

• Error: ERROR: Attempting to operate on hdfs namenode as root but there is no HDFS_NAMENODE_USER defined.

• Solution: Hadoop services should not run as the root user. The Dockerfile sets up a non-root user (dwdas) to run Hadoop and Hive services:

  ENV HDFS_NAMENODE_USER=dwdas
  ENV HDFS_DATANODE_USER=dwdas
  ENV HDFS_SECONDARYNAMENODE_USER=dwdas
  ENV YARN_RESOURCEMANAGER_USER=dwdas
  ENV YARN_NODEMANAGER_USER=dwdas
  

• Multiple SLF4J Bindings Error:

• Error: SLF4J: Class path contains multiple SLF4J bindings.
• Solution: This is usually just a warning, not an error. It means there are multiple SLF4J bindings in the classpath. It can generally be ignored, but if it causes issues, you might need to clean up the classpath by removing conflicting SLF4J jars.