#StackBounty: #docker #hadoop #yarn Unable to increase Max Application Master Resources

Bounty: 50

I am using uhopper/hadoop docker image to create yarn cluster. I have 3 nodes with 250GB RAM per node. I have added configuration

        - name: YARN_CONF_yarn_scheduler_minimum___allocation___mb
          value: "2048"
        - name: YARN_CONF_yarn_scheduler_maximum___allocation___mb
          value: "16384"
        - name:  MAPRED_CONF_mapreduce_framework_name
          value: "yarn" 
        - name: MAPRED_CONF_mapreduce_map_memory_mb
          value: "8192"
        - name: MAPRED_CONF_mapreduce_reduce_memory_mb
          value: "8192"
        - name: MAPRED_CONF_mapreduce_map_java_opts
          value: "-Xmx8192m"
        - name: MAPRED_CONF_mapreduce_reduce_java_opts
          value: "-Xmx8192m"

Max Application Master Resources is 10240 MB. I ran 5 spark jobs with each 3 GB driver memory, 2 jobs never came in RUNNING state due 10240MB. I am unable to fully utilize my hardware.
enter image description here
How I can increase the Max Application Master Resources memory ?


Get this bounty!!!

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.