Hadoop操作手册

preface1．introduction2．hdfs goals and motivation design daemons reading and writing data the read path the write path managing filesystem metadata namenode high availability namenode federation access and integration command—line tools fuse rest support3．mapreduce the stages of mapreduce introducing hadoop mapreduce daemons when it all goes wrong yarn4．planning a hadoop cluster picking a distribution and version of hadoop apache hadoop cloudera’s distribution including apache hadoop what should i use? hardware selection master hardware selection worker hardware selection cluster sizing blades，sans，and virtualization operating system selection and preparation deployment layout software hostnames．dns．and identmcation users，groups，and privileges kernel tuning vm．swappiness vm．overcommit_memory disk configuration choosing a filesystem mount options network design network usage in hadoop：a review 1 gb versus 10 gb networks typical network topologies 5．installation andconfiguration installing hadoop apache hadoop cdh configuration：an 0verview the hadoop xml configuration files environment variables and shell scripts logging configuration hdfs identification and location optimization and tuning formatting the namenode creating a／tmp directory namenode high availability fencing options basic configuration automatic failover configuration format and bootstrap the namenodes namenode federation mapreduce identification and location optimization and tuning rack topology security6．identity，authentication，and authorization identity kerberos and hadoop kerberos：a refresher kerberos support in hadoop authorization hdfs mapreduce other tools and systems tying it together7．resojjrcemanagement what is resource management? hdfs quotas mapreduce schedulers the fifo scheduler the fair scheduler the capacity scheduler the future8．clustermaintenance managing hadoop processes starting and stopping processes with into scripts starting and stopping processes manually hdfs maintenance tasks adding a datanode decommissioning a datanode checking filesystem integrity with fsck balancing hdfs block data dealing with a failed disk mapreduce maintenance tasks adding a tasktracker decommissioning a tasktracker killing a mapreduce job killing a mapreduce task dealing with a blacklisted tasktracker9．troubleshooting differential diagnosis applied to systems