Re: [Condor-users] hadoop with condor

["Ozcan ILIKHAN" <ilikhan@xxxxxxxxxxx>]

Hi Mag,

I am working on to make it more user friendly. Actually there are just a few 
parameters to set:
HDFS_NAMENODE, HDFS_NAMENODE_WEB
HDFS_BACKUPNODE (optional for data nodes), HDFS_BACKUPNODE_WEB (optional for 
data nodes),
HDFS_NAMENODE_ROLE (optional for data nodes)
HDFS_NODETYPE

Assume that we we want to configure  machines "myname" as (active) NameNode, 
"mybackup" as BackupNode and "mydata" as DataNode. Then configuration for 
each machine will be as follows (assuming they are under the domain 
"mydomain.com"):

########--- config for myname-----------
HDFS_NAMENODE = hdfs://myname.mydomain.com:9000
HDFS_NAMENODE_WEB = myname.mydomain.com:50070

HDFS_BACKUPNODE = hdfs://mybackup.mydomain.com:50100
HDFS_BACKUPNODE_WEB = mybackup.mydomain.com:50105

HDFS_NODETYPE = HDFS_NAMENODE

HDFS_NAMENODE_ROLE = ACTIVE

HDFS_NAMENODE_DIR = /tmp/hadoop_name
HDFS_DATANODE_DIR = /tmp/hadoop_data
#------ end of config ----------------------------------------

########---  config for mybackup -----------
# this will be same as configuration of "myname"
# just we need to chage HDFS_NAMENODE_ROLE

HDFS_NAMENODE = hdfs://myname.mydomain.com:9000
HDFS_NAMENODE_WEB = myname.mydomain.com:50070

HDFS_BACKUPNODE = hdfs://mybackup.mydomain.com:50100
HDFS_BACKUPNODE_WEB = mybackup.mydomain.com:50105

HDFS_NODETYPE = HDFS_NAMENODE

HDFS_NAMENODE_ROLE = BACKUP

HDFS_NAMENODE_DIR = /tmp/hadoop_name
HDFS_DATANODE_DIR = /tmp/hadoop_data
#------ end of config ----------------------------------------

########---  config for mydata -----------
# this will be same as configuration of "myname"
# just we need to change HDFS_NODETYPE
# we can simply  ignore HDFS_BACKUPNODE, HDFS_BACKUPNODE_WEB,
# and   HDFS_NAMENODE_ROLE

HDFS_NAMENODE = hdfs://myname.mydomain.com:9000
HDFS_NAMENODE_WEB = myname.mydomain.com:50070

HDFS_NODETYPE = HDFS_DATANODE

HDFS_NAMENODE_DIR = /tmp/hadoop_name
HDFS_DATANODE_DIR = /tmp/hadoop_data
#------ end of config ----------------------------------------


I hope this will help.
Sincerely,
---------------------------------
Ozcan ILIKHAN
PhD Student
Department of Computer Sciences
University of Wisconsin-Madison
http://pages.cs.wisc.edu/~ilikhan


--------------------------------------------------
From: "Mag Gam" <magawake@xxxxxxxxx>
Sent: Wednesday, January 05, 2011 5:57 AM
To: "Condor-Users Mail List" <condor-users@xxxxxxxxxxx>
Subject: [Condor-users] hadoop with condor

> Has anyone gotten hdfs to work with condor? I know the newer version
> of condor support it but the configuration is nightmarish
> _______________________________________________
> Condor-users mailing list
> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/condor-users/