Mailing List Archives
Public Access
|
|
|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Condor-users] hadoop with condor
- Date: Wed, 5 Jan 2011 08:02:20 -0600
- From: "Ozcan ILIKHAN" <ilikhan@xxxxxxxxxxx>
- Subject: Re: [Condor-users] hadoop with condor
Hi Mag,
I am working on to make it more user friendly. Actually there are just a few
parameters to set:
HDFS_NAMENODE, HDFS_NAMENODE_WEB
HDFS_BACKUPNODE (optional for data nodes), HDFS_BACKUPNODE_WEB (optional for
data nodes),
HDFS_NAMENODE_ROLE (optional for data nodes)
HDFS_NODETYPE
Assume that we we want to configure machines "myname" as (active) NameNode,
"mybackup" as BackupNode and "mydata" as DataNode. Then configuration for
each machine will be as follows (assuming they are under the domain
"mydomain.com"):
########--- config for myname-----------
HDFS_NAMENODE = hdfs://myname.mydomain.com:9000
HDFS_NAMENODE_WEB = myname.mydomain.com:50070
HDFS_BACKUPNODE = hdfs://mybackup.mydomain.com:50100
HDFS_BACKUPNODE_WEB = mybackup.mydomain.com:50105
HDFS_NODETYPE = HDFS_NAMENODE
HDFS_NAMENODE_ROLE = ACTIVE
HDFS_NAMENODE_DIR = /tmp/hadoop_name
HDFS_DATANODE_DIR = /tmp/hadoop_data
#------ end of config ----------------------------------------
########--- config for mybackup -----------
# this will be same as configuration of "myname"
# just we need to chage HDFS_NAMENODE_ROLE
HDFS_NAMENODE = hdfs://myname.mydomain.com:9000
HDFS_NAMENODE_WEB = myname.mydomain.com:50070
HDFS_BACKUPNODE = hdfs://mybackup.mydomain.com:50100
HDFS_BACKUPNODE_WEB = mybackup.mydomain.com:50105
HDFS_NODETYPE = HDFS_NAMENODE
HDFS_NAMENODE_ROLE = BACKUP
HDFS_NAMENODE_DIR = /tmp/hadoop_name
HDFS_DATANODE_DIR = /tmp/hadoop_data
#------ end of config ----------------------------------------
########--- config for mydata -----------
# this will be same as configuration of "myname"
# just we need to change HDFS_NODETYPE
# we can simply ignore HDFS_BACKUPNODE, HDFS_BACKUPNODE_WEB,
# and HDFS_NAMENODE_ROLE
HDFS_NAMENODE = hdfs://myname.mydomain.com:9000
HDFS_NAMENODE_WEB = myname.mydomain.com:50070
HDFS_NODETYPE = HDFS_DATANODE
HDFS_NAMENODE_DIR = /tmp/hadoop_name
HDFS_DATANODE_DIR = /tmp/hadoop_data
#------ end of config ----------------------------------------
I hope this will help.
Sincerely,
---------------------------------
Ozcan ILIKHAN
PhD Student
Department of Computer Sciences
University of Wisconsin-Madison
http://pages.cs.wisc.edu/~ilikhan
--------------------------------------------------
From: "Mag Gam" <magawake@xxxxxxxxx>
Sent: Wednesday, January 05, 2011 5:57 AM
To: "Condor-Users Mail List" <condor-users@xxxxxxxxxxx>
Subject: [Condor-users] hadoop with condor
Has anyone gotten hdfs to work with condor? I know the newer version
of condor support it but the configuration is nightmarish
_______________________________________________
Condor-users mailing list
To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/condor-users
The archives can be found at:
https://lists.cs.wisc.edu/archive/condor-users/