[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Condor-users] Condor Daemons Fail to run on node

I have seen the OS remount filesystems read-only when there is corruption. If you can, umount the affected file system and fsck it to be certain that it is in a good state. Then do a normal mount command and see which mode it gets remounted in .


James Burnash, Unix Engineering

T. 201-239-2248

jburnash@xxxxxxxxxx | www.knight.com

545 Washington Ave. | Jersey City, NJ


From: condor-users-bounces@xxxxxxxxxxx [mailto:condor-users-bounces@xxxxxxxxxxx] On Behalf Of Xenia Fave
Sent: Thursday, December 02, 2010 1:08 PM
To: Condor-Users Mail List
Subject: Re: [Condor-users] Condor Daemons Fail to run on node


Hi Ian,


I am almost positive quotas are not in use, however is there a command to check this for sure?


I was not able to touch the file and thus of course could not write in it. 

[xfave@compute-1-0 ~]$ sudo -u condor touch /scratch/condor/log/test.txt

touch: cannot touch `/scratch/condor/log/test.txt': Read-only file system


[xfave@compute-1-1 ~]$ sudo -u condor echo "hello world" >>/scratch/condor/log/test.txt

-bash: /scratch/condor/log/test.txt: Read-only file system


I tried to remount the file system and received the following. I have never used the command before so perhaps I'm doing something wrong?


[xfave@compute-1-0 condor]$ sudo umount /scratch /dev/sda3


umount: /scratch: device is busy

umount: /scratch: device is busy

umount: /scratch: device is busy

umount: /scratch: device is busy


Thank you


On Thu, Dec 2, 2010 at 12:38 PM, Ian Chesal <ichesal@xxxxxxxxxxxxxxxxxx> wrote:

On Thu, Dec 2, 2010 at 12:30 PM, Xenia Fave <xfave2008@xxxxxxxxxx> wrote:

The file system is not full nor is the file system mounted in read-only


[xfave@compute-1-0 condor]$ df -h

Filesystem            Size  Used Avail Use% Mounted on

/dev/sda1              19G  5.5G   13G  31% /

/dev/sda3             193G   13G  171G   7% /scratch

tmpfs                 7.9G     0  7.9G   0% /dev/shm

nas-0-0:/nas0         9.6T  6.8T  2.8T  72% /mnt/nas0


Are quotas in use on the filesystem (note: it doesn't look like it based on you fstab output, but check just in case)? Maybe the user hit it's quota?


Can you touch a new file, as the user Condor writes its log files, as on that file system?


sudo -u <user> touch /scratch/condor/log/test.txt


Can you put data in the file?


sudo -u <user> echo "hello world" >> /scratch/condor/log/test.txt


The relevant mount output looks the same for both machines, so the filesystem is mounted the same on working and non-working nodes (I trimmed the output so it's easier to see):


[xfave@compute-1-0 condor]$ mount

/dev/sda3 on /scratch type ext3 (rw)


However, the results of the mount command on one of the working nodes is slightly different, I'm not sure if that's relevant.


[xfave@compute-1-1 ~]$ mount

/dev/sda3 on /scratch type ext3 (rw)


Have you tried unmounting, remounting the /scratch share on a bad machine?


- Ian

Condor-users mailing list
To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting

The archives can be found at:


This e-mail, and any attachments thereto, is intended only for use by the addressee(s)named herein and
may contain legally privileged and/or confidential information. If you are not the intended recipient of this
e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail and any attachments

thereto, is strictly prohibited. If you have received this in error, please immediately notify me and permanently
delete the original and any printout thereof.
E-mail transmission cannot be guaranteed to be secure or error-free.
The sender therefore does not accept liability for any errors or omissions in the contents of this message which
arise as a result of e-mail transmission.
Knight Capital Group may, at its discretion, monitor and review the content of all e-mail communications.