[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Condor-users] Managing millions of log files from large DAGs

Stuart Anderson wrote:

How are people managing the O(1M) files generated by large DAGs (all the .out and .err files) that show up in the same directory. There are a few filesystems that scale well to having that many files in a single directory, but not many.

I have an application which generates millions of log files as well. I thought about this problem early on - to cope, I split up the single DAG into several levels of nested DAGs, and wrote a program to organize these DAGs into corresponding levels of nested directories. This strategy has worked quite well for me so far.


Armen Babikyan
MIT Lincoln Laboratory
armenb@xxxxxxxxxx . 781-981-1796