Re: [HTCondor-users] best practices for data untar/unzip

[Thomas Hartmann <thomas.hartmann@xxxxxxx>]

Hi Dimitri,

just a shoot in the dark but maybe ionicing the tars could improve the
situation?
Something like the best effort scheduler with a random priority so to
avoid choking on all tars with the same low priority or maybe going for
the idle scheduler (but no idea, how the grace period block would effect
the overall performance)?

Cheers,
  Thomas

On 2016-10-31 19:07, Dimitri Maziuk wrote:
> Hi everyone,
> 
> we have jobs that transfer a gigabyte zipped tarball of data, unzip and
> start crunching. The problem we're running into is when, say, 16 of them
> land on a 16-core worker node at once, the untar'ing completely chokes
> the drive. To the point where condor daemons wait too long trying to
> write their logs and die with status 44.
> 
> I expect that's a fairly common job pattern and I'm wondering what would
> be the best way to deal with it (FVVO "best"). Any suggestions?
> 
> TIA
> 
> 
> 
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/
>

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature