Re: [HTCondor-users] submitting issues when having huge transfer_input_files lists

[Henning Fehrmann <henning.fehrmann@xxxxxxxxxx>]

Hi Greg,

> Hi Henning:
> 
> A 8 Megabyte list of transfer file is ... rather large. Is there anyway the
> list can be shortened? HTCondor can be instructed to transfer a whole
> directory, recursively, which would help a lot. Note that the problem in
> this case isn't the size of the files to be transferred, but their
> description.

The submitfile is generated by Pegasus. We asked the Pegasus people,
whether directories can be used instead of a file list which is not
trivial. Also changing the pipeline to reduce the list length is
difficult. Would there be a way to make the submit more resilient to
larger arguments?

Cheers,
Henning