[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] Transfer out files question



TryÂ

When_to_transfer_output = ON_EXIT

If that doesn't work, try both "When_to_transfer_output" and "SCRIPT POST" to run a script to rename the files on the compute node, and updateÂTRANSFER_OUTPUT_FILES Âto reflect the new filenames.

--Mick

On Wed, Aug 30, 2017 at 1:10 PM, Dan, Bowen <bowen.dan@xxxxxxxx> wrote:

Hi Mick,

Â

Thanks for your quick response. I have tried the following:

Â

TRANSFER_OUTPUT_FILES = A/myfile.txt, B/myfile.txt

TRANSFER_OUTPUT_REMAPS = âA/myfile.txt=a_file.txt;B/myfile.txt=b_file.txtâ

Â

However, it looks to me that the myfile.txt was transferred back to the submit machine first, then the remap logic applied to it. Because of the file name collision, condor only transferred back one myfile.txt, so TRANSFER_OUTPUT_REMAPS = âA/myfile.txt=a_file.txt;B/myfile.txt=b_file.txtâ did not work.

Â

Due to some reasons, I do not have control over how files are created or named on the execute machine. Can condor rename the file before transferring it back? Or can condor only transfer the directory with just selective files under it?

Â

Thanks a lot!

Â

Best,

Bowen

Â

Â

From: HTCondor-users <htcondor-users-bounces@cs.wisc.edu> on behalf of "Mick T." <timony@xxxxxxxxxxxxxxxxxxx>
Reply-To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Date: Wednesday, August 30, 2017 at 11:54 AM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: Re: [HTCondor-users] Transfer out files question

Â

I think you need to remove the backslashes, and replace

Â

TRANSFER_OUTPUT_FILES = A/myfile.txt/, B/myfile.txt/Â

Â

withÂ

Â

TRANSFER_OUTPUT_FILES = A/myfile.txt, B/myfile.txt

Â

Tho' that may put the files in the current directory and overwrite one of the files with the other. So you may want to use theÂtransfer_output_remaps option, or easier still rename both files so there's name collision.

Â

Â

"When a path to an output file or directory is specified, it specifies the path to the file on the execute side. As a destination on the submit side, the file is placed in the job's initial working directory, and it is named using the base name of the original path. For example,Âpath/to/output_fileÂbecomesÂoutput_fileÂin the job's initial working directory. The name and path of the file that is written on the submit side may be modified by usingÂtransfer_output_remaps. Note that this remap function only works with files but not with directories."

Â

Cheers!

--Â

Mick Timony
Team Lead &ÂScientific Web Services Developer
SBGridÂCoreÂ- SBGrid.org
Harvard Medical School
--

Â

On Wed, Aug 30, 2017 at 12:20 PM, Dan, Bowen <bowen.dan@xxxxxxxx> wrote:

Hi,

Â

I have a question regards to how the transfer out files work. Currently on the execute machine, I have two folders A and B, both contains a considerable amount of data, the two directories both have a file named âmyfile.txtâ. It would be like this:

A/myfile.txt and B/myfile.txt. Instead of transferring whole directories (A and B) back to the submit machine, I only want to transfer A/myfile.txt, B/myfile.txt. So far, I have tried the following:

Â

TRANSFER_OUTPUT_FILES = A/myfile.txt/, B/myfile.txt/

Â

Unfortunately, it did not work. I was wondering if there is a way for me to only transfer the desired files back?

Â

Appreciate your help!

Â

Best,

Bowen

Â


_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@cs.wisc.edu with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/



Â

Â


_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@cs.wisc.edu with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/