Re: [HTCondor-users] Transfer output files also on Job removal?

[Michael Pelletier <michael.v.pelletier@xxxxxxxxxxxx>]

This came up in the 2019 HTCondor Week, and I discussed it a bit with the team, but I don't recall the specifics. Maybe one of them will chime in.

Michael V Pelletier
Principal Engineer

Raytheon Technologies
Information Technology
Digital Transormation & Innovation
 


-----Original Message-----
From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Oliver Freyermuth
Sent: Tuesday, October 20, 2020 7:27 AM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Cc: Peter Wienemann <peter.wienemann@xxxxxxxxxxx>
Subject: [External] [HTCondor-users] Transfer output files also on Job removal?

Dear HTCondor experts,

I've been looking for a knob, but could not find one for this use case:
We'd love to transfer the output files of some jobs even in case of removal, i.e. condor_rm, or periodic_remove e.g. when a time limit is hit.

The example use case here are interactive Jupyter notebooks: We don't want to share a file system with all the resources the notebooks may run on, and would love to use HTCondor file transfer to get things to and fro.
However, interactive jobs are usually either cancelled by a time limit (i.e. periodic_remove), or (in case of JupyterHub's batchspawner) by condor_rm, so file transfer would be especially helpful for interactive use cases without shared file system.

So it burns down to this:
  Is there a way to also trigger file transfer on job removal (I have control over both schedd and submit file)?


I have for now tried to set:
  when_to_transfer_output = ON_EXIT_OR_EVICT
  TransferOutputFiles = jupyter
  ShouldTransferFiles = True
and if I parse the logs correctly, indeed this triggers a file transfer to SPOOL on the schedd node, but then the files are purged afterwards, since the job was removed. Having them transferred back to the original place (or at least accessible) would be ideal in our case.

Cheers and thanks in advance,
	Oliver