Mailing List Archives

Public Access

 		[Computer Systems Lab]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] Conda env with workstation pool

Unfortunately, I don't think there's a consistent way to activate an environment post-facto. The problem is that conda environments bring their own Python, and they may include packages built against that version of Python. You could try adding the lib/pythonX.Y/site-packages directory inside the conda environment to sys.path after Python has started, but I suspect that for many packages that aren't pure-Python (i.e., important ones like numpy) you won't be able to import them correctly if you don't have the same Python version for the system (which is presumably what your script is running against) and whatever the packed environment has. Of course, if you can guarantee that everyone uses the same version of Python as is available on the system, then it might work!

In my experience, the easiest way to avoid using a wrapper script with Python programs that need custom environments is, unfortunately, to use a container... which is really just a giant implicit wrapper script :)

Josh Karpel


On Fri, Jul 24, 2020 at 6:50 AM West Matthew <matthew.west@xxxxxxxx> wrote:

Hi Josh,

I am working from the CHTC recommendation using conda pack. I am curious if you know a way to activate the conda environment from within a python script? Having to wrap my python executable with a bash script is rather frustrating when I am trying to get away from multi-language situation.

I can create the directory and extract the contents of the tarball. I just need to activate the env.

Cheers,

Matt

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Josh Karpel via HTCondor-users <htcondor-users@xxxxxxxxxxx>
Sent: Thursday, July 23, 2020 4:41:59 PM
To: HTCondor-Users Mail List
Cc: Josh Karpel
Subject: Re: [HTCondor-users] Conda env with workstation pool
Â
That pretty much lines up with what we tell people to do at CHTC:Âhttp://chtc.cs.wisc.edu/conda-installation.shtml


Josh Karpel


On Thu, Jul 23, 2020 at 8:25 AM Michael Pelletier via HTCondor-users <htcondor-users@xxxxxxxxxxx> wrote:

Iâve built a Singularity definition file that installs Miniconda and creates an environment YAML file, then builds the environment, and then configures the container so that the environment is activated automatically at the startup of the Singularity container. With Miniconda a CUDA 10.2 and 18.04 Ubuntu Singularity container file is about 850 megabytes in size.

Â

Singularity doesnât necessarily have an extra infrastructure layer, as it doesnât require any services from the host â I bet it would be possible to input-transfer the Singularity executable and run it on an input-transferred container.

Â

Alternatively, you could build out the full virtualenv in a directory with Miniconda, and then input-transfer that whole directory and activate it when the job starts up, which would eliminate the need for modules to be available on the exec node.

Â

Michael V Pelletier

Principal Engineer

Raytheon Technologies

Information Technology

Digital Transormation & Innovation

Â

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of West Matthew
Sent: Thursday, July 23, 2020 7:03 AM
To: htcondor-users@xxxxxxxxxxx
Subject: [External] [HTCondor-users] Conda env with workstation pool

Â

I am trying to run an analysis on my local workstation pool that relies on software in a conda virtual environment. When the job runs on a remote machine, it does not have access to the libraries in that env back on the submit machine.

Given that virtual environments are common practice when running locally, I an hoping there is some means of making python libraries accessible without resorting to an extra infrastructure layer like Docker.

Â

Cheers,

Matt

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/
_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/