Re: [HTCondor-users] Condor removed after Ubuntu updates

[Nils Irland <nirland@xxxxxxxxxxxxxxxxx>]

Hello Todd,

 

Iâm also no packaging expert and have been reading likely the same stackexchange articles as you, and came across that same conclusion â apt upgrade is âsafeâ in that it canât break any existing packages, while apt dist-upgrade can remove packages if they conflict with OS updates. For now, Iâll switch my ansible code to instead just do an upgrade and not a dist-upgrade so I avoid accidentally breaking anything. If your team has any more insights Iâm all ears, and thanks again for your help!

 

Also, as another note, I re-kickstarted a VM and did the whole install process from scratch and condor installed and configured itself correctly and added the VM to the pool, so I do think that this issue I ran into had to do specifically with the state of my pool â the version of condor I was running and the OS upgrades that were being applied, as a clean install worked without issue using the same ansible code.

 

-Nils

 

From: Todd Tannenbaum <tannenba@xxxxxxxxxxx>
Sent: Wednesday, January 19, 2022 2:23 PM
To: Nils Irland <nirland@xxxxxxxxxxxxxxxxx>; HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: Re: [HTCondor-users] Condor removed after Ubuntu updates

 

On 1/19/2022 2:06 PM, Nils Irland wrote:

 

As for what Ansible is doing, hereâs the ansible code for the updating:

 

tasks:

    - name: Update Apt repo

      apt: update_cache=yes

    - name: Upgrade packages

      apt: upgrade=dist

 

Basically, just running âapt-get update; apt-get dist-upgradeâ on the machines.

Thanks for the reply.  Again, I am no packaging expert, but according to google:
'apt-get upgrade' will seek to upgrade the installed software packages if newer versions are found in the repositories. It doesn't trigger the removal of any packages. Meanwhile, apt-get dist-upgrade will delete some packages when required to do so during the upgrade of the software packages.

So I tested again using gethtcondor to install HTCondor v9.5.0 (current), and did 'apt-get dist-upgrade' this time.  (earlier today I tried with just 'apt-get upgrade').  Again I was not able to repro the problem, i.e. no HTCSS binaries were removed in my testing.

Let's see if staff better versed in packaging on Ubuntu can figure something out on Friday...

Thanks
Todd

 

Hopefully that helps answer any questions. One thing I can always do is version-lock condor until Iâve had time to test it out on one of the VMs in the future to make sure there arenât weird issue, but I wanted to see if this was something not expected, as it certainly surprised me.

 

Thanks again!

 

-Nils

 

From: Todd Tannenbaum <tannenba@xxxxxxxxxxx>
Sent: Wednesday, January 19, 2022 1:09 PM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Cc: Nils Irland <nirland@xxxxxxxxxxxxxxxxx>
Subject: Re: [HTCondor-users] Condor removed after Ubuntu updates

 

On 1/14/2022 2:44 PM, Nils Irland via HTCondor-users wrote:

Hello all,

 

Iâm a condor neophyte and have been playing around with some test setups to get the hang of it. Iâm running condor on Ubuntu 20.04.3 LTS and my simply environment consisted of a submit / central manager node and 3 execute nodes, all installed via the getcondor script method.

 

Today I ran updates on those machines to install the latest patches and condor disappeared from all of them. Looking at dpkg.log in /var/log I can see the notes of it getting uninstalled, but I have no idea why. Is this something else anyone else has run into? Obviously having this happen in production would be a bad thingâ so Iâd like to know what exactly happened but Iâm not sure where to start.

Hi Nils,

This is indeed strange and unexpected, thank you for reporting your experience!  From the dpkg.log below, the only item that looks perhaps non-benign is the upgrade of libclassad, which is a HTCondor suite component and has dependencies on the rest of HTCondor. 

I tried to recreate this, but failed to do so.  Specifically I tried:
   1. Using gethtcondor to install the latest minihtcondor (v9.5.0) into a Ubuntu 20.03 container, then while HTCondor was running I did "apt-get update; apt-get upgrade".  Ubuntu updated to v20.04, and HTCondor kept running along just fine.
   2. Next I tried the same as #1, but this time I installed minihtcondor v9.4 (specifically, apt-get install minihtcondor=9.4.0-1.1) in an attempt to match the version of HTCondor you were using.  No problems after updating this time either.

Later this week (~friday) folks here at HTCondor that deal with packaging on Ubuntu (specifically Tim T) will also try to re-create your issue on Ubuntu 20 - perhaps he will have better luck than I did.

It would certainly help to know how your machines were updating, ie exactly what Ansible is doing. 

I am no Ubuntu packaging expert, but perhaps there is some incompatibility between using apt-get (which is used by gethtcondor), and some dpkg method being used by Ansible to update?  Do apt and dpkg always play nicely together?

In the meantime, have all the binaries been removed?  I.e. what happens doing "ls /usr/bin/*condor*" and/or "ls /usr/sbin/*condor*" ? On a machine where you ran gethtcondor, are you able to get the HTCondor binaries back again by re-running gethtcondor ?

regards,
Todd

If anyone else has had this happen or knows a good place to look for artifacts of the uninstall, that would be great. I used Ansible to run the updates, so I donât have the full command output of apt as it was run, just a list of packages updated, which looks fairly benign:

 

ok: [compute01.genetics.wisc.edu] => {

    "result.stdout_lines": [

        "upgrade python3-pil:amd64 7.0.0-4ubuntu0.4",

        "upgrade firefox:amd64 95.0.1+build2-0ubuntu0.20.04.1",

        "upgrade ubuntu-advantage-tools:amd64 27.4.2~20.04.1",

        "upgrade linux-firmware:all 1.187.24",

        "upgrade libclassad15:amd64 9.4.0-1.1"

    ]

}

ok: [pxetest.genetics.wisc.edu] => {

    "result.stdout_lines": [

        "upgrade firefox:amd64 95.0.1+build2-0ubuntu0.20.04.1",

        "upgrade ubuntu-advantage-tools:amd64 27.4.2~20.04.1",

        "upgrade linux-firmware:all 1.187.24",

        "upgrade libclassad15:amd64 9.4.0-1.1"

    ]

}

ok: [submit.genetics.wisc.edu] => {

    "result.stdout_lines": [

        "upgrade libnss-systemd:amd64 245.4-4ubuntu3.14",

        "upgrade systemd-timesyncd:amd64 245.4-4ubuntu3.14",

        "upgrade systemd-sysv:amd64 245.4-4ubuntu3.14",

        "upgrade libpam-systemd:amd64 245.4-4ubuntu3.14",

        "upgrade systemd:amd64 245.4-4ubuntu3.14",

        "upgrade libsystemd0:amd64 245.4-4ubuntu3.14",

        "upgrade udev:amd64 245.4-4ubuntu3.14",

        "upgrade libudev1:amd64 245.4-4ubuntu3.14",

        "upgrade ubuntu-advantage-tools:amd64 27.4.2~20.04.1",

        "upgrade linux-firmware:all 1.187.24",

        "upgrade libclassad15:amd64 9.4.0-1.1"

    ]

}

ok: [exectest.genetics.wisc.edu] => {

    "result.stdout_lines": [

        "upgrade ubuntu-advantage-tools:amd64 27.4.2~20.04.1",

        "upgrade firefox:amd64 95.0.1+build2-0ubuntu0.20.04.1",

        "upgrade linux-firmware:all 1.187.24",

        "upgrade libclassad15:amd64 9.4.0-1.1"

    ]

}

 

Thanks very much, and if anyone has any ideas or would like more information, please let me know!

 

-Nils

 

Nils Irland

IT Manager, Laboratory of Genetics, UW-Madison

608-263-9898

nirland@xxxxxxxxxxxxxxxxx

Working remotely Tuesday and Thursday

 

_______________________________________________

HTCondor-users mailing list

To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a

subject: Unsubscribe

You can also unsubscribe by visiting

https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:

https://lists.cs.wisc.edu/archive/htcondor-users/

-- 

Todd Tannenbaum <tannenba@xxxxxxxxxxx>  University of Wisconsin-Madison

Center for High Throughput Computing    Department of Computer Sciences

Calendar: https://tinyurl.com/yd55mtgd  1210 W. Dayton St. Rm #4257

Phone: (608) 263-7132                   Madison, WI 53706-1685 

-- 

Todd Tannenbaum <tannenba@xxxxxxxxxxx>  University of Wisconsin-Madison

Center for High Throughput Computing    Department of Computer Sciences

Calendar: https://tinyurl.com/yd55mtgd  1210 W. Dayton St. Rm #4257

Phone: (608) 263-7132                   Madison, WI 53706-1685