High CPU usage of /usr/bin/landscape-package-reporter

Can you check if that happens for a long period of time? If so, for how long? landscape-package-reporter does basically apt-get update. Can you also run "sudo apt-get update" and check if there are errors or if it takes a long time to complete?

Hi Alex,

Thank you for the quick response. It takes landscape-package-reporter 30 seconds to finish in each interval, 'sudo apt-get update' only takes 5 seconds to finish.

Here is the details:

1. urgent-message-exchange-interval=120 in client.conf
2. landscape-package-reporter gets triggered every 120 seconds
3. When it gets triggered, apt-get update finished quickly in seconds, but landscape-package-reporter remains running for the next 30 seconds, and its CPU usage is almost 100%
4. When CPU is high, we can see a big chunk of data payload in broker.log, like this:
{'accepted-types': '\xaew\x01)\xe7\xfc\xec\x02fw\x9f\xfe\x1c\xd0l\xad',
 'client-api': '3.7',
 'messages': [{'api': '3.3', .......

Here is the settings in our client.conf file:

exchange_interval = 900
registration_key = **
log_level = debug
include_manager_plugins = ScriptExecution
url = https://**/message-system
computer_title = ABCD
apt_update_interval = 900
account_name = standalone
script_users = ALL
ping_url = http://**/ping
access_group =**
ssl_public_key = /etc/landscape/landscape_server.pem
urgent_exchange_interval = 120
package_monitor_interval = 1800
ping_interval = 60

Your client.conf contains a "log_level = debug" setting, which causes landscape-client to log a lot of debug information, including the content of messages being exchanged with the server. That causes increased cpu and IO usage.

Please try commenting/removing that line in the config, and restart landscape client (sudo service landscape-client restart).

Thanks a lot! Our QA will have another try when he has time and I will post result here afterwards.

Hello,

It looks like removing the log level did not help too much. It reduced running time from 30 seconds to 25 seconds, but still too long. Is there any other possibility can cause this? Thanks!

Here are some more information we found may help you advise:

If we keep both /etc/apt/sources.list and files in /etc/apt/sources.list.d/ folder, the landscape-package-reporter process takes more than 20 seconds.
If we keep only files in /etc/apt/sources.list.d/ folder and remove sources.list file, the landscape-package-reporter process takes seconds to finish.

Looking into sources.list file, we found that the slowness seems to be caused by fetching xenial/universe and xenial/multiverse repos. These 2 repos have large number of packages.
However, apt-get update is faster because apt-get update does not check into xenial/universe and xenial/multiverse every time, it caches for a while.

Do you know why landscape-package-reporter needs to check every repo in sources.list and sources.list.d every time it runs?
Is there a way to prevent the process checking repo so often or only update the repo we want?

Thanks!

This question was expired because it remained in the 'Open' state without activity for the last 15 days.

Canonical Team, any additional help here? Right now, its checking against every package in the public repos in sources.list.

What we'd like is a lightweight ping every 60 seconds, and only run an update if something is available to update. Is there a way to set this up with our registration parameters? Using package profiles?

Still happens with Ubuntu 20.04, here using around 100 seconds of CPU time (i7-4700MQ CPU) every half hour (with default desktop installation with no additional repo and basic landscape deployement). Uses 100% of one CPU over 100 seconds. Some noticeable effect on user interface, and fans triggered.

Hello Canonical, this issue still happens on newer version of Ubuntu as well. Can you please look into this?

Hi Vincent. Can you provide us with the version of landscape-client you are running, the version of Ubuntu it is installed on, and your client.conf file?

I confirm that this problem exists on the latest landscape client version 23.08+git6296-0ubuntu0

We are self-hosting landscape server on-premise

I include the package-reporter.log file, the timestamp start when I restart landscape-client

2024-06-13 09:30:11,344 INFO [MainThread] Downloaded hash=>id database from https://landscape.torc.tech/hash-id-databases/546032b8-5a2f-11ee-bc68-fffb099aadbe_jammy_amd64
2024-06-13 09:30:11,545 WARNING [MainThread] Removing cached hash=>id database /var/lib/landscape/client/package/hash-id/546032b8-5a2f-11ee-bc68-fffb099aadbe_jammy_amd64
2024-06-13 09:30:16,817 INFO [MainThread] Queuing request for package hash => id translation on 500 hash(es).
2024-06-13 09:31:00,538 INFO [MainThread] Queuing message with changes in known packages: 854 installed, 83936 available, 3 available upgrades, 0 locked, 1 autoremovable, 19900 security, 0 not installed, 0 not available, 0 not available upgrades, 0 not locked, 0 not autoremovable, 0 not security.
2024-06-13 09:31:25,918 INFO [MainThread] Downloaded hash=>id database from https://landscape.torc.tech/hash-id-databases/546032b8-5a2f-11ee-bc68-fffb099aadbe_jammy_amd64
2024-06-13 09:31:26,041 INFO [MainThread] Received 500 package hash => id translations, 0 hashes are unknown.
2024-06-13 09:31:31,149 INFO [MainThread] Queuing request for package hash => id translation on 500 hash(es).
2024-06-13 09:32:14,605 INFO [MainThread] Queuing message with changes in known packages: 56 installed, 482 available, 1 available upgrades, 0 locked, 0 autoremovable, 152 security, 0 not installed, 0 not available, 0 not available upgrades, 0 not locked, 0 not autoremovable, 0 not security.
2024-06-13 09:32:34,573 INFO [MainThread] Received 500 package hash => id translations, 0 hashes are unknown.
2024-06-13 09:32:39,776 INFO [MainThread] Queuing request for package hash => id translation on 500 hash(es).
2024-06-13 09:33:23,438 INFO [MainThread] Queuing message with changes in known packages: 62 installed, 475 available, 0 available upgrades, 0 locked, 1 autoremovable, 163 security, 0 not installed, 0 not available, 0 not available upgrades, 0 not locked, 0 not autoremovable, 0 not security.
2024-06-13 09:33:43,987 INFO [MainThread] Received 500 package hash => id translations, 0 hashes are unknown.
2024-06-13 09:33:49,160 INFO [MainThread] Queuing request for package hash => id translation on 500 hash(es).
2024-06-13 09:34:32,798 INFO [MainThread] Queuing message with changes in known packages: 50 installed, 476 available, 0 available upgrades, 0 locked, 1 autoremovable, 165 security, 0 not installed, 0 not available, 0 not available upgrades, 0 not locked, 0 not autoremovable, 0 not security.
2024-06-13 09:34:51,776 INFO [MainThread] Received 500 package hash => id translations, 0 hashes are unknown.
2024-06-13 09:34:56,889 INFO [MainThread] Queuing request for package hash => id translation on 500 hash(es).
2024-06-13 09:35:40,638 INFO [MainThread] Queuing message with changes in known packages: 53 installed, 476 available, 0 available upgrades, 0 locked, 0 autoremovable, 169 security, 0 not installed, 0 not available, 0 not available upgrades, 0 not locked, 0 not autoremovable, 0 not security.
2024-06-13 09:35:59,769 INFO [MainThread] Received 500 package hash => id translations, 0 hashes are unknown.
2024-06-13 09:36:04,961 INFO [MainThread] Queuing request for package hash => id translation on 500 hash(es).
2024-06-13 09:36:49,275 INFO [MainThread] Queuing message with changes in known packages: 65 installed, 477 available, 0 available upgrades, 0 locked, 2 autoremovable, 148 security, 0 not installed, 0 not available, 0 not available upgrades, 0 not locked, 0 not autoremovable, 0 not security.
2024-06-13 09:37:08,282 INFO [MainThread] Received 500 package hash => id translations, 0 hashes are unknown.
2024-06-13 09:37:13,397 INFO [MainThread] Queuing request for package hash => id translation on 500 hash(es).
2024-06-13 09:37:57,404 INFO [MainThread] Queuing message with changes in known packages: 63 installed, 475 available, 0 available upgrades, 0 locked, 0 autoremovable, 149 security, 0 not installed, 0 not available, 0 not available upgrades, 0 not locked, 0 not autoremovable, 0 not security.
2024-06-13 09:38:16,174 INFO [MainThread] Received 500 package hash => id translations, 0 hashes are unknown.
2024-06-13 09:38:21,333 INFO [MainThread] Queuing request for package hash => id translation on 500 hash(es).
2024-06-13 09:39:06,183 INFO [MainThread] Queuing message with changes in known packages: 74 installed, 478 available, 0 available upgrades, 0 locked, 1 autoremovable, 158 security, 0 not installed, 0 not available, 0 not available upgrades, 0 not locked, 0 not autoremovable, 0 not security.
2024-06-13 09:39:20,684 INFO [MainThread] Received 500 package hash => id translations, 0 hashes are unknown.
2024-06-13 09:39:25,870 INFO [MainThread] Queuing request for package hash => id translation on 500 hash(es).
2024-06-13 09:40:10,679 INFO [MainThread] Queuing message with changes in known packages: 58 installed, 479 available, 0 available upgrades, 0 locked, 1 autoremovable, 153 security, 0 not installed, 0 not available, 0 not available upgrades, 0 not locked, 0 not autoremovable, 0 not security.
2024-06-13 09:40:32,264 INFO [MainThread] Received 500 package hash => id translations, 0 hashes are unknown.
2024-06-13 09:40:37,431 INFO [MainThread] Queuing request for package hash => id translation on 500 hash(es).
2024-06-13 09:41:22,691 INFO [MainThread] Queuing message with changes in known packages: 62 installed, 484 available, 0 available upgrades, 0 locked, 0 autoremovable, 174 security, 0 not installed, 0 not available, 0 not available upgrades, 0 not locked, 0 not autoremovable, 0 not security.
2024-06-13 09:41:40,815 INFO [MainThread] Received 500 package hash => id translations, 0 hashes are unknown.
2024-06-13 09:41:45,984 INFO [MainThread] Queuing request for package hash => id translation on 500 hash(es).
2024-06-13 09:42:31,659 INFO [MainThread] Queuing message with changes in known packages: 59 installed, 487 available, 0 available upgrades, 0 locked, 0 autoremovable, 169 security, 0 not installed, 0 not available, 0 not available upgrades, 0 not locked, 0 not autoremovable, 0 not security.
2024-06-13 09:42:48,596 INFO [MainThread] Received 500 package hash => id translations, 0 hashes are unknown.
2024-06-13 09:42:53,821 INFO [MainThread] Queuing request for package hash => id translation on 500 hash(es).

This is the the timing config we currently have in client.conf file

apt_update_interval = 36000
package_monitor_interval = 3600
ping_interval = 60
exchange_interval = 900
urgent_exchange_interval = 60
snap_monitor_interval = 3600

Having this problem too in Ubuntu server 24.04.1 LTS.

There is a cron job that runs the hash-id-databases script. It runs weekly. Make sure it is running. This should help speed things up

In my case it was the package_monitor_interval, set to run every 3600 seconds, that consumed 100% CPU for 10 minutes each time it ran. I changed it to run every 4 hours instead. This problem only occurred on the server running on a Raspberry Pi 3 and not on the Raspberry Pi 5, which is considerably faster.

This seems to be a problem on both servers on which I am running landscape. How is it this bug is still ongoing after 9 years, or does it come and go over the years?

24.04 I'm having this same issue.

/etc/landscape/client.conf:

[client]
log_level = info
url = https://landscape.canonical.com/message-system
ping_url = http://landscape.canonical.com/ping
data_path = /var/lib/landscape/client
computer_title = zarko-ThinkPad-X230-Tablet
account_name = xxx
registration_key = xxx
package_monitor_interval = 78000

I've also noticed this high CPU usage (on a laptop as part of an organisation that uses landscape, so my main concern is battery consumption) and have done a bit of investigation on the client side.

As far as I can tell, each time it shows "Received 500 package hash => id translations" in the log, it will invoke an instance of `/usr/bin/landscape-package-reporter`, which will use nearly 100% CPU for almost a minute.

The vast majority of CPU time in `landscape-package-reporter` is due to the use of the "python-apt" property `Version.origins`, called multiple times for each package: https://git.launchpad.net/landscape-client/tree/landscape/client/package/reporter.py?h=24.12#n710 .

This command line basically simulates the use of the `.origins` property as used by each invocation of `landscape-package-reporter`:

```
$ time python3 -c 'import apt; print(len([origin for package in apt.cache.Cache() if package.name == package.shortname for version in package.versions for origin in [*version.origins, *version.origins, *version.origins]]))'
544719

real 0m52.465s
user 0m33.427s
sys 0m19.029s
```

Digging deeper, the reason this takes so long is because each time "python-apt" constructs an `Origin` object, it will call `find_index`, which involves walking through all package index files:
https://salsa.debian.org/apt-team/python-apt/-/blob/33f0c2b1d6e01df8194661642fa07c22127f4f86/apt/package.py#L308
https://salsa.debian.org/apt-team/python-apt/-/blob/33f0c2b1d6e01df8194661642fa07c22127f4f86/python/sourcelist.cc#L28
https://salsa.debian.org/apt-team/apt/-/blob/6ee18adb85633afc39bdca25209dcfac0ae5c3c5/apt-pkg/sourcelist.cc#L465

If I stub out the call to `find_index`, the command above only takes 1.3 seconds rather than 52.

I imagine the most efficient solution would be to get `python-apt` to cache the index objects per package index file instead of walking through the filesystem every time. Otherwise there could be some smaller gains (maybe cutting time to a third) within landscape-client by storing `package.origins` in a variable instead of invoking the property multiple times.

It probably also helps not having a large number of package index files (/var/lib/apt/lists/*_Packages). My work system has 37 of them—I'm not sure if this is because of Ubuntu or because of my organisation, but my personal Debian systems have 4 or fewer.

Hi Max, thanks for your thoughts and helping us to resolve this issue. I've been doing some testing with your suggested change and I'd like to share my findings.

Checkout my branch in landscape-client:

cd landscape-client
git fetch --all
git checkout lndeng-2048-package-origins-high-cpu

You will see I made the change you suggested in landscape/client/package/reporter.py on line 710.

I have been using our unit testing to run some benchmarking in landscape/client/package/tests/test_reporter.py on line 965:

def test_detect_packages_changes_benchmark(self):
        """
        Benchmark test for package change detection performance.
        """
        import statistics
        import time
        import base64

        NUM_ITERATIONS = 1000
        NUM_PACKAGES = 1000

        message_store = self.broker_service.message_store
        message_store.set_accepted_types(["packages"])

        hash_ids = {
            base64.decodebytes(
                base64.b64encode("/ezv4AefpJJ{i}DuYFSq4RiEHJYP4=".encode())
            ): i
            for i in range(NUM_PACKAGES)
        }
        self.store.set_hash_ids(hash_ids)

        stats = []
        for _ in range(NUM_ITERATIONS):
            start = time.perf_counter()
            self.reporter.detect_packages_changes()
            elapsed = time.perf_counter() - start
            stats.append(elapsed)

        self.fail(f"\n\nAverage: {statistics.mean(stats) * 1000} ms\n")

and doing 10 trials on my branch and main with:

for i in $(seq 1 10); do python3 -m twisted.trial landscape.client.package.tests.test_reporter.PackageReporterAptTest.test_detect_packages_changes_benchmark | grep Average: | awk '{print $2}'; done

You can find my observations here: https://docs.google.com/spreadsheets/d/13igujEUpvfU7_IylL1eetVxZqQAQ5UWPHLqoPziPe8E/edit?gid=0#gid=0

I wasn't able to get a significant performance increase as a result. Can you try out my testing to see if this is what you had in mind?

Thanks

Hi Jan-Yaeger,

Thanks for looking at this.

This might be due to my unfamiliarity with Launchpad, but I don't seem to be able to find your branch. It's not in the main "landscape-client" repository (https://git.launchpad.net/landscape-client), and it doesn't look like there's a landscape-client repository under your user.

I've added your test case manually, but it doesn't seem to trigger the issue, since the benchmark records about 0.2 ms, similar to in your spreadsheet.

I noticed your test has a bug though: it looks like the string that gets base64-encoded is meant to be interpolated, but it's missing the `f` prefix, so you end up with `len(hash_ids) == 1`. Fixing this still doesn't trigger the issue however.

I'm not familiar enough with landscape-client to reproduce it properly within a test case. After a bit of experimentation I can see that the test setup only has 1 "Packages" file (eg, "/tmp/tmpcxbyi5tr/var/lib/apt/lists/_tmp_tmp68rjyjh0_._Packages") rather than 37 that exist on my real Ubuntu system. Additionally there are only 3 test packages in the apt repository with one version each ("name1-version1-release1", "name2-version2-release2", "name3-version3-release3", can't see where these are generated), whereas my real system has 106235 relevant versions based on the following command:

  $ python3 -c 'import apt; [print(version) for package in apt.cache.Cache() if package.name == package.shortname for version in package.versions]' | wc -l
  106235

As I alluded to in my previous comment, I think the issue is dependent on the number of packages in the apt repository and the number of Packages files. I manually added some logging into "/usr/lib/python3/dist-packages/apt/package.py" to see when the `Origin` objects were created, and under your test it would only create 3 of them corresponding with the 3 packages in the test `AptFacade` instance, not 1000 (or 1 due to the aforementioned bug) corresponding with each hash id. I'd guess if the test setup can be adjusted to match a real system (37 package index files, 100k package versions), it would probably trigger the issue.

Hi again Max,

I've been going through Landscape Client to reproduce the issue and capture it in a test. Some of this legacy code can be quite nebulous honestly, but I think I may have gotten it.

git clone --single-branch --branch lndeng-2048-package-origins-high-cpu https://github.com/jansdhillon/landscape-client.git

python3 -m twisted.trial landscape.lib.apt.package.tests.test_facade.AptFacadeTest.benchmark

I've been using this test in landscape/lib/apt/package/tests/test_facade.py:

def benchmark(self):
        """
        Benchmark test for performance.
        """
        import time

        NUM_PKGS = 10000
        self.facade.clear_channels()
        start = time.perf_counter()

        for i in range(NUM_PKGS):
            name = f"test-{i}"
            self._add_system_package(name)
            self._hash_packages_by_name(
                package_name=name, store=self.store, facade=self.facade
            )

        self.facade.reload_channels()
        self.facade.get_packages()

        self.fail(f"\nRuntime: {(time.perf_counter() - start)} s\n")

Testing with:
python3 -m twisted.trial landscape.lib.apt.package.tests.test_facade.AptFacadeTest.benchmark

You can find the updated results here: https://docs.google.com/spreadsheets/d/13igujEUpvfU7_IylL1eetVxZqQAQ5UWPHLqoPziPe8E/edit?gid=0#gid=0

It does seem to be creating the specified amount of packages now, and we definitely see the slowdown. However, I'm not sure if the change I made ended up making a difference. Based off my understanding, this would involve a much bigger change than what I've done.

Please let me know your thoughts and if you are able to see my branch.

Hi again,

I think it has more to do with the number of package index files, since it's possible to have an arbitrary number of those (eg, about 37 on the Ubuntu systems I have access to) whereas the total number of packages will probably be similar across systems (eg, about 100k). The time to iterate over the packages is basically multiplied by the number of package index files, so 100k packages across 40 indexes will take 20 times as long as 100k packages across 2 indexes.

I don't think your latest benchmark test triggers the relevant code, since it doesn't seem to call `detect_packages_changes` and hence doesn't access the `.origins` property.

I've managed to put together this updated version of your original test (to be placed within "test_reporter.py", `PackageReporterAptTest`) that seems to trigger the issue:

    def test_detect_packages_changes_benchmark(self):
        """
        Benchmark test for performance.
        """
        import time

        NUM_PKGS = 10000
        NUM_LISTS = 400

        print(f"{time.perf_counter()}: generating packages")
        for i in range(NUM_PKGS):
            name = f"test-{i}"
            self._add_system_package(name)
            self._hash_packages_by_name(
                package_name=name, store=self.store, facade=self.facade
            )

        print(f"{time.perf_counter()}: generating lists")
        repos_dir = self.makeDir()
        for i in range(NUM_LISTS):
            repo_dir = os.path.join(repos_dir, f"{i}")
            os.makedirs(repo_dir)
            packages_file = os.path.join(repo_dir, "Packages")
            with open(packages_file, "wb") as packages:
                packages.write(b"")
            self.facade.add_channel_apt_deb(f"file://{repo_dir}", "./", trusted=True)

        print(f"{time.perf_counter()}: apt update")
        self.facade.reload_channels()
        start = time.perf_counter()
        print(f"{time.perf_counter()}: detecting packages changes")
        self.reporter.detect_packages_changes()
        print(f"{time.perf_counter()}: done")

        self.fail(f"\nRuntime: {(time.perf_counter() - start)} s\n")

I've used an exaggerated number of package index files (400 instead of 37), since the test setup takes long enough for only 10k packages, let alone 100k (looking at it through strace, adding each package to the test involves reading/writing the existing package list, so the setup time will be quadratic WRT the number of packages).

Normally this test reports 73 seconds for `detect_packages_changes` on my system, reduced down to 22 seconds if I modify the python-apt `Origin.__init__` method (usually installed in "/usr/lib/python3/dist-packages/apt/package.py") to do `self.trusted = True; return` before it calls `find_index`. Most of the remaining 22 seconds seems to be after the package iteration loop; haven't looked into what it's doing there.

You do seem to be correct however in that the change in landscape-client doesn't actually help that much, since it appears that the `.origins` property is usually only accessed once per package (accessing it a second/third time only occurs based on some *-backports/*-security logic that I don't fully understand, but would guess it doesn't apply to most packages).

I guess then this can only feasibly be solved by having "python-apt" cache the result of `find_index` against the `SourceList` object (`pkg._pcache._list` in `Origin.__init__`, which is seems to be reused across packages).

> Most of the remaining 22 seconds seems to be after the package iteration loop; haven't looked into what it's doing there.

Just to add to this, these 22 seconds are spent in `AptFacade.get_locked_packages`, and it looks like the time here is mostly dependent on the number of packages installed (or selected?). Eg, in this test scenario there are 10k packages installed, though on my real machine it's only 3.6k, so overall it's not very significant (maybe 8 seconds) compared to the other 51 seconds spent due to iterating the package lists. Also note that these 22 seconds are independent of the number of package lists.

Hi again Max, thanks again for your help and thoughts. I've been doing some refactoring of the code to avoid calling package.origins and I think I may have some promising results. Someone on my team pointed out that we only need the archives to check if the given package version is in the backport or security archives, which we can construct directly without calling package.origins. You can find my branch here: https://github.com/jansdhillon/landscape-client/tree/lndeng-2048-package-origins-high-cpu

I've included the git diffs and the benchmarking results below. As you can see, this does seem to reflect the speedup you were seeing by manually stubbing out the call to find_index in the apt package. And if we look at the function calls, find_index is only called in the unmodified code. Package reporter keeps working as expected as well. Are you able to try this new testing out and let me know if this is more what you had in mind?

Modified:

```
diff --git a/landscape/client/package/reporter.py b/landscape/client/package/reporter.py
index b784317b..36b697ab 100644
--- a/landscape/client/package/reporter.py
+++ b/landscape/client/package/reporter.py
@@ -654,6 +654,22 @@ class PackageReporter(PackageTaskHandler):
         return False

     def _compute_packages_changes(self): # noqa: max-complexity: 13
+ import cProfile
+ import pstats
+
+ profile = cProfile.Profile()
+ profile.enable()
+ result = self.compute_packages_change_inner()
+ profile.disable()
+
+ output_path = "/tmp/lib/landscape/client/result.txt"
+ with open(output_path, "a") as fp:
+ stats = pstats.Stats(profile, stream=fp)
+ stats.strip_dirs().sort_stats("cumulative").print_stats()
+
+ return result
+
+ def compute_packages_change_inner(self):
         """Analyse changes in the universe of known packages.

         This method will verify if there are packages that:
@@ -705,14 +721,11 @@ class PackageReporter(PackageTaskHandler):
             # support pinning, but we don't yet. In the mean time, we
             # ignore backports, so that packages don't get automatically
             # upgraded to the backports version.
- backport_origins = [
- origin
- for origin in package.origins
- if origin.archive == backports_archive
- ]
- if backport_origins and (
- len(backport_origins) == len(package.origins)
- ):
+ archives = [a[0].archive for a in package._cand.file_list]
+ backports_origins = all(
+ archive == backports_archive for archive in archives
+ )
+ if backports_origins:
                 # Ignore the version if it's only in the official
                 # backports archive. If it's somewhere else as well,
                 # e.g. a PPA, we assume it was added manually and the
@@ -736,10 +749,11 @@ class PackageReporter(PackageTaskHandler):

                 # Is this package present in the security pocket?
                 security_origins = any(
- origin
- for origin in package.origins
- if origin.archive == security_archive
+ archive
+ for archive in archives
+ if archive == security_archive
                 )
+
                 if security_origins:
                     current_security.add(id)

```

Results: https://pastebin.com/kJGPfPnN

Unchanged (with benchmark):

```
diff --git a/landscape/client/package/reporter.py b/landscape/client/package/reporter.py
index b784317b..faf64aa0 100644
--- a/landscape/client/package/reporter.py
+++ b/landscape/client/package/reporter.py
@@ -654,6 +654,22 @@ class PackageReporter(PackageTaskHandler):
         return False

     def _compute_packages_changes(self): # noqa: max-complexity: 13
+ import cProfile
+ import pstats
+
+ profile = cProfile.Profile()
+ profile.enable()
+ result = self.compute_packages_change_inner()
+ profile.disable()
+
+ output_path = "/tmp/lib/landscape/client/result.txt"
+ with open(output_path, "a") as fp:
+ stats = pstats.Stats(profile, stream=fp)
+ stats.strip_dirs().sort_stats("cumulative").print_stats()
+
+ return result
+
+ def compute_packages_change_inner(self):
         """Analyse changes in the universe of known packages.

         This method will verify if there are packages that:
```

Results: https://pastebin.com/8Tk5Kk3q

Hi,

Those changes seem to solve the performance issue, thanks! I don't fully understand the logic in the old or new code so can't comment on the correctness of it, but it seems to avoid calling the problematic code as you mentioned.

That's great to hear! I'll keep testing to ensure everything is still working as intended. For reference here is the updated spreadsheet: https://docs.google.com/spreadsheets/d/1LNKyu6FtlnjWpKuBkDDKWtODcoyaUS_GK0XeWsnHl4Y/edit?usp=sharing. I saw a 78% runtime reduction.

Hopefully we can get this fix out soon! Keep an eye on our repo (https://github.com/canonical/landscape-client) and thanks again for your help 😄

Just an update that I've opened a PR for this issue :)

https://github.com/canonical/landscape-client/pull/303

The PR has been approved and merged!

https://github.com/canonical/landscape-client/pull/303#pullrequestreview-2654614947