Category Archives: Ubuntu Cloud Images

Package changes between two Ubuntu images

I work on the Canonical Public Cloud team and we publish all of the Ubuntu server images used in the cloud.

We often get asked what the differences are between two released images. For example what is the difference between the Ubuntu 20.04 LTS image kvm optimised image from 20200921 and the Ubuntu 20.04 LTS image kvm optimised image from 20201014, specifically what packages changed and what was included in those changes?

For each of our download images published to http://cloud-images.ubuntu.com/ we publish a package version manifest which lists all the packages installed and the versions installed at that time. It also lists any installed snaps the the revision of that snap currently installed. This is very useful for checking to see if an image you are about to use has the expected package version for your requirements or has the expected package version that addresses a vulnerability.

Example snippet from a package version manifest:

<snip>
python3-apport	2.20.11-0ubuntu27.9
python3-distutils	3.8.5-1~20.04.1
</snip>

This manifest is also useful to determine the differences between two images. You can do a simple diff of the manifests which will show you the version changes but you can also, with the help of a new ubuntu-cloud-image-changelog command line utility I have published to the Snap store, determine what changed in those packages.

ubuntu-cloud-image-changelog available from the snap store
ubuntu-cloud-image-changelog available from the snap store

I’ll work through an example of how to use this tool now:

Using the the Ubuntu 20.04 LTS image kvm optimised image from 20200921 manifest and the Ubuntu 20.04 LTS image kvm optimised image from 20201014 manifest we can find the package version diff.

$ diff 20200921.1-ubuntu-20.04-server-cloudimg-amd64-disk-kvm.manifest 20201014-ubuntu-20.04-server-cloudimg-amd64-disk-kvm.manifest
<snip>
426c426
< python3-apport	2.20.11-0ubuntu27.8
---
> python3-apport	2.20.11-0ubuntu27.9
446c446
< python3-distutils	3.8.2-1ubuntu1
---
> python3-distutils	3.8.5-1~20.04.1
</snip>

This snippet above is a subset of the packages that changed but you can easily see the version changes. Full diff available @ https://pastebin.ubuntu.com/p/mzVBzfC5tw/ .

To see the actual changelog for those package version changes…

$ #install ubuntu-cloud-image-changelog
$ sudo snap install ubuntu-cloud-image-changelog
$ ubuntu-cloud-image-changelog --from-manifest=20200921.1-ubuntu-20.04-server-cloudimg-amd64-disk-kvm.manifest --to-manifest=20201014-ubuntu-20.04-server-cloudimg-amd64-disk-kvm.manifest
<snip>
Snap packages added: []
Snap packages removed: []
Snap packages changed: ['snapd']
Deb packages added: ['linux-headers-5.4.0-1026-kvm', 'linux-image-5.4.0-1026-kvm', 'linux-kvm-headers-5.4.0-1026', 'linux-modules-5.4.0-1026-kvm', 'python3-pexpect', 'python3-ptyprocess']
Deb packages removed: ['linux-headers-5.4.0-1023-kvm', 'linux-image-5.4.0-1023-kvm', 'linux-kvm-headers-5.4.0-1023', 'linux-modules-5.4.0-1023-kvm']
Deb packages changed: ['alsa-ucm-conf', 'apport', 'bolt', 'busybox-initramfs', 'busybox-static', 'finalrd', 'gcc-10-base:amd64', 'gir1.2-packagekitglib-1.0', 'language-selector-common', 'libbrotli1:amd64', 'libc-bin', 'libc6:amd64', 'libgcc-s1:amd64', 'libpackagekit-glib2-18:amd64', 'libpython3.8:amd64', 'libpython3.8-minimal:amd64', 'libpython3.8-stdlib:amd64', 'libstdc++6:amd64', 'libuv1:amd64', 'linux-headers-kvm', 'linux-image-kvm', 'linux-kvm', 'locales', 'mdadm', 'packagekit', 'packagekit-tools', 'python3-apport', 'python3-distutils', 'python3-gdbm:amd64', 'python3-lib2to3', 'python3-problem-report', 'python3-urllib3', 'python3.8', 'python3.8-minimal', 'secureboot-db', 'shim', 'shim-signed', 'snapd', 'sosreport', 'zlib1g:amd64']

</snip>

<snip>
======================================================================
python3-apport changed from version '2.20.11-0ubuntu27.8' to version '2.20.11-0ubuntu27.9'

Source: apport
Version: 2.20.11-0ubuntu27.9
Distribution: focal
Urgency: medium
Maintainer: Brian Murray < - >
Timestamp: 1599065319
Date: Wed, 02 Sep 2020 09:48:39 -0700
Changes:
 apport (2.20.11-0ubuntu27.9) focal; urgency=medium
 .
   [ YC Cheng ]
   * apport/apport/hookutils.py: add acpidump using built-in
     dump_acpi_tables.py. (LP: #1888352)
   * bin/oem-getlogs: add "-E" in the usage, since we'd like to talk to
     pulseaudio session and that need environment infomation. Also remove
     acpidump since we will use the one from hook.
 .
 apport (2.20.11-0ubuntu27.8) focal; urgency=medium
 .
   [Brian Murray]
   * Fix pep8 errors regarding ambiguous variables.

======================================================================
python3-distutils changed from version '3.8.2-1ubuntu1' to version '3.8.5-1~20.04.1'

Source: python3-stdlib-extensions
Version: 3.8.5-1~20.04.1
Distribution: focal-proposed
Urgency: medium
Maintainer: Matthias Klose <->
Timestamp: 1597062287
Date: Mon, 10 Aug 2020 14:24:47 +0200
Closes: 960653
Changes:
 python3-stdlib-extensions (3.8.5-1~20.04.1) focal-proposed; urgency=medium
 .
   * SRU: LP: #1889218. Backport Python 3.8.5 to 20.04 LTS.
   * Build as well for 3.9, except on i386.
 .
 python3-stdlib-extensions (3.8.5-1) unstable; urgency=medium
 .
   * Update 3.8 extensions and modules to the 3.8.5 release.
 .
 python3-stdlib-extensions (3.8.4-1) unstable; urgency=medium
 .
   * Update 3.8 extensions and modules to the 3.8.4 release.
 .
 python3-stdlib-extensions (3.8.4~rc1-1) unstable; urgency=medium
 .
   * Update 3.8 extensions and modules to 3.8.4 release candidate 1.
 .
 python3-stdlib-extensions (3.8.3-2) unstable; urgency=medium
 .
   * Remove bytecode files for 3.7 on upgrade. Closes: #960653.
   * Bump debhelper version.
 .
 python3-stdlib-extensions (3.8.3-1) unstable; urgency=medium
 .
   * Stop building extensions for 3.7.
   * Update 3.8 extensions and modules to 3.8.3 release.

======================================================================
</snip>

Above is a snippet of the output where you can see the exact changes made between the two versions. Full changelog available @ https://pastebin.ubuntu.com/p/cJVwVqzfgh/.

I have found this very useful when tracking why a package version changes and also if a package version change includes patches addressing a specific vulnerability.

We don’t yet publish package version manifests for all of our cloud images so to help in generating manifests I published the ubuntu-package-manifest command line utility to easily generate a package version manifest for any Ubuntu or Debian based image or running instance for later use with ubuntu-cloud-image-changelog.

ubuntu-package-manifest available from the snap store
ubuntu-package-manifest available from the snap store
$ sudo snap install ubuntu-package-manifest
$ # This is a strict snap and requires you to connect the system-backup interface
$ # https://snapcraft.io/docs/the-system-backup-interface 
$ # to access the host system package list. This is access read-only.
$ snap connect ubuntu-package-manifest:system-data
$ sudo ubuntu-package-manifest

You can even use this on a running desktop install to track package version changes.

ps. We’re hiring in the Americas and in EMEA 🙂

Ubuntu cloud images and how to find the most recent cloud image – part 2/3

TLDR;

sudo snap install image-status

This will install a snap of the very useful `image-status` utility.

image-status cloud-release bionic

This will show you the serial for the most recent Ubuntu 18.04 Bionic cloud image in QCOW format.

image-status ec2-release bionic

This will show you the AWS EC2 AMIs for the most recent Ubuntu 18.04 Bionic AWS EC2 cloud images.


Part two of a three part series.

Following on from part 1 where I detailed simplestreams and sstream-query I present to you the `image-status` utility which is a very neat and useful wrapper around sstream-query.

image-status is hosted on github as part of Scott Moser‘s talk-simplestreams repo.

I recently submitted a pull request which added the ability to package image-status as a snap. This was merged and you can now install image-status on any linux distribution supporting snaps using the following command.

sudo snap install image-status

Once installed you can start querying the simplestreams feeds for details on the most recent Ubuntu cloud images.

Usage:

image-status --help # to see all available options

image-status cloud-release bionic # to see most recent Ubuntu Bionic release images on http://cloud-images.ubuntu.com/
image-status cloud-daily bionic # to see most recent Ubuntu Bionic daily images on http://cloud-images.ubuntu.com/

image-status gce-release bionic # to see most recent Ubuntu Bionic release images on GCE
image-status gce-dailybionic # to see most recent UbuntuBionic daily images on GCE

image-status ec2-release bionic # to see most recent Ubuntu Bionic release AMIs on EC2
image-status ec2-daily bionic # to see most recent UbuntuBionic daily AMIs on EC2

image-status azure-release bionic # to see most recent Ubuntu Bionic release images on Azure
image-status azure-daily bionic # to see most recent UbuntuBionic daily images on Azure

image-status maas-release bionic # to see most recent Ubuntu Bionic release images for maas V2
image-status maas-daily bionic # to see most recent UbuntuBionic daily images for maas V2

image-status maas3-release bionic # to see most recent Ubuntu Bionic release images for maas V3
image-status maas3-daily bionic # to see most recent Ubuntu Bionic daily images for maas V3

I find this very useful when trying to quickly see what is the most recent Ubuntu release on any particular public cloud. eg:

image-status ec2-release bionic | grep eu-west-1 | grep hvm | grep ssd | awk '{split($0,a," "); print a[6]}'

This will return the ID for the most recent HVM EBS Ubuntu 18.04 (Bionic) in the eu-west-1 AWS EC2 region. This can be achieved using sstream-query too but I find filtering using grep to be easier to understand and iterate with.

I hope the above is helpful with your automation.

Ubuntu cloud images and how to find the most recent cloud image – part 1/3

TLDR;

sstream-query --json --max=1 --keyring=/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg http://cloud-images.ubuntu.com/releases/streams/v1/com.ubuntu.cloud:released:download.sjson arch=amd64 release_codename='Xenial Xerus' ftype='disk1.img' | jq -r '.[].item_url'

This will show you the URL for the most recent Ubuntu 16.04 Xenial cloud image in QCOW format.


Part one of a three part series.

There are a few ways to find the most recent Ubuntu cloud image an the simplest method is to view the release page which lists the most recent release.

Another method is to use the cloud image simple streams data which we also update every time we (I work on the Certified Public Cloud team @ Canonical) publish an image.

We publish simple streams data for major public clouds too but this post deals with the base Ubuntu cloud image. I will follow up this post with details on how to use the cloud specific streams data.

Simple streams

Simple streams is a structured format describing the Ubuntu cloud image releases.

You can parse the Ubuntu’s release cloud image stream json yourself or you can use a combination of sstream-query and jq (install packages “ubuntu-cloudimage-keyring“, “simplestreams” and “jq“) to get all or specific data about the most recent release.

Query all data from most recent release

sstream-query --json --max=1 --keyring=/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg http://cloud-images.ubuntu.com/releases/ arch=amd64 release='xenial' ftype='disk1.img'

This will return all data on the release including date released and also the checksums of the file.

[
 {
 "aliases": "16.04,default,lts,x,xenial",
 "arch": "amd64",
 "content_id": "com.ubuntu.cloud:released:download",
 "datatype": "image-downloads",
 "format": "products:1.0",
 "ftype": "disk1.img",
 "item_name": "disk1.img",
 "item_url": "http://cloud-images.ubuntu.com/releases/server/releases/xenial/release-20180126/ubuntu-16.04-server-cloudimg-amd64-disk1.img",
 "label": "release",
 "license": "http://www.canonical.com/intellectual-property-policy",
 "md5": "9cb8ed487ad8fbc8b7d082968915c4fd",
 "os": "ubuntu",
 "path": "server/releases/xenial/release-20180126/ubuntu-16.04-server-cloudimg-amd64-disk1.img",
 "product_name": "com.ubuntu.cloud:server:16.04:amd64",
 "pubname": "ubuntu-xenial-16.04-amd64-server-20180126",
 "release": "xenial",
 "release_codename": "Xenial Xerus",
 "release_title": "16.04 LTS",
 "sha256": "da7a59cbaf43eaaa83ded0b0588bdcee4e722d9355bd6b9bfddd01b2e7e372e2",
 "size": "289603584",
 "support_eol": "2021-04-21",
 "supported": "True",
 "updated": "Wed, 07 Feb 2018 03:58:59 +0000",
 "version": "16.04",
 "version_name": "20180126"
 }
 ]

Query only the url to the most recent release

sstream-query --json --max=1 --keyring=/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg http://cloud-images.ubuntu.com/releases/streams/v1/com.ubuntu.cloud:released:download.sjson arch=amd64 release_codename='Xenial Xerus' ftype='disk1.img' | jq -r '.[].item_url'

This will show you the URL for the most recent Ubuntu 16.04 Xenial cloud image in QCOW format.

"http://cloud-images.ubuntu.com/releases/server/releases/xenial/release-20180126/ubuntu-16.04-server-cloudimg-amd64-disk1.img"

Query only the serial of the most recent release

sstream-query --json --max=1 --keyring=/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg http://cloud-images.ubuntu.com/releases/ arch=amd64 release_codename='Xenial Xerus' ftype='disk1.img' | jq ".[].version_name"

This will show you the serial of the most recent Ubuntu 16.04 Xenial cloud image.

"20180126"

The above streams are signed using keys in the ubuntu-cloudimage-keyring keyring but you can replace the –keyring option with –no-verify to bypass any signing checks. Another way to bypass the checks is to to use the unsigned streams.

It is also worth noting that OpenStack can be configured to use streams too.

I hope the above is helpful with your automation.