Live Environment changes

Thanks for the detailed steps; I'll try it on my end.

(By the way, Cubic does exclude the contents of "/var/crash/*" when creating the ISO, but other directories in "/var" are not excluded).

What happens when you perform steps 11-16.
Then test the generated ISO in virtual box?
Do you still not see /var/log/cubic ?

will re-run now and test in vbox on the same machine without burning to USB

performed steps 11-16 to generate a new iso. I also ran the command groupadd cubic inside the live environment. I checked to make sure /var/log/cubic existed and the cubic group existed before finalizing the iso.

after the iso was created, i opened vbox on the same machine and created a new vm with the default of 1024 MB of memory and no virtual disk. I chose the test ISO for the optical drive and checked the Live CD/DVD box and started the VM.

I picked Try Ubuntu and in the live USB environment I pressed Alt F2 and ran the commands:
ls -al /var/log/ - the cubic directory was there
cat /etc/group - the cubic directory was there.

I used etcher to put the same iso I tested in Vbox on a USB drive to test on hardware. I booted the laptop i created the image on with the USB and ran the same commands:
ls -al /var/log/ - the cubic directory was NOT there
cat /etc/group - the cubic group was there

This is very interesting.
It looks Etcher is excluding this directory? (If so, why???).

Were other files or directories, that existed before you "burned" using Etcher, also missing?
(In other words was /var or /var/log completely empty after using Etcher?)

As you know, the /var directory contains files that need to persist across boots, so the files in /var should not be removed.

I checked an USB I had recently created using Startup Disk Creator (usb-creator-gtk), and all the files in /var were there.

All other files and directories are in /var on the live USB, just not the one I created inside the chroot environment. It's weird that using vbox, it works. maybe an init/sda mounting issue?

to see if it was etcher, I burned the ISO again using the ubuntu startup disk creator and the /var/log/cubic folder still missing in the live usb when I boot to the live environment on a laptop.

The changes to the file system must be on the disk, but the overlay must only apply to a virtual machine somehow.

What further confuses me is that this does not happen with any image based on ubuntu 18.04. Even an Ubuntu 18.04 image with the latest kernel

When I used Startup Disk Creator (usb-creator-gtk) to "burn" the 20.04 ISO, I didn't have the issue.

If you have time, you may want to try that and see if you get similar results.

I am still getting the same result using the Ubuntu Start-up disk creator.

To make sure I do not have an issue with my computer:

1. I took an old Lenovo T440 laptop and erased it
2. from another computer, I downloaded a fresh copy of ubuntu-20.04.1-desktop-amd64.iso from Ubuntu’s website and flashed it onto a new USB stick using Ubuntu start-up disk creator.
3. I installed stock Ubuntu 20.04.1 onto the Lenovo Laptop, taking all defaults (normal installation, download updates, erase disk and install Ubuntu)

I will not install anything else other than updates and cubic and will perform steps 11-19 above and post the results

after installing updates and adding the cubic release ppa, I installed cubic and followed steps 11-19.

I flashed the ISO on a USB stick using the Ubuntu startup disk creator preinstalled on 20.04.1 and booted to the disk.

The group was on the live environment, the directory was not in /var/log/

WOW!

I "burned" and ISO to USB and booted into the ISO.

The files in var/log are indeed different.

The files in Cubic were:

        - - - - - - - - - - - - - - - -
        alternatives.log
        apt
        bootstrap.log
        btmp
        cubic
        cups
        dist-upgrade
        dpkg.log
        faillog
        fontconfig.log
        gdm3
        hp
        journal
        lastlog
        openvpn
        private
        samba
        speech-dispatcher
        ubuntu-advantage.log
        unattended-upgrades
        wtmp
        - - - - - - - - - - - - - - - -

But the files in the live environment were:
        - - - - - - - - - - - - - - - -
        auth.log
        boot.log
        btmp
        casper.log
        cups
        dmesg
        gdm3
        gpu-manager.log
        installer
        kern.log
        lastlog
        private
        syslog
        unattended-upgrades
        vbox-setup.log
        wtmp
        Xorg.0.log
        Xorg.0.log.old
        - - - - - - - - - - - - - - - -

-------------------------
OBSERVATIONS
-------------------------
1.
Cubic does ~not~ exclude the customized contents of /var/log when generating the *.iso file.

2.
The generated *.iso file ~does~ have the files in /var/log that you put there while customizing.
I checked by manually extracting the customized Linux file system from the *.iso file.

3.
The USB created from the *.iso file ~also~ has the files in /var/log that you put there while customizing.
I checked by manually extracting the customized Linux file system from the USB.

4.
HOWEVER, when you boot into the USB, the contents of /var/log in the Live Environment is DIFFERENT.

5.
I do not know what happens if you install the OS from the USB.
a. Does it keep the contents of /var/log from the Live environment?
b. Does it keep the contents of /var/log from the customization on the USB?
c. Does it does it get an entirely new or different contents for /var/log when it 1st boots up?

-------------------------
CONCLUSION
-------------------------

The contents of /var/log is overlayd when the system boots up from the *.iso or USB.

-------------------------
NEXT STEPS
-------------------------

From a user perspective, I think you can ~not~ rely on the contents of /var/log being the same as what you placed there during customization.

If you are willing to spend the time, try testing 5b to see what happens when you actually install the OS. (I suspect the result will be 5c).

From a Cubic perspective, I am considering actively ~excluding~ the contents of /var/log since it doesn't seem to be used, and leaving it out would save some space in the generated ISO.

Note: I can still see the customized contents of /var/log in Virtual box; I'm not sure why there is a difference between booting into a USB vs booting the *.iso into Virtual Box, because the customized files are available in both the USB and the *.iso.

That is what confuses me as well. It works in virtual box, but not on physical media. AND 18.04 works in both formats. There has got to be a file system overlay introduced in 20.04 that is different

Maybe this is relevant
https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1877222