XUbuntu Install Touched a Drive I told it not to

I did an XUbuntu install, manually partitioned an external drive, and selected "install bootloader to this drive"
Rebooted, went into my UEFI and booted the external device, worked fine.
Unplugged the device, tried to boot the system off my internal Windows drive, it failed to boot.

Went into UEFI boot manager, and was dismayed to see that Ubuntu installed a bootloader both on my internal disk AND the external disk. I selected the Windows boot manager and booted windows.

I was pretty dismayed that a Linux distro touched a disk I didn't tell it to, I would expect that more out of Windows.

I mounted the EFI partition on Windows and I see there is an "ubuntu" folder in there

B:\EFI\ubuntu>dir
 Volume in drive B has no label.
 Volume Serial Number is 64B6-49B5

 Directory of B:\EFI\ubuntu

01/04/2018 03:03 AM <DIR> .
01/04/2018 03:03 AM <DIR> ..
01/04/2018 03:03 AM <DIR> fw
01/04/2018 03:18 AM 67,176 fwupx64.efi
01/04/2018 03:03 AM 1,111,928 grubx64.efi
01/04/2018 03:03 AM 121 grub.cfg
01/04/2018 03:03 AM 1,169,992 shimx64.efi
01/04/2018 03:03 AM 1,168,464 mmx64.efi
01/04/2018 03:03 AM 108 BOOTX64.CSV
               6 File(s) 3,517,789 bytes
               3 Dir(s) 70,635,520 bytes free

What did this do, and how do I clean it up?

I know with MBR based installs - Grub will install to the first MB or so of the drive, with a small kickstarter in the first block on the drive to load Grub.

UEFI Grub, I don't know what it does, did it do anything at a raw disk level to my Windows disk, or just put files in the EFI partition?