init: support chroots

yes.... I also have this issue:

start: Unable to connect to Upstart: Failed to connect to socket /com/ubuntu/upstart: Connection refused

In fact how to fix it?

Replacing /sbin/initctl with a symlink to /bin/true is a fairly standard
way to disable services in a chroot.

Scott
--
Scott James Remnant
<email address hidden>

Sounds like we'll need to write a release note to ensure that people are informed of this, since in practice it is a change in behaviour from what people are used to. I've added a bug task for that.

Since there's now a Release Notes target, I don't think there's any outstanding fix for this for Upstart

Unfortunately there's the same problem in vservers.

Things like this won't work any more:
 sudo reboot (needs to be restarted from "outside the guest")
 sudo start/stop/restart <service> (very problematic as there's a few daemons already that don't ship "old style" init scripts any more: cron, rsyslog, probably others)

Text added at <https://wiki.ubuntu.com/KarmicKoala/ReleaseNotes#Upstart%20jobs%20cannot%20be%20run%20in%20a%20chroot>:

Upstart jobs cannot be started in a chroot because upstart acts as a service supervisor, and processes within the chroot are unable to communicate with the upstart running outside of the chroot (Bug:430224). This will cause some packages that have been converted to use upstart jobs instead of init scripts to fail to upgrade within a chroot. Users are advised to configure their chroots with /sbin/initctl pointing to /bin/true, with the following commands run within the chroot:

{{{
dpkg-divert --local --rename --add /sbin/initctl
ln -s /bin/true /sbin/initctl
}}}

I don't think release noting this is a very satisfactory solution in the long term.

> Replacing /sbin/initctl with a symlink to /bin/true is a fairly standard
> way to disable services in a chroot.

No; a) it's an upstart-specific way to disable services in a chroot, and b) this bug isn't about how to configure chroots so that services don't run in them, it's about the fact that chroots now require additional configuration in order for package upgrades to not fail. I think more discussion is still warranted here.

Would you like to begin that discussion?

Do you feel I should file a separate ubuntu-releasenotes / upstart bug about vserver related problems?

On Mon, 2009-10-26 at 06:11 +0000, Daniel Holbach wrote:

> Do you feel I should file a separate ubuntu-releasenotes / upstart bug
> about vserver related problems?
>
No, because I'll just mark it Won't Fix - the problems are with vserver,
not Upstart

Scott
--
Scott James Remnant
<email address hidden>

Is there a reason that initctl couldn't respect /usr/sbin/policy-rc.d?

Especially since any real-world policy-rc.d file just exits 101 (disallowed) 99% of the time, it doesn't seem too problematic if sysadmins have to rewrite theirs to deal with a few more potential arguments from initctl.

Is there some other solution than disabling the initctl? I use initrds that deploy vmware/quemu/bare instances via network, where most of the initial installation operation is done using debootstrap (reports error when rsyslog is configured) and chroot to the target disk (install, update fails) before the instance is really booted.

At the moment I move initctl to tmpfile at start of target setup and move it back afterwards. Does this have any side effects on functionality of installed packages?

This is a problem for builders/testers, not just people using virtual machines.

Making it harder to customize/test ubuntu will reduce quality going forward.

I build a customized LiveCD for a research group at Caltech, which is also made available to similar groups at other universities.

Package installation into a chroot'ed copy of a distribution is required by both customization methodologies described in the ubuntu wiki. The two methodologies are 'by scratch' starting with debootstrap and then chrooting; or 'from an existing distribution' which copies and unpacks the squashfs in the standard LiveCD and then one chroots into that. The chroot allows upgrading and interacting with the system in a familiar and straightforward way, using, for example, apt-get to fetch new packages or dpkg-reconfigure to change a configuration.

Not only will this issue affect tinkerers and customisation builders, I would think Ubuntu's internal testers might need to do similar things when building/testing future distributions.

I started a discussion regarding this at Ubuntu Forums in Lucid testing:

http://ubuntuforums.org/showthread.php?t=1326721

Already one question I can't answer. Following:

dpkg-divert --local --rename --add /sbin/initctl
ln -s /bin/true /sbin/initctl

and completing the upgrade or other procedure, should a person not:

rm /sbin/initctl
dpkg-divert --local --remove /sbin/initctl

before exiting the chroot?

This is quite important to me because I try to help as much as possible at Absolute Beginners Talk in the forums and every failed attempt at helping someone holds the potential of losing a new Ubuntu user.

That and I don't like to fail. It's one of those character flaws.

If you're running a chroot or some other environment where you never
want upstart jobs to run, you shouldn't un-divert the initctl.

On Mon, Nov 16, 2009 at 9:04 AM, Erick Brunzell <email address hidden> wrote:
> I started a discussion regarding this at Ubuntu Forums in Lucid testing:
>
> http://ubuntuforums.org/showthread.php?t=1326721
>
> Already one question I can't answer.  Following:
>
> dpkg-divert --local --rename --add /sbin/initctl
> ln -s /bin/true /sbin/initctl
>
> and completing the upgrade or other procedure, should a person not:
>
> rm /sbin/initctl
> dpkg-divert --local --remove /sbin/initctl
>
> before exiting the chroot?
>
> This is quite important to me because I try to help as much as possible
> at Absolute Beginners Talk in the forums and every failed attempt at
> helping someone holds the potential of losing a new Ubuntu user.
>
> That and I don't like to fail. It's one of those character flaws.
>
> --
> misc: packages cannot be upgraded in a chroot
> https://bugs.launchpad.net/bugs/430224
> You received this bug notification because you are a direct subscriber
> of the bug.
>

But if you are running a chroot where you will want upstart to run, for example, when building a live cd environment from scratch, you do want to remove the diversion before you exit.

Running:

rm /sbin/initctl
dpkg-divert --local --remove /sbin/initctl

breaks gdm in both Karmic and mint Helena.

Which leaves egg on my face since I'd included it in my chroot script for some time!

How can I fix gdm?

In regards to that last post, the problem was easily fixed just by an install --reinstall upstart.

I'm marking this as invalid as its only actually sysvinit scripts when policy-rc.d hasn't been configured correctly that causes problems; native upstart jobs seem to just ignore not being able to connect to upstart regardless.

> native upstart jobs seem to just ignore not being able to connect to upstart

No, they do not. There are various stock maintainer script snippets that will ignore these failures on the maintainer script's behalf, but:
a) this removes an important feedback channel about other problems that cause services to fail to start on package upgrade
b) it prevents running any services in chroots, which up to now has been supported.

I just noticed this problem has broken cron in some of my chroots.

There needs to be some standard way to run services such as cron within a chroot. If nothing else, at least a convenience script that can start and stop services outside of upstart supervision.

My understanding is that this bug is marked incomplete because it's currently lacking discussion on how we should handle this.

I propose that the way to resolve this is to implement chroot awareness to upstart and its commands. Perhaps either of these is possible:
- allow starting a separate upstart in each chroot to start/track/manage processes in there
- allow the first upstart to track processes in chroots and add a communication channels in chroots back to the main upstart
- provide a set of compatibility commands to start/stop/restart upstart jobs in chroots keeping basic track of them but not following/respawning them, perhaps in a new upstart-chroot-compat package
- other ideas?

You can only have one init daemon on a system, so having a separate Upstart per-chroot doesn't work - the prime system's init daemon would get the SIGCHLDs, etc. (unless you use pid namespaces and a kernel patch that Lennart once wrote to redirect init behaviour to other pids)

For me, the right solution would be that the init daemon could be made aware of chroots, e.g. through a config file. It'd then read the /etc/init directory inside the chroot, and manage those jobs along-side those in the real system - just automatically chrooting before processing the job.

The only thing missing there is some way to determine *which* job you mean; since it wouldn't be obvious to Upstart which chroot the "initctl" command was being run from

initctl blocks on Upstart returning, right?

Could you do something clever with /proc/$INITCTL_PID/root?

initctl doesn't have to block, no (--no-wait)

Moved this to be an Upstream bug.

I think that the most probably implementation will be that you declare chroots in /etc/init.conf that you wish the init daemon to manage. init will then also look in CHROOT/etc/init for jobs, and record these as belonging to that chroot. All actions on these jobs will first result in the chroot() to the right location.

To figure out still is how to make sure that "start apache" within a chroot, starts the apache within the chroot and not the one outside.

It makes sense that "initctl list" only shows the jobs in your chroot (or the real system).

The difficult bit is going to be dealing with things like events; I think logically events are global (you can only have one udev running) - but this means that "started apache" is an event - and that means chroots won't necessarily work right?

Using a global upstart instance to manage the chroot wouldn't work - simply because you want to be able to run a Ubuntu chroot from a Debian Lenny host, or a RHEL host, or whatever. (We actually serve a Lucid netboot nfs-root from Debian Etch atm). You can't expect every linux distribution in the world to use upstart.

Just a hint to keep that in mind when fixing this bug, please.

When chrooting on a broken system, I'd like at least to have a way to start the services manually. I don't care about supervision. Just that, for example, mysqld is started with the system defaults so that I can check it's state.

What about adding an option: `start --chroot <service_name>` where the service would be run with the /etc/defaults/<service-name> , but that would not fork/background.

I followed http://linux-vserver.org/Upstart_issues regarding this problem in connection with VServer and upstart. It fixed it.

I don't know if this sheds any light over a possible way to do things, but I hope so :)

Regards

Fixed it for mysql /apache that I installed anyhow.. not for deactivating swap and what not.
so I could do stop/start mysql , but it still gets connection refused when the vserver shuts down.
Oh well.

This really is a major pain for system builders. Had to work around this problem today to allow TKLPatch to work with Lucid. My approach was to temporarily lobotomize upstart, replacing it with a shell script that reproduces the sysvinit functionality in the old init scripts:

http://github.com/turnkeylinux/tklpatch/commit/a2bd433ed9483b5faaa631098eb372eecc0e4f12

This way we can run start and stop mysql "normally" inside chroot. It's a hack, but if we need we can extend this approach to other services as well. Maybe someone else will find the approach useful.

Other server services which are known to use upstart in Lucid are samba, vsftpd, squid. But I think for tklpatch we only need the ability to start mysql within the chroot (e.g., to manipulate the databases).

If upstart can't support chrooted services out of the box, it would be nice if it was at least configurable (e.g., via an environment variable) so you could enable a workaround of some kind for when you actually do need to start/stop services inside chroot. I don't see why you couldn't bypass communication with the supervision god-process over dbus and just execute the initialization hooks directly. You know, the way we've been doing it with sysvinit forever...

That way you get all the wiz bang async service management stuff while still allowing services to work when you don't need/want it.

The same problem.
I need to run some service in chroot.
I've done:
mount -R /dev /mnt/dev
mount -R /proc /mnt/proc
mount -R /var/run /mnt/var/run
mkdir -m 777 /var/run/mysqld
chroot /mnt
Everything works fine, but still can't figure out how to run simple service!
Why /etc/init.d/mysql start doesn't work anymore?

Similar issues:
http://newyork.ubuntuforums.org/showthread.php?t=1506533&page=2

Workaround is to run it manually:
mysqld --user mysql

if you are not able to make the mounts like described above try it like when using a Live-CD:

# sudo mount -o bind /dev /mnt/dev
# sudo mount -o bind /sys /mnt/sys
# sudo mount -t proc /proc /mnt/proc
# sudo cp /mnt/proc/mounts /mnt/etc/mtab
# sudo chroot /mnt /bin/bash

see also: http://wiki.ubuntuusers.de/chroot/Live-CD

hopefully you might recover things now...

I have setup where I put clients in chroots, each client has it's own chroot. has been running on jaunty, and everything was great.

Now I upgraded one chroot to lucid, and I can't start services in client chroot:

[chroot] # /etc/init.d/atd start
Rather than invoking init scripts through /etc/init.d, use the service(8)
utility, e.g. service atd start

Since the script you are attempting to invoke has been converted to an
Upstart job, you may also use the start(8) utility, e.g. start atd
start: Unable to connect to Upstart: Failed to connect to socket /com/ubuntu/upstart: Connection refused

[chroot] # start atd
start: Unable to connect to Upstart: Failed to connect to socket /com/ubuntu/upstart: Connection refused

While all talk about future enhancements is great - what I can do *now* to get the atd/crond/others to run in the chroot?

We've been hit by this, too and managed to create a workaround that requires no changes in the initscripts of the services and still allows starting them the traditional way in a Lucid-based chroot. See the (quite long) description of or problems and our solution in this blog post: http://gyp.blogs.balabit.com/2011/01/using-upstart-in-a-chroot/

To quote the most important part:

Thus we wrote “upstart-dummy“, an almost-drop-in replacement for Upstart. It’s a Python script that can be put in place of the standard Upstart installation in /sbin/initctl. The commands “start”, “stop”, “restart”, “reload” and “status” are symlinked to /sbin/initctl by default and this script is able handle it when it is called through these symlinks. It tries to find the appropriate config file for the service, parse it and run the necessary commands, including pre-start, post-stop etc. scripts that can be found in them. What it cannot do though, and this is why it’s not a complete replacement, is to work as proper standalone init process: it is not able to figure out the dependencies and does not know what services it needs to start on a certain runlevel. But if you’re using it on a system that kept the System V init compatibility layer (and Lucid is one of them), all you need to do is to re-add the symlinks in “/etc/rc*.d” to “/etc/init.d/“, which will be a symlink to “/lib/init/upstart-job“, which will call this fake Upstart daemon, which will, this time, indeed work in a chroot. After this, you can simply start “/etc/init.d/rcS” and “/etc/init.d/rc 2” (which are still there in a stock Lucid) which will boot the chrooted system properly.

It might not be suitable for everyone but it did the job for us.

Code to support chroot sessions has been written (lp:~canonical-scott/upstart/session-support), and will hopefully land in the not too distant future.

This breakage is really another major nuisance that upstart creates. Booh! I waited for a while before updating my vserver to lucid assuming that the wrinkles would be ironed out by now only to find that the system is now completely unusable (not even ssh will start).

Colin, will that code need to run on the host or guest?

Would it be possible to see something similar to what Peter proposed in comment 35 available as a package (officially if at all possible)? Not being able to use the latest LTS as a vserver is a very terrible joke. I'd even be willing to help with the packaging. Peter, are you taking the bait?

Rolf, my solution is more a hack than a real solution -- it works fine in the very controlled environment I'm using but even I wouldn't recommend adding it officially to any distribution. The way I'd go is to backport the new upstart that is said to include support for chroots (it'll be need to be installed in the guests). I plan to play around with it in the upcoming weeks if I have the time and report back about my results.

Scott, are there any plans to officially backport a newer upstart (that has support for chroots) to Lucid? Are there any incompatible changes that'd make it more difficult than just compiling & packaging it?

The new upstart has a lot of other things that make it a little too invasive for lucid-backports. Certainly we can create a PPA for it though.

Peter, would it be that hard to make upstart-dummy work for 99% of cases with a little bit of TLC development? I feel a bit silly as I just posted on your blog before reading this comment. ;)

Clint, the honest answer is that I don't know :) The problem is that it's really just a bring-me-back-my-good-old-system-V-init type of hack, and any boot process that truly relies on upstart doing its more advanced things (handling dependencies, using triggers, restarting failed services etc.) would fail miserably. I could manually check and fix all the ~15 services that are started this way in my case, but I'm not sure it's what people want to do. However, the good news is that in my case, none of the initscripts had to be modified, but I was not comfortable until I've manually checked all of them.

That being said, if you took a look at what it does and as a someone having deeper knowledge of the ubuntu boot process think that it could work for others, too, I guess it could be at least something for the guys trying to run lucid in a chroot instead of the current "well, it's not working" state of things.

Regarding your question on my blog (sorry, the commenting system there's a mess, I haven't looked at it since the web guys moved it to wordpress, I tried to fix things now) about bringing it to Launchpad: of course, feel free to do that. It indeed does not have a bug tracker or anything as I did not mean it to be a proper released product but just something I could give back to the FOSS community. And I'm happy to handle bug entries and fix small issues as they come up, so you can count me as an active developer base :) There's one thing though: it relies heavily on having an additional feature in start-stop-daemon, so that'd need to be added to that, too. I've sent the patch for it in to the dpkg devs, but the guys there were a bit reluctant to include it upstream as they thought it could allow for even more nasty hacks :) That discussion can be found here: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=610719

James, thank you for working on this. Can you please leave a short comment how this was fixed and when it's due to hit Ubuntu? Are we going to see a backport of the fix?

Rolf, this is actually available in natty.

I'm not entirely certain of the mechanics thus far, but I know that schroot takes advantage of it perfectly. You can see how it works by setting up an sbuild environment and then schrooting in, and installing mysql. It will install mysql, and start the upstart job *inside* the chroot.

Marking Fix Released in the Ubuntu task.

Chroot support is now available in Upstart 1.3.

Thank you, James. Does this ticket need to be reopened for oneiric? 1.3 has not yet been released to any Ubuntu series. I'm surprised things seem to be working for Clynt. Are you running a self-compiled package?

Does the host or the guest need to have upstart >= 1.3? I suppose it's for the host in which case I really think we need some kind of a backport, even to lucid-updates. All hardy hosts on a vserver currently have no working upgrade path.

Its already available in Natty and Oneiric. It was made as an Ubuntu-specific patch for 0.9.x.

OK

What can we do about lucid?

Excerpts from Rolf Leggewie's message of Sat Jul 16 14:04:17 UTC 2011:
> got a bit mixed up with the versions. It is hardy vservers that
> currently have no update path to anywhere., intrepid is already EOL and
> an update to lucid is impossible because of bug 430224.

Rolf, I see your point, and I think we can offer a "less than perfect"
solution.

I've created a daily build PPA for the latest version of Upstart in
Ubuntu, which will build an upstart with chroot support for lucid and
maverick.

I'll just host it under my own PPA for now, but eventually we should
put this under the ubuntu-server-edgers team stewardship. Once 12.04 is
released, we can just freeze this PPA and that would cover lucid upgrades
for its lifecycle.

The PPA is

ppa:clint-fewbar/upstart

It should build upstart in the next 20 minutes or so.

Maybe you can give it a try on lucid and if it seems to work and provide
the chroot support you're looking for, we can add it to the edgers setup
and for users who have extensive chroot needs, this will work.

I already talked with the backports maintainer, Scott Kitterman, and he
felt that upstart is too core to the system to provide in the official
backports.

Clynt, thank you for taking this up.

My point is essentially the same as was raised by Lukas in comment 27. We need to be sure that a lucid guest will work no matter the host OS. I'm not opposed to a PPA workaround but IMHO some code should be added probably to dapper somewhere to make sure that if it's detected that dapper is running inside a chroot/vserver an update to lucid is not performed without that PPA workaround. Or else the user will run into the situation where the vserver is completely unreachable because ssh is down (happened to me, booh!).

I will have to say that I'm a bit disappointed this wasn't taken into account seriously much earlier. The warnings were there early enough.

Your PPA was building packages for natty right? I'm not sure what direction it is you're taking there, but I'm sure you will let me know. Are you trying to fix the host side?

I see you also attempted to build lucid packages, I guess they were added after I went to bed last night. None of the packages built successfully, though.

Excerpts from Rolf Leggewie's message of Sun Jul 17 09:09:58 UTC 2011:
> Clynt, thank you for taking this up.
>
> My point is essentially the same as was raised by Lukas in comment 27.
> We need to be sure that a lucid guest will work no matter the host OS.
> I'm not opposed to a PPA workaround but IMHO some code should be added
> probably to dapper somewhere to make sure that if it's detected that
> dapper is running inside a chroot/vserver an update to lucid is not
> performed without that PPA workaround. Or else the user will run into
> the situation where the vserver is completely unreachable because ssh is
> down (happened to me, booh!).
>
> I will have to say that I'm a bit disappointed this wasn't taken into
> account seriously much earlier. The warnings were there early enough.
>
> Your PPA was building packages for natty right? I'm not sure what
> direction it is you're taking there, but I'm sure you will let me know.
> Are you trying to fix the host side?

I want to provide a chroot-capable upstart for all relevant versions
of Ubuntu, namely maverick and lucid (natty has chroot support). Hardy
isn't going to happen, as, while it has upstart, that version of upstart
is incompatible with the versions in 10.04 and later. That does mean that
you won't be able to use Hardy to host chroots for anything after it. For
that, you might want to look into the "upstart-dummy" program here:

http://blogs.balabit.com/2011/01/21/using-upstart-in-a-chroot/

We test what we can, and respond to what is raised. Sometimes the rapid
pace of development causes a situation like this, where a fundamental
change is made and users only have 1 cycle to test it out (IIRC, ssh was
moved to upstart in 9.10). This situation is at least going to be handled
in 12.04, so you should be safe in waiting for it, and then leap frogging
to it by upgrading to 10.04 first (sorry not sure if your ssh will work)
and then to 12.04.

So right now, I'd suggest running tests in 11.04, to make sure we actually
solved your problem and don't just think we did.

Anyway, the PPA is still empty because the packages haven't been able
to build yet, but I'll keep working on it.. watch the PPA and here
for updates.

My apologies for keeping on mixing up hardy and dapper. I appreciate your work, but if you look at comment 27 and my situation it may not be enough.

I am running a vserver on some hosting service out there (I actually have no idea what OS they use for the host). I am running hardy and think time has come to upgrade to lucid. I go through the motions which include a reboot and boom, without warning I am COMPLETELY shut off from my vserver. ssh does not come up because it relies on a working upstart. People lucky enough not to have upgraded yet from hardy are currently left without any upgrade path at all and without a warning of the potential upstart trouble.

Steve and Daniel warned about potential problems very early on, almost two years ago, but it seems that interest in investigating them thoroughly was bypassed in favor of pushing upstart out the door. That's the beef I have with it.

If thanks to your efforts we see a backport of a working upstart to lucid then I think some code changes should be pushed to hardy as well (probably into upstart-manager) that will refuse an update hardy->lucid when running inside a vserver/chroot when that PPA-solution you provide is not available ("Please enable PPA XY before continuing with the release update to lucid!")

When we talk about releases we need to carefully differentiate whether we talk about host or guest. I'm usually more concerned about the guest side of things. If you have control over the host your options are obviosly much better. Although thinking about it, it's not really pretty, either, I guess. Because if I understand the comments here correctly then none of the currently available LTS are candidates for hosting a bunch of vservers because of problems with upstart.

Excerpts from Rolf Leggewie's message of Sun Jul 17 14:47:43 UTC 2011:
> My apologies for keeping on mixing up hardy and dapper. I appreciate
> your work, but if you look at comment 27 and my situation it may not be
> enough.
>
> I am running a vserver on some hosting service out there (I actually
> have no idea what OS they use for the host). I am running hardy and
> think time has come to upgrade to lucid. I go through the motions which
> include a reboot and boom, without warning I am COMPLETELY shut off from
> my vserver. ssh does not come up because it relies on a working
> upstart. People lucky enough not to have upgraded yet from hardy are
> currently left without any upgrade path at all and without a warning of
> the potential upstart trouble.
>
> Steve and Daniel warned about potential problems very early on, almost
> two years ago, but it seems that interest in investigating them
> thoroughly was bypassed in favor of pushing upstart out the door.
> That's the beef I have with it.
>
> If thanks to your efforts we see a backport of a working upstart to
> lucid then I think some code changes should be pushed to hardy as well
> (probably into upstart-manager) that will refuse an update hardy->lucid
> when running inside a vserver/chroot when that PPA-solution you provide
> is not available ("Please enable PPA XY before continuing with the
> release update to lucid!")
>

AHH..

I think actually the upstart-dummy program might actually work for
you. I've talked to the author about patching and maintaining it.. but
never knew if there was a good enough reason to do so.

It wouldn't be perfect, but it might work for this situation.

Until then, I believe OpenVZ and LXC work fine for providing lucid
and later. Maybe look into a provider that uses containers like that,
rather than just chroot jails. You get the added benefit of network
isolation. For the provider there is really no additional cost other
than running an extra init for each container. Thats an extra 24k or so.

As far as stopping update-manager from upgrading.. thats not a bad idea
at all. Is that already proposed as a bug somewhere else?

On 07/17/2011 12:52 PM, Clint Byrum wrote:
> Excerpts from Rolf Leggewie's message of Sun Jul 17 14:47:43 UTC 2011:
>> My apologies for keeping on mixing up hardy and dapper. I appreciate
>> your work, but if you look at comment 27 and my situation it may not be
>> enough.
>>
>> I am running a vserver on some hosting service out there (I actually
>> have no idea what OS they use for the host). I am running hardy and
>> think time has come to upgrade to lucid. I go through the motions which
>> include a reboot and boom, without warning I am COMPLETELY shut off from
>> my vserver. ssh does not come up because it relies on a working
>> upstart. People lucky enough not to have upgraded yet from hardy are
>> currently left without any upgrade path at all and without a warning of
>> the potential upstart trouble.
>>
>> Steve and Daniel warned about potential problems very early on, almost
>> two years ago, but it seems that interest in investigating them
>> thoroughly was bypassed in favor of pushing upstart out the door.
>> That's the beef I have with it.
>>
>> If thanks to your efforts we see a backport of a working upstart to
>> lucid then I think some code changes should be pushed to hardy as well
>> (probably into upstart-manager) that will refuse an update hardy->lucid
>> when running inside a vserver/chroot when that PPA-solution you provide
>> is not available ("Please enable PPA XY before continuing with the
>> release update to lucid!")
>>
>
> AHH..
>
> I think actually the upstart-dummy program might actually work for
> you. I've talked to the author about patching and maintaining it.. but
> never knew if there was a good enough reason to do so.
>
> It wouldn't be perfect, but it might work for this situation.
>
> Until then, I believe OpenVZ and LXC work fine for providing lucid
> and later.

I run many Lucid containers based on OpenVZ and Virtuozzo with great
success. The only problem regarding SSH is discussed in LP: 634900. The
fix to this problem is basically to comment the "oom never" line in
/etc/init/ssh.conf.

I've opened bug 826544 against update-manager to propose that hardy->lucid updates should currently not happen until this ticket is properly resolved.

It's great to hear that OpenVZ, LXC, Virtuozzo and whatnot hosts are not affected (if that is indeed the case). I don't know what my hoster is running and I shouldn't have to. From some of the messages I exchanged with the owner I'd venture to guess that he is running linux-vserver.org. FWIW. I've had plans to replicate that setup locally, but never got around to it.

So, where are we? It seems that Clint's efforts to backport upstart have lately stalled. And unless I am mistaken that was about fixing the host side of things anyhow. Upstart-dummy was put forward as a possible workaround. I've read the page a few times but I'm not yet sure I fully understand the necessary steps. I'd actually like to see that script packaged so that things are easier to replicate on different machines. I'm not much of a wizzard when it comes to packaging python, so I'd appreciate some collaboration for further evaluation.

I've opened bug 826544 against update-manager to propose that hardy->lucid updates should currently not happen until this ticket is properly resolved. (I guess I should add that the ticket clearly states that hardy-lucid updates are fine as long as it does not involve an update in a chroot/vserver type of situation)

I've come across this problem while setting up a headless 12.04 server via debootstrap and chroot. To be able to upgrade or install services (like mdadm, which requires postfix) I had to use Ericks workaround and divert initctl.

It worked fine, however, there seems to be something missing in Ericks proposed command to delete the divert. Maybe this is the cause for his problems with gdm afterwards.

Here's what worked for me:

Set the divert:

# dpkg-divert --local --rename --add /sbin/initctl
# ln -s /bin/true /sbin/initctl

Remove the divert:

# rm /sbin/initctl
# dpkg-divert --local --rename --remove /sbin/initctl

Note the added '--rename' option that was missing in Ericks post. Without this, dpkg-divert says it removed the divert, however, /sbin/initctl.distrib is not renamed back to the original initctl command. Which in consequence breaks all services.

Took me a while to find out why my headless remote system got stuck at boot ;-)
Hope this is of help to someone, since the underlying problem seems to persist.

Cheerz, Martin.

I have installed Ubuntu Trusty (14.04) as a chroot inside Debian Wheezy (7) and this problem happens there too.
The workaround by Martin on the last comment (#58) worked well.
Thanks Martin and Ericks.

# Workaround and future

It may be of interest to observe that future releases of Ubuntu (after 14.04) will no longer use upstart but systemd.
Ref: [Mark Shuttleworth » Blog Archive » Losing graciously](http://www.markshuttleworth.com/archives/1316)

In the strict context of this bug report which aims at having traditional chroot work in an upstart context, this information should be considered a workaround at best.
In the broader context of making some software future-proof, systemd seems like the target to study. Systemd provides containers which can be thought of like "chroot done right".

Good news is: it appears one can test systemd with existing Ubuntu releases (from 13.04 onward, I haven't tested myself):

* [How can I replace upstart with systemd? - Ask Ubuntu](http://askubuntu.com/questions/420917/how-can-i-replace-upstart-with-systemd)
* [systemd - Ubuntu Wiki](https://wiki.ubuntu.com/systemd)

Thank you for your attention.

Just for the record, when I install Ubuntu Trusty (14.04) as a chroot inside Ubuntu Trusty (14.04) this problem does not happen. The problem here is with starting mysql inside the chroot: `service mysql start`. It tries to start the mysql of the main (host) system.

The easy workaround for this (which was not so easy for me to find) is to start mysql like this: `/etc/init.d/mysql start`
I hope this is useful to someone, since "systemd" may need some time to become working and stable. By the way, I tried to test it on 14.04 and it didn't work.

On 19 June 2014 14:15, Dashamir Hoxha <email address hidden> wrote:
> Just for the record, when I install Ubuntu Trusty (14.04) as a chroot
> inside Ubuntu Trusty (14.04) this problem does not happen. The problem
> here is with starting mysql inside the chroot: `service mysql start`. It
> tries to start the mysql of the main (host) system.
>
> The easy workaround for this (which was not so easy for me to find) is to start mysql like this: `/etc/init.d/mysql start`
> I hope this is useful to someone, since "systemd" may need some time to become working and stable. By the way, I tried to test it on 14.04 and it didn't work.
>

In 14.04, chroot support in upstart is turned off by default, to
enable please boot with "--chroot-sessions" kernel cmdline option
(which is passed to init) see $ man 8 init , for more details. After
that "service mysql start" will work inside the chroot.

However, instead of using chroots I strongly recommend for you to use
lxc container, which is almost as lightweight as chroot, but has extra
protections against affecting the host system and it has fully working
upstart inside.

--
Regards,

Dimitri.