mongorestore fails with createIndexes command in oplog
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| mongodb (Ubuntu) |
Fix Released
|
Critical
|
Unassigned | ||
| Bionic |
Fix Released
|
Critical
|
Andreas Hasenack | ||
| Cosmic |
Fix Released
|
Critical
|
Andreas Hasenack | ||
Bug Description
[Impact]
When an operator of Juju attempts to restore a backup, it fails with a mongorestore related error[0]. Under the hood, Juju uses the supplied mongo related command line interfaces (mongodump, mongorestore etc), using these in conjunction with the oplog related flags for the former CLIs, the mongorestore fails with the following error:
> Failed: restore error: error applying oplog: applyOps: applyOps had one or more errors applying ops
The only work around for Juju at the moment is to update mongod, so that it carries the following commit[1]. MongoDB have an identical issue, under a different name, which fixes this[2], but there is unfortunately no work around without upgrading mongod so it caries the former commit.
The original investigation around Juju failing to restore a previous dump can be seen here[3]
I've verified with the 3.6.11 and the same restore, worked fine.
[Test Case]
The following test case doesn't require Juju, and can be replicated quite easily inside a lxc container. It may require 2 or 3 goes, depending on timings of the creation of the index and the mongodump, but I've been getting this consistently.
Please do mind that timing is important, so putting the commands in a script will help a lot to reliably trigger the issue.
You can also find instructions in one pastebin[4]
// Launch a container
lxc launch ubuntu:18.04 mongo
lxc exec mongo bash
// Install a version of mongo that Juju is using.
sudo apt update
sudo apt install -y mongodb
// Edit /etc/mongodb.conf and add the following line:
replSet = "rs0"
// Restart the service
sudo systemctl restart mongodb.service
// not required, but helps diagnose bsondump
sudo snap install jq
// launch the mongo CLI, running each command
mongo --eval "rs.initiate()"
mongo --eval "rs.status()"
mongo --eval "db.foo.
mongo --eval "db.foo.
// Get the dump
mongodump --oplog
// check if it's going to fail
// if the output contains a createIndex, but doesn't contain an oid then it'll fail.
bsondump dump/oplog.bson | jq
// Drop the foo collection, so we don't get any duplication errors
mongo --eval "db.foo.drop()"
// Now restore the dump
mongorestore --oplogReplay dump/
// It should fail with
2019-03-
2019-03-
2019-03-
2019-03-
2019-03-
2019-03-
2019-03-
After updating the packages, re-run the drop and try to import again, this time it should work:
# mongo --eval "db.foo.drop()"
MongoDB shell version v3.6.3
connecting to: mongodb:
MongoDB server version: 3.6.3
true
r
# mongorestore --oplogReplay dump/
2019-03-
2019-03-
2019-03-
2019-03-
2019-03-
2019-03-
2019-03-
[Other Info]
0. https:/
1. https:/
2. https:/
3. https:/
4. https:/
Related branches
- Christian Ehrhardt (community): Approve
- Robie Basak: Pending requested
- Canonical Server: Pending requested
-
Diff: 260 lines (+220/-0)6 files modifieddebian/changelog (+14/-0)
debian/patches/168c8d65556ab18b05e33b5f4006d1184c30f8f7.patch (+29/-0)
debian/patches/SERVER-36951-applyOps-createIndexes-without-uuid.patch (+116/-0)
debian/patches/fix-boost-compile.patch (+26/-0)
debian/patches/gcc8-parentheses.patch (+31/-0)
debian/patches/series (+4/-0)
- Christian Ehrhardt (community): Approve
- Canonical Server: Pending requested
-
Diff: 147 lines (+125/-0)3 files modifieddebian/changelog (+8/-0)
debian/patches/SERVER-36951-applyOps-createIndexes-without-uuid.patch (+116/-0)
debian/patches/series (+1/-0)
| Changed in mongodb (Ubuntu): | |
| assignee: | nobody → Andreas Hasenack (ahasenack) |
| status: | New → In Progress |
| importance: | Undecided → Critical |
| Andreas Hasenack (ahasenack) wrote | #1 |
| description: | updated |
| description: | updated |
| Christian Ehrhardt (paelzer) wrote | #2 |
| description: | updated |
| Changed in mongodb (Ubuntu): | |
| status: | In Progress → Fix Released |
| Changed in mongodb (Ubuntu Bionic): | |
| status: | New → In Progress |
| Changed in mongodb (Ubuntu Cosmic): | |
| status: | New → In Progress |
| Changed in mongodb (Ubuntu Bionic): | |
| assignee: | nobody → Andreas Hasenack (ahasenack) |
| Changed in mongodb (Ubuntu Cosmic): | |
| assignee: | nobody → Andreas Hasenack (ahasenack) |
| Changed in mongodb (Ubuntu): | |
| assignee: | Andreas Hasenack (ahasenack) → nobody |
| Changed in mongodb (Ubuntu Bionic): | |
| importance: | Undecided → Critical |
| Changed in mongodb (Ubuntu Cosmic): | |
| importance: | Undecided → Critical |
| Andreas Hasenack (ahasenack) wrote | #3 |
| description: | updated |
| Timo Aaltonen (tjaalton) wrote Please test proposed package | #4 |
| Changed in mongodb (Ubuntu Cosmic): | |
| status: | In Progress → Fix Committed |
| tags: | added: verification-needed verification-needed-cosmic |
| Changed in mongodb (Ubuntu Bionic): | |
| status: | In Progress → Fix Committed |
| tags: | added: verification-needed-bionic |
| Timo Aaltonen (tjaalton) wrote | #5 |
| Andreas Hasenack (ahasenack) wrote | #6 |
| tags: |
added: verification-done-bionic removed: verification-needed-bionic |
| Andreas Hasenack (ahasenack) wrote | #7 |
| Andreas Hasenack (ahasenack) wrote | #8 |
| tags: |
added: verification-done-cosmic removed: verification-needed-cosmic |
| Andreas Hasenack (ahasenack) wrote | #9 |
| Andreas Hasenack (ahasenack) wrote | #10 |
| Andreas Hasenack (ahasenack) wrote | #11 |
| Launchpad Janitor (janitor) wrote | #12 |
| Changed in mongodb (Ubuntu Cosmic): | |
| status: | Fix Committed → Fix Released |
| Brian Murray (brian-murray) wrote Update Released | #13 |
| Launchpad Janitor (janitor) wrote | #14 |
| Changed in mongodb (Ubuntu Bionic): | |
| status: | Fix Committed → Fix Released |
> https:/
I was asked, is this commit OK regarding the upsream licensing change? Specifically: "Did they do a release following the commit before the license change?"