Overnight, I ran this test case on both an Ubuntu 16.04 ppc64le system and a RHEL 7.1 ppc64le system.
The test ran 219 times on Ubuntu, with 15 cores, for a failure rate of around 5%. Most of the time corruption was detected in the Canary ctor (before doing other work), but a few times in the dtor:
$ grep "10000[12]" resmoke.log
[js_test:fsm_all_sharded_replication] 2016-11-09T21:06:42.140+0000 s40019| 2016-11-09T21:06:42.140+0000 I - [conn59] Fatal Assertion 100002 at src/mongo/bson/util/bson_extract.cpp 50
[js_test:fsm_all_sharded_replication] 2016-11-09T23:11:56.413+0000 s40019| 2016-11-09T23:11:56.412+0000 I - [conn1] Fatal Assertion 100001 at src/mongo/bson/util/bson_extract.cpp 46
[js_test:fsm_all_sharded_replication] 2016-11-10T03:49:43.577+0000 s40019| 2016-11-10T03:49:43.576+0000 I - [conn47] Fatal Assertion 100001 at src/mongo/bson/util/bson_extract.cpp 46
[js_test:fsm_all_sharded_replication] 2016-11-10T04:54:20.748+0000 s40019| 2016-11-10T04:54:20.745+0000 I - [conn28] Fatal Assertion 100001 at src/mongo/bson/util/bson_extract.cpp 46
[js_test:fsm_all_sharded_replication] 2016-11-10T04:56:43.929+0000 s40020| 2016-11-10T04:56:43.929+0000 I - [conn30] Fatal Assertion 100001 at src/mongo/bson/util/bson_extract.cpp 46
[js_test:fsm_all_sharded_replication] 2016-11-10T07:26:14.013+0000 s40019| 2016-11-10T07:26:14.013+0000 I - [conn3] Fatal Assertion 100001 at src/mongo/bson/util/bson_extract.cpp 46
[js_test:fsm_all_sharded_replication] 2016-11-10T07:52:45.732+0000 s40019| 2016-11-10T07:52:45.698+0000 I - [conn1] Fatal Assertion 100002 at src/mongo/bson/util/bson_extract.cpp 50
[js_test:fsm_all_sharded_replication] 2016-11-10T07:54:45.606+0000 s40020| 2016-11-10T07:54:45.606+0000 I - [conn30] Fatal Assertion 100001 at src/mongo/bson/util/bson_extract.cpp 46
[js_test:fsm_all_sharded_replication] 2016-11-10T08:28:04.278+0000 s40020| 2016-11-10T08:28:04.277+0000 I - [conn13] Fatal Assertion 100001 at src/mongo/bson/util/bson_extract.cpp 46
[js_test:fsm_all_sharded_replication] 2016-11-10T09:28:24.221+0000 s40020| 2016-11-10T09:28:24.221+0000 I - [conn32] Fatal Assertion 100001 at src/mongo/bson/util/bson_extract.cpp 46
[js_test:fsm_all_sharded_replication] 2016-11-10T09:41:14.880+0000 s40019| 2016-11-10T09:41:14.800+0000 I - [conn48] Fatal Assertion 100001 at src/mongo/bson/util/bson_extract.cpp 46
[js_test:fsm_all_sharded_replication] 2016-11-10T10:35:58.143+0000 s40019| 2016-11-10T10:35:58.118+0000 I - [conn2] Fatal Assertion 100001 at src/mongo/bson/util/bson_extract.cpp 46
[js_test:fsm_all_sharded_replication] 2016-11-10T11:06:01.027+0000 s40019| 2016-11-10T11:06:01.027+0000 I - [conn1] Fatal Assertion 100001 at src/mongo/bson/util/bson_extract.cpp 46
[js_test:fsm_all_sharded_replication] 2016-11-10T12:37:43.100+0000 s40019| 2016-11-10T12:37:43.100+0000 I - [conn1] Fatal Assertion 100001 at src/mongo/bson/util/bson_extract.cpp 46
[js_test:fsm_all_sharded_replication] 2016-11-10T13:31:31.912+0000 s40020| 2016-11-10T13:31:31.912+0000 I - [conn1] Fatal Assertion 100001 at src/mongo/bson/util/bson_extract.cpp 46
The test ran 227 times on RHEL without issue. I will try valgrind without --track-origins=yes today on the Ubuntu 16.04 system.
Overnight, I ran this test case on both an Ubuntu 16.04 ppc64le system and a RHEL 7.1 ppc64le system.
The test ran 219 times on Ubuntu, with 15 cores, for a failure rate of around 5%. Most of the time corruption was detected in the Canary ctor (before doing other work), but a few times in the dtor:
$ grep "10000[12]" resmoke.log fsm_all_ sharded_ replication] 2016-11- 09T21:06: 42.140+ 0000 s40019| 2016-11- 09T21:06: 42.140+ 0000 I - [conn59] Fatal Assertion 100002 at src/mongo/ bson/util/ bson_extract. cpp 50 fsm_all_ sharded_ replication] 2016-11- 09T23:11: 56.413+ 0000 s40019| 2016-11- 09T23:11: 56.412+ 0000 I - [conn1] Fatal Assertion 100001 at src/mongo/ bson/util/ bson_extract. cpp 46 fsm_all_ sharded_ replication] 2016-11- 10T03:49: 43.577+ 0000 s40019| 2016-11- 10T03:49: 43.576+ 0000 I - [conn47] Fatal Assertion 100001 at src/mongo/ bson/util/ bson_extract. cpp 46 fsm_all_ sharded_ replication] 2016-11- 10T04:54: 20.748+ 0000 s40019| 2016-11- 10T04:54: 20.745+ 0000 I - [conn28] Fatal Assertion 100001 at src/mongo/ bson/util/ bson_extract. cpp 46 fsm_all_ sharded_ replication] 2016-11- 10T04:56: 43.929+ 0000 s40020| 2016-11- 10T04:56: 43.929+ 0000 I - [conn30] Fatal Assertion 100001 at src/mongo/ bson/util/ bson_extract. cpp 46 fsm_all_ sharded_ replication] 2016-11- 10T07:26: 14.013+ 0000 s40019| 2016-11- 10T07:26: 14.013+ 0000 I - [conn3] Fatal Assertion 100001 at src/mongo/ bson/util/ bson_extract. cpp 46 fsm_all_ sharded_ replication] 2016-11- 10T07:52: 45.732+ 0000 s40019| 2016-11- 10T07:52: 45.698+ 0000 I - [conn1] Fatal Assertion 100002 at src/mongo/ bson/util/ bson_extract. cpp 50 fsm_all_ sharded_ replication] 2016-11- 10T07:54: 45.606+ 0000 s40020| 2016-11- 10T07:54: 45.606+ 0000 I - [conn30] Fatal Assertion 100001 at src/mongo/ bson/util/ bson_extract. cpp 46 fsm_all_ sharded_ replication] 2016-11- 10T08:28: 04.278+ 0000 s40020| 2016-11- 10T08:28: 04.277+ 0000 I - [conn13] Fatal Assertion 100001 at src/mongo/ bson/util/ bson_extract. cpp 46 fsm_all_ sharded_ replication] 2016-11- 10T09:28: 24.221+ 0000 s40020| 2016-11- 10T09:28: 24.221+ 0000 I - [conn32] Fatal Assertion 100001 at src/mongo/ bson/util/ bson_extract. cpp 46 fsm_all_ sharded_ replication] 2016-11- 10T09:41: 14.880+ 0000 s40019| 2016-11- 10T09:41: 14.800+ 0000 I - [conn48] Fatal Assertion 100001 at src/mongo/ bson/util/ bson_extract. cpp 46 fsm_all_ sharded_ replication] 2016-11- 10T10:35: 58.143+ 0000 s40019| 2016-11- 10T10:35: 58.118+ 0000 I - [conn2] Fatal Assertion 100001 at src/mongo/ bson/util/ bson_extract. cpp 46 fsm_all_ sharded_ replication] 2016-11- 10T11:06: 01.027+ 0000 s40019| 2016-11- 10T11:06: 01.027+ 0000 I - [conn1] Fatal Assertion 100001 at src/mongo/ bson/util/ bson_extract. cpp 46 fsm_all_ sharded_ replication] 2016-11- 10T12:37: 43.100+ 0000 s40019| 2016-11- 10T12:37: 43.100+ 0000 I - [conn1] Fatal Assertion 100001 at src/mongo/ bson/util/ bson_extract. cpp 46 fsm_all_ sharded_ replication] 2016-11- 10T13:31: 31.912+ 0000 s40020| 2016-11- 10T13:31: 31.912+ 0000 I - [conn1] Fatal Assertion 100001 at src/mongo/ bson/util/ bson_extract. cpp 46
[js_test:
[js_test:
[js_test:
[js_test:
[js_test:
[js_test:
[js_test:
[js_test:
[js_test:
[js_test:
[js_test:
[js_test:
[js_test:
[js_test:
[js_test:
The test ran 227 times on RHEL without issue. I will try valgrind without --track-origins=yes today on the Ubuntu 16.04 system.