GCC testsuite results

It seems you got too many unexpected failures. Here is result for latest 4.7 Q2 release:

  === gcc Summary ===
# of expected passes 80579
# of unexpected failures 25
# of expected failures 247
# of unsupported tests 1456
  === g++ Summary ===
# of expected passes 45212
# of unexpected failures 1
# of expected failures 279
# of unsupported tests 781

Please check your gcc.log file, you may find something there.

After eliminating a bunch of failures because of a newlib configuration issue, I'm now getting the below results for gcc. The total number of tests is much lower than what you reported. We are running all the tests with cflags set to -mthumb -mcpu=cortex-m3 - are you guys doing the same?

Is there some place where I can look at the gcc.sum and gcc.log files of your test run?

        === gcc Summary ===

# of expected passes 67381
# of unexpected failures 27
# of expected failures 183
# of unresolved testcases 1
# of unsupported tests 1250

        === g++ Summary ===

# of expected passes 43392
# of unexpected failures 1
# of unexpected successes 2
# of expected failures 279
# of unsupported tests 774

I am sorry that the .sum and .log files are not approved to share publicly yet. But if you want to figure out the differences, you can email me how you run the test along with your sum and log files. I am glad to help.

I'm nearly there now, after enabling LTO (which was previously disabled when configuring). The results now look like

  === gcc Summary ===

# of expected passes 80514
# of unexpected failures 28
# of expected failures 247
# of unresolved testcases 1
# of unsupported tests 1417

 === g++ Summary ===

# of expected passes 44953
# of unexpected failures 1
# of unexpected successes 2
# of expected failures 281
# of unsupported tests 782

I've posted the gcc and g++ sum and log files online (see https://docs.google.com/file/d/0B1Ce3LcbUJuIOFphWk94c3huUE0). The dejagnu board configuration for qemu looks like this

      global SIM
      set target_list { "arm-qemu" }
      set board_info(arm-qemu,cflags) "-mthumb -mcpu=cortex-m3"
      set board_info(arm-qemu,ldflags) "--specs=rdimon.specs -lc -lrdimon -u _printf_float"
      set board_info(arm-qemu,sim,options) "-cpu cortex-m3 -nographic -serial null -monitor null -semihosting -kernel "
      set SIM "qemu-system-arm"

Some testsuite(s) are obviously getting skipped, but I couldn't figure out which. Can you take a look and tell which ones we are missing, please?

With your log and sum files, I found some differences from ours:

1). When configure Binutils, option --enable-plugins should be added. Otherwise there will be message "xgcc: error: -fuse-linker-plugin is not supported in this configuration" in .sum file. When torture cases with different options, this option makes differences. I guess the missing is from here.

2). In our regression test, we run following cases which can't be found in your .sum/.log files:
UNSUPPORTED: gcc.dg/simulate-thread/atomic-load-int.c -O0 -g
UNSUPPORTED: gcc.dg/simulate-thread/atomic-load-int.c -O2 -g
UNSUPPORTED: gcc.dg/simulate-thread/atomic-load-int.c -O3 -g
UNSUPPORTED: gcc.dg/simulate-thread/atomic-load-int128.c -O0 -g
UNSUPPORTED: gcc.dg/simulate-thread/atomic-load-int128.c -O2 -g
UNSUPPORTED: gcc.dg/simulate-thread/atomic-load-int128.c -O3 -g
UNSUPPORTED: gcc.dg/simulate-thread/atomic-load-longlong.c -O0 -g
UNSUPPORTED: gcc.dg/simulate-thread/atomic-load-longlong.c -O2 -g
UNSUPPORTED: gcc.dg/simulate-thread/atomic-load-longlong.c -O3 -g
UNSUPPORTED: gcc.dg/simulate-thread/atomic-load-short.c -O0 -g
UNSUPPORTED: gcc.dg/simulate-thread/atomic-load-short.c -O2 -g
UNSUPPORTED: gcc.dg/simulate-thread/atomic-load-short.c -O3 -g
UNSUPPORTED: gcc.dg/simulate-thread/atomic-other-int.c -O0 -g
UNSUPPORTED: gcc.dg/simulate-thread/atomic-other-int.c -O2 -g
UNSUPPORTED: gcc.dg/simulate-thread/atomic-other-int.c -O3 -g
UNSUPPORTED: gcc.dg/simulate-thread/atomic-other-int128.c -O0 -g
UNSUPPORTED: gcc.dg/simulate-thread/atomic-other-int128.c -O2 -g
UNSUPPORTED: gcc.dg/simulate-thread/atomic-other-int128.c -O3 -g
UNSUPPORTED: gcc.dg/simulate-thread/atomic-other-longlong.c -O0 -g
UNSUPPORTED: gcc.dg/simulate-thread/atomic-other-longlong.c -O2 -g
UNSUPPORTED: gcc.dg/simulate-thread/atomic-other-longlong.c -O3 -g
UNSUPPORTED: gcc.dg/simulate-thread/atomic-other-short.c -O0 -g
UNSUPPORTED: gcc.dg/simulate-thread/atomic-other-short.c -O2 -g
UNSUPPORTED: gcc.dg/simulate-thread/atomic-other-short.c -O3 -g

3). We run regression test with command "make check -k -i RUNTESTFLAGS="--target_board=arm-none-eabi-qemu/-mthumb/-mcpu=cortex-m3"" rather than setting "-mthumb -mcpu=cortex-m3" in board config file. I guess it makes difference here too.

Please try with my comments and let me know the results. Thanks.

Thanks a lot for your patience.

After making the changes you suggested, gcc regression results exactly match yours; g++ results are still off though.

=== g++ Summary ===

# of expected passes 45204
# of unexpected failures 1
# of unexpected successes 2
# of expected failures 281
# of unsupported tests 783

I've attached the g++.{log, sum} files here (https://docs.google.com/file/d/0B1Ce3LcbUJuING1TOTRpYkdVcEE/edit?usp=sharing). Can you take a look at it, please?

Hi,

I have built 4.9.3 toolchain using your source and build scripts. Following is test result summary (test config: qemu with cortex-m3).
  === gcc Summary ===

# of expected passes 91700
# of unexpected failures 51
# of expected failures 241
# of unsupported tests 1789

  === g++ Summary ===

# of expected passes 75185
# of unexpected failures 8
# of expected failures 438
# of unsupported tests 3312

Could you please check if the above results aligns with yours?

Thanks Thomas Preud'homme, that solved my question.