Failed tests are marked as "error" instead of "failed" in report

Asked by zhenya on 2018-08-31

Hello!
I'm using unittest and htmltestrunner in my tests.
And when Sikuli doesn't find image, report puts "error" status against corresponding test instead of "failed".
Should it work so?
If yes, can you please provide me an example of how should I modify code to turn fails to find image into test fails?

I've thought it might be something like that, but it didn't work for me:

class TestsBasic (unittest.TestCase):
 def test_example(self):
  testresult = False
  expectedresult = True
  click("startProgram.jpg")
  wait("programStarted.jpg")
  if findAll(Pattern("programStarted.jpg").similar(0.99)):
   testresult = True
  self.assertEqual(expectedresult, testresult)

Question information

Language:
English Edit question
Status:
Solved
For:
Sikuli Edit question
Assignee:
No assignee Edit question
Solved by:
RaiMan
Solved:
2018-09-02
Last query:
2018-09-02
Last reply:
2018-09-02
RaiMan (raimund-hocke) said : #1

use exists() instead of wait()

zhenya (zhenya.kozhevnikov) said : #2

Just did it. It is very strange, but after I changed wait() to exists(), test which should fail somehow passed.
Although web browser is in fullscreen mode and I'm sure that image within "exists()" is absent on the screen.
I've tried 2 options:
1. exists("image.jpg")
2. exists(Pattern("image.jpg").similar(99))

Unfortunately, result was the same. What could cause it? Could it be some mistake in the code?
Because, at the same time, "wait()" instead of "exists()" clearly see that image is absent on the screen, just with wrong resolution ("error" instead of "failed")

zhenya (zhenya.kozhevnikov) said : #3

To be more accurate, I'm providing code of both modules:

1. testScenarios.py:

import org.sikuli.script.SikulixForJython
from sikuli import *
import unittest

class TestsBasic (unittest.TestCase):
    def setUp(self):
        click("browser_icon.jpg")
        wait(2)
        type("f", Key.CTRL + Key.CMD)
        wait(5)
        click("url.jpg")
        type("my-site.com" + Key.ENTER)
        wait(5)
    def test_example(self):
        wait("password_field_safari.jpg")
    def test_negative_example(self):
        exists(Pattern("result_popup.jpg").similar(99))
    def test_negative_example2(self):
        exists("result_popup.jpg")
    def tearDown(self):
        type("q", Key.CMD)
        wait(2)
        click("quit.jpg")

2. main.py:

import unittest
import HTMLTestRunner
from testScenarios import *

suite = unittest.TestLoader().loadTestsFromTestCase(TestsBasic)
outfile = open("/Users/zhenya/Desktop/Report.html", "w")
runner = HTMLTestRunner.HTMLTestRunner(stream=outfile, title='TestsBasic report', description='description')
runner.run(suite)

test_negative_example and test_negative_example2 should return "Failed" result, because "result_popup" image is absent on the screen during their run - but they return "Passed" instead. when I used wait(), they were returning "Error" result.

Best RaiMan (raimund-hocke) said : #4

basic unittesting knowledge:

a test has to return false or raise assert exception to signal failed.

if a test does not return anything, it is taken as passed.

hence:
assert exists()

zhenya (zhenya.kozhevnikov) said : #5

Thanks RaiMan, that solved my question.