[HowTo] Nice unit test runner with html output

Asked by TestMechanic

Just want to share with you nice runner that output an html result file.

http://tungwaiyip.info/software/HTMLTestRunner.html

Download and put the module in same folder as your .py file

Your test file now should look like
======================
import unittest
import HTMLTestRunner

class BDTests(unittest.TestCase):

    def test_1
    def test_2

suite = unittest.TestLoader().loadTestsFromTestCase(BDTests)
outfile = open("C:\Report.html", "w")
runner = HTMLTestRunner.HTMLTestRunner(stream=outfile, title='Test Report', description='This is demo' )
runner.run(suite)
==========================================

You need to run the tests as required when non default(UI) runner is used -> Sikuli-IDE.bat -r smoketest.sikuli

Hope this helps

Question information

Language:
English Edit question
Status:
Solved
For:
SikuliX Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
RaiMan (raimund-hocke) said :
#1

Thanks for contribution.

Set to solved, to not get it expired

Revision history for this message
Jose Pablo Jimenez Velez (jpjimenezv) said :
#2

Hey I love the HTML report thanks for sharing!

I do have a question and I hope some one can help help me.

So if an assert fails it will raise an exception and stop the script, I know I can capture that exception, to prevent it from stoping the execution, and log my own message BUT I do want to show on the log as failed.

Any idea how to do that? log the fail without stopping the execution

Thanks in advance!

Revision history for this message
RaiMan (raimund-hocke) said :
#3

@ Jose

When using the unittest framework, it does not make sense, to catch the assert exceptions, because they are used by the framework, to create the final report.

what you can do, is the following:

now:

assert exists(some-image)

modified:

if not exists(some-image):
    # make your log message
    assert False
else:
    assert True

This means in general, you have to check the assert condition your self and then use assert False/True according to the result. So before issuing the assert False, you can write your log message.

Another possibility is to modify the HTMLTestRunner.py in the TestResult class in the method addFailure(), which is called, when a test fails (this is the place, where you could e.g. make a screenshot of the failure situation). Come back if you want this solution.

Revision history for this message
sreetama (harshaohri) said :
#4

what is def test_1, def test_2??

Revision history for this message
TestMechanic (ndinev) said :
#6

Those are definitions of your tests in form of methods. The actual test code goes there

Revision history for this message
Chris (shortcuts-shadow) said :
#7

"Download and put the module in same folder as your .py file" where is this location, i tried multiple locations with no success...

Revision history for this message
RaiMan (raimund-hocke) said :
#8

@Chris
Here you have the HTMLTestRunner with the added feature of saving screenshots when failing.
https://dl.dropboxusercontent.com/u/42895525/HTMLTestRunner.zip

Put the HTMLTestRunner.py in the same folder as your test.sikuli and the import should work,

Revision history for this message
Chris (shortcuts-shadow) said :
#9

Great thanks :D that worked but now the script just says EEE time elapsed 0:00:00... I really love sikuli it will be awesome for non-coder-testers, sry for all the bother but is there anything i'm just missing?Here is a print out of my script, hope you don't mind me posting this large post...

import unittest
import HTMLTestRunner
#first we need to define any applications we want to use and their location
app = App("C:\\Program Files (x86)\\Microsoft Office\\Office14\\WINWORD.exe")

def setUp(self):
        #open app
    self.app.open()
          #wait until initialized
    wait(7)
    #tear down happens after tests compelte and should close anything that was opened
def tearDown(self):
        #close app
    self.app.close()
        #slight delay for safety(probably nto needed)
    wait(2)
#set up method will happen before tests

class BDTests(unittest.TestCase):
#each test needs to start with test

    def test_cInitialization(self):
    #using regions improves performance and reduces the chance of errors
    # because the same UI component exists elsewhere
        with self.app.window():
            wait(2)
            wait(Pattern("SelectEditin.png").targetOffset(21,1),5)
            click(Region(1647,139,24,21))
            wait("Ready-1.png")
            if not exists("Ready-1.png"):
                assert False
            else:
                assert True

    def test_bSavingFavorite(self):
        with self.app.window():
        #wait(5)
            click(Region(1648,140,24,22))
            wait(6)
            click(Pattern("1365194458088.png").targetOffset(-1,0))
            wait(2)
            doubleClick("SaveFavourit.png")
            click("Name-1.png")
            type("blarg")
            click("1365453234338.png")
            wait(2)
            click("ClassifyifTr.png")
            wait(2)
            click(Pattern("TEST2.png").targetOffset(-16,1))

            if not exists("blarg.png"):
                assert False
            else:
                assert True

    def test_aDeletingafavorite(self):
        with self.app.window():
            wait(4)
            click(Pattern("FindiscRepla.png").targetOffset(42,47))
            wait(5)
            click(Pattern("1.png").targetOffset(-1,-1))
            wait(2)
            click(Pattern("1365519420122.png").targetOffset(-13,0))
            wait(4)
            rightClick("blarg.png")
            click("Delete.png")
            wait(5)
            click("1.png")
            wait(2)

            if not exists("TEST2.png"):
                assert False
            else:
                assert True

suite = unittest.TestLoader().loadTestsFromTestCase(BDTests)
outfile = open("C:\\Users\\testuser454\\Documents\\results.html","w")
runner = HTMLTestRunner.HTMLTestRunner(stream=outfile,title='Test Report demo',description='Test description')
runner.run(suite)

Revision history for this message
Chris (shortcuts-shadow) said :
#10

In the report, when i click the error it says: "AttributeError: 'BDTest' object has no attribute 'app' "

Revision history for this message
Chris (shortcuts-shadow) said :
#11

Nvm i figured it out my app definition and my setup and teArdown go inside the class :P obviously, thanks for the help RaiMan! :D
Now i just need to figure out why word is not loading the plugins when it opens lol. Life is study!!!

Revision history for this message
Chris (shortcuts-shadow) said :
#12

Lol just figured it out, running sikuli as admin makes word opened from sikuli run as admin as well and my plugins were installed per user. Weird but ok! :D

Revision history for this message
RiSol (rinasolomon) said :
#13

@RainMan

"Here you have the HTMLTestRunner with the added feature of saving screenshots when failing."

Does the solution work for errors as well?

I run htmltest sikuli script that was in the zip file but I see no screenshot for errors.

Revision history for this message
RaiMan (raimund-hocke) said :
#14

@ RiSol
no, currently not.

But it is easy to add:
in HTMLTestRunner.py thes are the 2 methods:

    def addError(self, test, err):
        self.error_count += 1
        TestResult.addError(self, test, err)
        _, _exc_str = self.errors[-1]
        output = self.complete_output()
        self.result.append((2, test, output, _exc_str, ''))
        if self.verbosity > 1:
            sys.stderr.write('E ')
            sys.stderr.write(str(test))
            sys.stderr.write('\n')
        else:
            sys.stderr.write('E')

    def addFailure(self, test, err):
        self.failure_count += 1
        TestResult.addFailure(self, test, err)
        _, _exc_str = self.failures[-1]
        output = self.complete_output()
        self.result.append((1, test, output, _exc_str, self.generateTestScreenshot(test))) # modified by RaiMan
        if self.verbosity > 1:
            sys.stderr.write('F ')
            sys.stderr.write(str(test))
            sys.stderr.write('\n')
        else:
            sys.stderr.write('F')

just replace the line self.result.append.... in addError with the line ... # modified by RaiMan from addFailure

Revision history for this message
RiSol (rinasolomon) said :
#15

@RainMan

"just replace the line self.result.append.... in addError with the line ... # modified by RaiMan from addFailure"

I replaced with the line self.result.append((2, test, output, _exc_str, self.generateTestScreenshot(test))) and it's working GREAT.

Thanks!

Revision history for this message
sathya (bksathyamoorthy) said :
#16

I'm new to sikuli and this is my first question here.
I just want to add a screenshot in my html report..Below is my requirement.
Thanks in advance :)

    def test():
        print ("Devices update and current data verification check")

        if exists ("device_update_cdata_click.JPG"):
            click(Pattern("device_update_cdata_click.JPG").targetOffset(-58,-1))
            wait(2)
            print( "Devices are getting updated successfully")
     <<Here i need to attach the screenshot of the popup window>>

I am using htmlrunner.py from "http://tungwaiyip.info/software/HTMLTestRunner.html"

**i cant access this modified version link --
https://dl.dropboxusercontent.com/u/42895525/HTMLTestRunner.zip

Can anyone help me with this?

Revision history for this message
Hardik (hardikchotaliya26) said :
#17

@RainMan

Where to add this code for screenshot of failure..??? I mean after or before of script ??

def addError(self, test, err):
        self.error_count += 1
        TestResult.addError(self, test, err)
        _, _exc_str = self.errors[-1]
        output = self.complete_output()
        self.result.append((2, test, output, _exc_str, ''))
        if self.verbosity > 1:
            sys.stderr.write('E ')
            sys.stderr.write(str(test))
            sys.stderr.write('\n')
        else:
            sys.stderr.write('E')

    def addFailure(self, test, err):
        self.failure_count += 1
        TestResult.addFailure(self, test, err)
        _, _exc_str = self.failures[-1]
        output = self.complete_output()
        self.result.append((1, test, output, _exc_str, self.generateTestScreenshot(test))) # modified by RaiMan
        if self.verbosity > 1:
            sys.stderr.write('F ')
            sys.stderr.write(str(test))
            sys.stderr.write('\n')
        else:
            sys.stderr.write('F')

Revision history for this message
kavitha (tkavi03) said :
#18

Hi Raiman,

    I have run the following scripts in sikuli but I have found all the parameters as '0' in the generated report.

from sikuli import *
import unittest
import HTMLTestRunner
class TestReport(unittest.TestCase):
 click("1486657432621.png")
 wait(2)
 type("calculator")
 wait(1)
 type(Key.ENTER)
 wait(1)
 click("1487677101662.png")
 wait(0.5)
 click("1487677117547.png")
 wait(0.5)
 click("1487677137680.png")
 wait(0.5)
 click("1487677146827.png")
 wait(2)
suite = unittest.TestLoader().loadTestsFromTestCase(TestReport)
outfile = open("C:\\Htmltestreport\\report.html", "w")
runner = HTMLTestRunner.HTMLTestRunner(stream=outfile, title=' Report Title', description='desc..' )
runner.run(suite)
outfile.close()

   Kindly help me finding the solution.

Thanks,
Kavitha

Revision history for this message
RaiMan (raimund-hocke) said :
#19

@Kavitha
go back and read carefully all information about unittesting:
you have not defined any testcase!

Revision history for this message
kavitha (tkavi03) said :
#20

Time Elapsed: 0:00:00 is coming

Revision history for this message
RaiMan (raimund-hocke) said :
#21

again:
go back and read carefully how to set up and run a unittest class (Python knowledge!)

you do not have any testcase defined (as alredy mentioned in comment #19 !!!!!!!!!!!

Revision history for this message
RaiMan (raimund-hocke) said :
#22

again:
go back and read carefully how to set up and run a unittest class (Python knowledge!)

you do not have any testcase defined (as alredy mentioned in comment #19 !!!!!!!!!!!

Revision history for this message
kavitha (tkavi03) said :
#23

Yes. Now its working. Thanks RaiMan.

Revision history for this message
Sreelekshmi (sree2604) said :
#25

Can you please share the modified HTMLTestRunner.py to include screenshot.

Revision history for this message
RaiMan (raimund-hocke) said :
#26

--- share the modified HTMLTestRunner.py
... it is bundled with the sikulix... jar files and hence available out of the box

Revision history for this message
Christoph (chris961234) said :
#29

Hi

I'm completely new to sikuli and I'm trying to use the HTMLTestRunner. My problem is that I can't import neither the unittest.py nor the HTMLTestRunner.py. When I add

Import HTMLTestRunner

to my file I get the following error:

[error] script [ sikulix ] stopped with error in line 334 at column 52
[error] SyntaxError ( "mismatched input ',' expecting NAME", )

I have searched but found nothing. Is there something I am missing? Something I don't get?

Hopefully someone here can help me out or had similar problems in the past.
I'm using SikuliX Version 2.0.4

Thanks
Chris