Separating identical test ideas needed.

Asked by sdraganov on 2018-06-30

I'm facing probably very common issue but don't know what the best way to deal with it.
The issue is that I'm running one sikuli-script that runs the same test over and over again, and if something goes wrong it just stops.
Currently it takes about 3-4 hours to complete the entire test and that time constantly increases because of adding more functionality. Very often the script stops because of missing image, and I am wondering is there a way to tell sikuli to not stop entire test but to do some operations like quit running program and continue running script from specific place.

 The obvious way to resolve this is probably by separating that giant test to number of small one, but here is the catch. The script is not that big at all, it just repeats around 80 times. I don't know how to split it to 80 individual tests since those repetitions depends from the number of the builds that are about to be tested. I place the builds into a folder, read it's content, then I loop the script for each of the builds into the folder. The number of the builds is not the same each day so I can't hardcode 80 individual tests.

May someone share a reliable way to deal with this case? What I want basically is if something fails to stop immediately, close everything started by the script, and restart it from begging.

I am open for any ideas. Maybe the entire logic of my tests is senseless but I really have no idea how to do it.

Here is a video from my latest test which failed because of missing image at ~38min.
https://drive.google.com/file/d/1PuIQ3ZwWbSr2qpyUlLRUwVKbrpB5np2i/view?usp=sharing

Question information

Language:
English Edit question
Status:
Answered
For:
Sikuli Edit question
Assignee:
No assignee Edit question
Last query:
2018-06-30
Last reply:
2018-07-03
RaiMan (raimund-hocke) said : #2

@masuo
additionally this question is relevant:
https://answers.launchpad.net/sikuli/+question/670456

The docs are not up to date yet.

RaiMan (raimund-hocke) said : #3

Generally you have to identify the parts, that are repeated the same way, but with different parameters (as I understand: your builds) and move those to reusable functions, that might reside in modules (other scripts). These functions are then called with a specific parameter set one after the other.
For this you have to get familiar with the Python function concept and read:
http://sikulix-2014.readthedocs.io/en/latest/scripting.html#importing-other-sikuli-scripts-reuse-code-and-images

Another option is to use the unittest approach, where you define different testcases and a setup/teardown that is repeated for every testcase. The testcases (if implemented accordingly) are completely independent. Then you can run a configurable series of testcases where each case may pass or fail, but all will be run.
This might well be combined with the features mentioned in comment #1.
https://docs.python.org/2/library/unittest.html

Of interest might then be the HTMLTestRunner feature, that produces nice testreports that can be opened in a browser.
https://answers.launchpad.net/sikuli/+question/670542

sdraganov (sdraganov) said : #4

Thanks for the suggestions guys.
I was familiar with setFindFailedResponse() and setFindFailedHandler() from the docs but I didn't know they would be useful in this case. I am still learning the basics of programing/scripting and still missing some elementary things like this.

Regarding the RaiMan tips. I am familiar with importing and reusing code and images. A few weeks ago I split my script into separate modules like MAIN module, FunctionLib module and ImageLib module, here is a scheme of how each module is imported into other ones: https://i.imgur.com/kUyyuVG.png

For the test I am using a simple for-loop operator, and I am basically executing the same test over and over again.
I'm not very familiar with unit-tests. I have heard of them but never used them into my scripts. I already read some of the information on python unit test help but still cannot wrap my head how to execute the same test multiple times and got results for multiple tests. Setup and TearDown methods could be used to run some actions before and after the actual test, I can see how I can use them but still don't have idea the way I should code the test to run multiple times with different input parameters, should I use some kind of loop or it's something completely different?

RaiMan (raimund-hocke) said : #5

--- Setup and TearDown methods could be used to run some actions before and after the actual test, I can see how I can use them but still don't have idea the way I should code the test to run multiple times with different input parameters, should I use some kind of loop or it's something completely different?

Since I do not know your final intention about becoming some expert with unit-testing with all its aspects (data-driven, fixtures, ...), this would be a tailored approach for your specific situation.

Be aware: to keep it simple some statements are "pseudocode" ;-)

import unittest
import HTMLTestRunner # because it makes nice reports
from testLogic import innerTest # your test function to be repeated

class MyTest(unittest.TestCase):
  def setUp(self):
    print "MyTest: setUp"
    global parm1, parm2, ...
    # somehow fill parmX with a set of values for one test run
    # here you might decide to stop the outer loop using system exit

  def testToBeRepeated(self):
    print "something to show up in the report"
    whatEver = innerTest(parm1, parm2, ...)
    assert whatEver

  def tearDown(self):
    print "MyTest: tearDown"
    # here you might do something

testReport = file(os.path.join(getParentPath(), 'testReport.html'), 'a')
# each run will append its report

testSuite = unittest.TestLoader().loadTestsFromTestCase(MyTest)
runner = HTMLTestRunner.HTMLTestRunner(stream = testReport)

# simply repeat the testrun until it is killed from inside
while True:
   runner.run(testSuite)

Can you help with this problem?

Provide an answer of your own, or ask sdraganov for more information if necessary.

To post a message you must log in.