Test in Yzis
From Yzis Wiki
We believe strongly in testing to support software quality. Let's look at what kind of tests you can find in Yzis.
All tests are compiled by default, unless you disable them when running CMake with a -DENABLE_TESTS=OFF .
There are two kinds of tests : Unit tests and functional tests, which are done in Lua and test the code on a higher level.
Contents |
Unit tests
Introduction
Unit test are tests written in C++, using Qt unit testing framework. The goal of unit testing is to test an individual class; they are not meant to test complicated stuff. Unit testing is fundamental in Extreme Programming philosophy. Independently of Extreme Programming, experience shows that unit tests ensure very low defect rate in the long term.
There are not many unit tests at this point. There used to be more but the migration to Qt4 and various development have made previous versions of the test too tedious to maintain. New features making it into Yzis are getting their unit test nowadays, so the number will grow again.
Running unit tests
The unit tests are located in tests/unittest. They integrate source code from libyzis directly, so they do not depend on libyzis. You can run them without any libyzis library lying around.
To execute all the tests, simply run the generated executable: yzis_unittest (yzis_unitests.exe on windows).
There is also a run_unittests.sh script in the source tree that runs the tests.
The executable will run each test suite and display something like this:
D:\work\work\yzis-dev\yzis1\build-cmake\tests\unittest>yzis_unittest.exe ********* Start testing of TestYZDebugBackend ********* Config: Using QTest library 4.2.2, Qt 4.2.2 PASS : TestYZDebugBackend::initTestCase() QDEBUG : TestYZDebugBackend::testAreaLevel() YZDebugBackend::YZDebugBackend() co nstructor PASS : TestYZDebugBackend::testAreaLevel() PASS : TestYZDebugBackend::testSubAreaLevel() PASS : TestYZDebugBackend::testSprintf() PASS : TestYZDebugBackend::testWhere() PASS : TestYZDebugBackend::testParseArgv() PASS : TestYZDebugBackend::testParseArgv2() PASS : TestYZDebugBackend::testParseRcFile() PASS : TestYZDebugBackend::cleanupTestCase() Totals: 9 passed, 0 failed, 0 skipped ********* Finished testing of TestYZDebugBackend ********* ********* Start testing of TestColor ********* Config: Using QTest library 4.2.2, Qt 4.2.2 PASS : TestColor::initTestCase() PASS : TestColor::testColor1() PASS : TestColor::cleanupTestCase() Totals: 3 passed, 0 failed, 0 skipped ********* Finished testing of TestColor ********* ********* Start testing of TestResource ********* Config: Using QTest library 4.2.2, Qt 4.2.2 PASS : TestResource::initTestCase() PASS : TestResource::testResourceMgrNewDel() PASS : TestResource::testYzisHomeEnv() PASS : TestResource::testResourceMgrCreatesYzisDir() PASS : TestResource::testUserScript() PASS : TestResource::testConfigScript() PASS : TestResource::testSyntaxFile() PASS : TestResource::testIndentFile() PASS : TestResource::testConfigFile() PASS : TestResource::testWConfigFile() PASS : TestResource::cleanupTestCase() Totals: 11 passed, 0 failed, 0 skipped ********* Finished testing of TestResource *********
The executable will return the number or tests failing in its exit status, which means 0 when everything goes fine.
If something goes wrong, the output will show which tests are failing, at which line in the code, with the information of what is expected and what was received.
It is possible to run only one test suite, by passing its name as first argument:
D:\work\work\yzis-dev\yzis1\build-cmake\tests\unittest>yzis_unittest.exe TestColor ********* Start testing of TestColor ********* Config: Using QTest library 4.2.2, Qt 4.2.2 PASS : TestColor::initTestCase() PASS : TestColor::testColor1() PASS : TestColor::cleanupTestCase() Totals: 3 passed, 0 failed, 0 skipped ********* Finished testing of TestColor *********
Writing Unit Tests
Writing unit tests is quite simple. There is Qt documentation for this at: doc.trolltech.com
Briefly, you need to:
- create files TestSomething.h and TestSomething.cpp that contain your test class
- add the cpp file to the list of sources in tests/unittest/CMakeLists.txt and add the header file to the list of files needing moc.
- in main.cpp , include your "testSomething.h" at the beginning.
- near the end of main.cpp, add a line RUN_MY_TEST( TestSomething )
Then, you can start to write your tests. The main.cpp looks a bit complicated because QtUnit is designed to run only one test suite. We had to hack around to run multiple test suites at once. But you don't need to understand, just add your test line and that's it.
If you write some unit tests that depend on some options, you may run into problems because of the user settings in his ~/.yzis . The proper way around this has been implemented in TestResource.cpp: init(), cleanup(), initTestCase() and cleanupTestCase() take care of renaming the existing ~/.yzis user directory into something else during the tests, and rename it back when the test is finished. The test won't execute if the renaming is not successful.
This was very important for example, when unit-testing the different mechanism of resource loading.
Libyzisrunner
Libyzisrunner is kind of an empty GUI front-end. It implements the frontend interface with minimal code, displays nothing. It serves as a test reference frontend for libyzis. With libyzisrunner, we are sure that there are no bugs introduced by the gui.
Libyzisrunner is mainly used for Lua tests.
Lua tests
Lua tests are functional tests of Yzis. The idea is to use Yzis scripting to validate its internal behavior. Lua tests are meant to be more elaborate than unit tests. They cover a wider functionality set.
Ideally, every feature of Yzis would have its corresponding Lua test, so that we can validate every aspect of Yzis engine. We are not in an ideal world, but the situation still looks quite good.
Lua tests are located in tests/scripts
How it works
One of the Yzis frontend is run with a -s test_something.lua argument. Yzis will execute the Lua test_something.lua . At the end of the script, there is a function call setLuaReturnValue() which sets a string containing the return value for Yzis. That string is converted into an integer (if possible) and is used as the exit code of Yzis.
The test_something.lua script is based on a Lua Unit Testing library (written especially for Yzis), with a classical infrastructure: assertions, test cases with several test functions, runner to run all tests.
Inside the Lua script, a pseudo-class (Lua does not have real classes) whose name starts with Test must be present, and contain several methods whose name starts with test. The LuaUnit framework will analyse all this and run all the tests of all the classes. One simple way to disable a test is to make his name not start by test, for example by prefixing it with an X.
It is possible at the end of the Lua script, to tell to run only one class, or only one test method of one class. Uncomment the appropriate lines, it is self-explanatory.
Running Lua tests
Those tests are in the directory tests/scripts
Introduction
In theory, running Lua tests should be as simple as typing:
qyzis -s test_something.lua
In practice, there are a few traps that need to be overcome:
- you want to run the tests on the latest, just compiled Yzis engine. This means that the frontend must pick up the library that was built as part of the build process, and not the libyzis library which is installed.
- the ~/.yzis directory of the user may mess up the tests, or the tests may mess up the ~/.yzis directory
This is why we had to develop a bash script, that gets around those problems: run_scripttests.sh
run_scripttest.sh usage
On Linux, run_scripttest.sh is self executable with bash.
On Windows, the situation is a bit more complicated. You need the cygwin bash to run the script. But remember that Yzis compilation can not be done with cygwin, only with a dos window. We hope to simplify the situation in the future.
run_scripttests.sh may take two arguments:
- executable argument: qyzis, nyzis, libyzisrunner or gdb. Runs the script argument with the designated frontend. By default, libyzisrunner is used. With gdb, libyzisrunner is loaded into gdb
- script argument: a script file. Runs the designated lua file, or test_all.lua if the argument is not provided.
Examples:
- Running all the tests with qyzis:
./run_scripttests.sh qyzis
- Running test_movements.lua with libyzisrunner
./run_scripttests.sh test_movements.lua
Only one lua script can be run at a given time. The script will display whether it is successful or not.
The lua scripts test the internal of libyzis. So in theory, there should be no difference between the different frontends. But there is no harm testing it.
Example of running a script:
User@Phil_vaio /cygdrive/d/work/work/yzis-dev/yzis2/tests/scripts $ ./run_scripttests.sh test_regexp.lua Copying yzis files Testing with: libyzisrunner test_regexp.lua LANG=C ./libyzisrunner -c ":source test_regexp.lua <ENTER><ESC>:qall!<ENTER>" YZDebugBackend::YZDebugBackend() constructor >>>>>>>>> TestRegexp >>> TestRegexp:test_capture_info >>> TestRegexp:test_doc_examples >>> TestRegexp:test_get_pattern >>> TestRegexp:test_matchIndex >>> TestRegexp:test_real_regexp >>> TestRegexp:test_regexp_options >>> TestRegexp:test_replace >>> TestRegexp:test_try_to_force_finalize >>> TestRegexp:test_two_re >>> TestRegexp:test_usage >>> TestRegexp:test_vim_regexp
Writing Lua tests
Writing lua tests is extremely simple. The simple way to do it is to take an existing lua test, and modify it to your needs. For example, test_movements.lua is quite simple but contains everything you need to know.
TestMovements = {} --class
function TestMovements:setUp()
clearBuffer()
sendkeys( "<ESC>" )
end
function TestMovements:tearDown()
clearBuffer()
end
function TestMovements:test_insert_mode()
assertEquals( bufferContent(), "" )
sendkeys( "i" )
assertEquals( bufferContent(), "" )
assertEquals(mode(), MODE_INSERT)
assertPos( 1, 1 )
-- ...
end
-- ...
-- ...
if not _REQUIREDNAME then
-- ret = LuaUnit:run('TestMovements:test_char_movement') -- will execute only one test
ret = LuaUnit:run() -- will execute all tests
setLuaReturnValue( ret )
end
The first line is a pseudo class declaration. Lua does not have real classes, so you declare a class by declaring a list. But you don't really care, just copy the line verbatim. The rule is that your class name must start with Test
The functions setUp() and tearDown(), if declared are called before and after each test.
Each test function name should start with test . Inside the function, you can send key asif you would be typing them with sendKeys(). And you can make assertions with assertEquals( currentResult, expectedResult ). There is a shortcut, assertPos() which checks the cursor position. The file utils.lua contains several small utility function like this one.
At the end of the file, the _REQUIREDNAME checks whether the file was launch standalone or if it was imported by another script. If it was launched as standalone, it calls LuaUnit:run() which will take care of discovering the test classes, the test functions and run everything. LuaUnit:run() will return the number of tests that failed, so 0 if everything went fine.
setLuaReturnValue() sets an integer or a string which will be taken by Yzis as the return value from the script.
That's it.
When there is an error in a script, lua displays a stack trace, and you can see all the detailed information. Check Scripting with Lua for a more information on what you can do with Yzis and Lua.
Files for testing
The directory tests/files contains a few files that can be useful for testing the display:
- files with different encodings
- a file with all combinations of tab and spaces
- a file in C++ to check syntax coloration
Running all the tests at once
At this point, it is not possible to run all the tests at once, with one command. However, you can run the two sets of tests easily.
You can run all the unit tests with:
make tests
And you can run all the Lua tests with:
cd tests/scripts && run_scripttests.sh
make tests uses CMake and CTest but there is no output at the moment of why things fails when a test does not pass. Still, it displays the test results. Run build/tests/unittests/yzis_unittests to see when every test fails.
run_scripttests.sh requires bash, which is complicated to manage on Windows. That's why it is difficult to integrate it simply with CMake and CTest.
Still, if you want to check if Yzis is working properly, you can do:
make tests cd tests/scripts && run_scripttests.sh
Future
More tests will be added in the future.
(Future) doc tests
The classes that have been properly documented generate 0 warnings with doxygen. Unfortunately, this concerns only part of the Yzis source code. Anyway, in the future, we may have tests that enforce that for some classes, no doxygen warnings are allowed.
This will ensure that all arguments and return values are properly documented.
(Future) ebn tests
EBN stands for English Breakfast Network, which you can find here. EBN maintains a set of scripts that validate some good coding practices. Yzis already passes most of those scripts. At some point, we want to have the tests be run regularly and published on the EBN network.
