User:LeifAndersen/GSoC2010/DesignDocuments

From BlenderWiki

Note: This is an archived version of the Blender Developer Wiki. The current and active wiki is available on wiki.blender.org.

Developer Documentation

The Blender Regression Tests Have several components which are used to test various functions of blender. These parts include:

Tests Operators
A Test Python Module
A Render Test
PyUnit
Hashes
CTest
Gui

Test Operators

There are several test operators used for this test, these are mainly just the functions that are performed often or would be usefull when testing blender while using it. These operators are all written in python and can be found in:

release/scripts/op/test_*.py

Currently, there are two files

test_hash.py
test_render.py

tests_hash.py

test_hash.py is currently the most commonly used file, it contains the operator for generating a hashcode for each blend file, as well as the means for reading and writing the hashtable to a hashfile.

(Note, I use the usage of the word hash in a slightly non-traditional sence. The idea is to create a unique number for the contents of a blend file, this needs to be fairly quick, but not as instantaneous as a usual hash function needs to be, also, these hashes aren't actually used for storing data in a hashtable, rather, the blend names is used for that).

Hash Function/Operator

Because Python's hash() function is not system independent, the test uses python's hashlib, which returns a hexadecimal string, capable of uniquely identifying the blend file (based on it's contents).

Because hashlib is used, every piece of data must be turned into a unique (but not system dependent) encoded unicode string, before it can be added to the total hash.

Currently, the hash function only cares about data that has been associated with an object in a scene. This was originally done because it seemed like that would be the only data that was cared about. However, when porting several of the python tests to the Blender 2.5x API, it was determined that some tests may be useful if they cared about raw data, not just that associated to the scenes, as such, the hash function eventually should be changed accordingly.

The largest possible issue in the hash function is how Blender calculates floating point numbers, which is system dependent. Fortunately, in most cases the numbers are similar, and only differ in roundoff error. This has been compensated for by simply rounding down the number before it's added to the hash. However, there are a few cases where a number turns out to be fundamentally different (0.3851 vs 0.4153 for example). A few lines that commonly caused this were commented out, but there are still a few tests that will fail on different computers.

Once the hash is complete, the data will be stored in the tests module, in particular, tests.hashfile.last_hash (which by default is 'NotRun'). If the hash function somehow fails in the middle of the hash, than the resulting hash will be 'Working' (otherwise, it will change from 'Working' to the proper hash value when it's completed).

Import/Export Operators

There are two functions for saving/loading hashfiles. (A hashfile is just a text file with each line in the form of:

 (hash_name) = (hash_value)

This format was chosen to keep the data persistent after blender closes, the main hashfile can be found in tests/hashfile.txt).

The hash operators will bring in the hashfile, and store it in a dictionary. The dictionary will be placed in the tests module, in specific, tests.hashfile.data.

tests_render.py

The render tests are currently not as complete as I would like them to be. There are two reasons for this.

Limitations in the Blender API prevent using normal operator API calls to do needed functions (which would require some hacking to get around, aka, calling blender within blender)
PIL is currently not running on python 3.x, and it was outside of the scope of this project to port PIL to python 3.x.

As such, the current solution is very horrible, and only their as a placeholder for when a better solution is made. As put by Andrea Weikert (elubie), your using blender, to run a python 3 script, to run a python 2 script, which runs blender (good/testing), to render your immage and compare it.

Basically, these tests simply have the use pick a blend file, and it calls the Image comparison test (talked about later), to compare the images. Do to limitations of the API, blender currently assumes that the good version of blender it's testing against, is in the users path (a bad idea). Optionally, the user can rely on the good images being pre-built, but it than has to conform to the file structure that the test imposes.

The run status of the render tests can be found in the tests module, in specific tests.render

A Test Python Module

The Tests module can be found in:

release/scripts/modules/tests/

The Test module is by far the most clunky part of the current suite. It was originally introduced back at the beginning of the project, before I learned how blender operators worked (or rather, thought in that type of mindset), as such, it originally stored many of the functions that are now as operators. (It also stored several other functions, that are now deemed pointless, and no longer in the codebase).

Now, the only reason the code exists is do to what appears to be a possible flaw (although it is an architectural design), of the way Blender operators work, which is to say the only data they can returned is pass/fail/not run/etc. data. Rather, they are supposed to manipulate the context/data of the blend file. As such, there was two options. I could modify bContext to include tests, or simply keep the tests module I was about to throw out (on a code cleanup cycle). Although modifying bContext is probably the preferred option, the time it would have taken to inject the tests portions would not have been worth it, as such, I opted for the tests module. In time though, it would probably be better to remove it altogether.

In other words, the tests module is nothing more than a giant global struct, that tests operators can dump values into and get values out of.

A Render Test

Implementation

The Render tests are a suite of tests that can be run independently, or as part of a larger CTest suite, for comparing rendered images in Blender. can be found in:

tests/render

Unlike the rest of the suite, the render test has been written in python 2.x. This is entirely because PIL currently does not work with Python 3.x. Two major redactors must be made before it will work with python 3.x:

the parentheses around print
string.replace(stringVar, old, new) -> stringVar.replace(old, new)

other than that, it should work with python 3.x.

The structure of these tests consist of an image test case (not associated with pyunit, or the XUnit framework at all), and an animation test case, which is extended from the image test case. It also has some surrounding functions for generating HTML output, getting the local tests, and parsing parameters.

The work flow for each test case is:

Initialize the test case.
Set it up for image comparisons (give it a good path, test path, and path for the blender file)
Render the test
Compare the rendered images
(optional) save a diff(s) of the images

Steps 1 and 2 were initially done at the same time, however this was changed after the animation tests were made, because the cases were going to be set up with alternate routes to follow, mainly comparing hashes. However, this became to complex and was later removed, although the structure is now probably sound enough to allow it to be placed back in, the functionality is not currently needed. As such, the split steps remains as an artifact, in case hash tests ever do get put back in.

Output

The HTML output relies upon some CSS to format the table, this CSS is based upon the CSS from the Sphinx generated Blender API documentation. At one point, the script generated the documentation itself, but it really killed the clarity of the script, so it simply assumes it's in the same folder as the created HTML file.

Image Comparison Algorithm

The algorithm for comparing images is thus: for every pixel in the images: find the absolute value of the difference between the pixels. (in the three color bands) If the sum of the three differences is too large, add it to the failed pixel count Add the sum of the three differences to the total difference some of the whole image if the different pixels are too large, the test failed if the total difference is too large, the test failed

For animation tests, it preforms this algorithm for every rendered image, it than allows for a small margin of error in images failing (about 1%), before it fails the animation.

Interpretation of Test Results

Currently, the largest problem with this test is particles. I'm not sure whether they should be considered as passed, but particle simulations are slightly different in different versions of Blender, leading to high image differences.

Integration

In addition to working as an independent unit, these tests can also be plugged into the whole test suite. See the CMakeLists.txt file in the tests/render folder to see how it is done. Essentially, for every blend file in the folder, it runs the render test independently with that one blend file. See the tests arguments (in the manual, or with the -h flag) for the arguments.

PyUnit

With the exception of the render tests, pyunit is really the skeleton of the rests of the tests. Currently though, pyunit only does one main thing, which is to compare hashes. There are currently two pyunit files in the build, they can be found at:

tests/hash_compare.py

and

tests/python/hash_compare.py

Both scripts are built with python 3.x, and designed to be run in blender.

The first script simply compares the hash of the current blend file to the one on record (stored in hashfile.txt, put in the bin folder at build time), gives a pass or a fail, and exits blender.

The second one is slightly more complicated. This second one tests the hash compared to the one stored on file (stored as filename.py.blend_start), assuming it passes, it loads the tests python commands (stored as filename.py), and executes the scripts func() method. (It uses __import__ to load the module at runtime). It then compares the new hash of the blend file to a file one stored in hashfile.txt (stored as filename.py.blend_end), returns the pass or fail, and exits blender.

A few of these tests are currently not ported to the Blender 2.5x Python API, these are listed in the tests/python/README.txt file).

The first file is used for most of the tests in the system, the second (thus far) is only used for python tests.

If you look at the CMakeLists.txt file for the python folder, you'll notice that Blender is opening in the foreground, running the script, and then closing blender (and rapidly timing out in case of a script failure). The reason for this (and not simply opening up in the background)are limitations in the API, that were originally causing problems when using it in background mode. Thus, until they get fixed, they need to stay in the foreground.

For more details about what each folder does, view the user documentation. Also, note that not every folder that uses these tests use them exclusively, some of them also use their own baking tests, which can cause some problems with duplicate tests due to the loop CMake uses to generate the CTest tests.

Hashes

Hashes are the meat of the test suite. As mentioned above, the word hash is not being used in it's traditional computer science sense. But rather, to mean a unique ID for the data of each blend file. Currently, they do need to be system independent, however, this may eventually be found to be limiting, and a way to make it work while being system dependent may be found.

Hashfile

A hashfile is simply a text file where each line is:

(hashname) = (hashcode)

both hashname and hashcode will become strings, and will be read into a dictionary by the read_hashfile() operator. Lines with # are comments, and empty lines are allowed. The reason that this format was chosen, was because something that was persistent across blender running was needed, and setting up a full database seemed inappropriate, as well as storing it in a specially crafted blend file.

The main hashfile for the tests can be found in tests/hashfile.txt, but at build time, will be put in the build directory's bin folder, which is where most scripts will be looking for it.

Hash Details

A hash is simply a string generated by hashlib. In particular, it needs to be a hexstring, this was decided just in case it was needed for the hash to be safe from accidental parsing, otherwise, it could just be any string.

CTest

CTest (powered by CMake), is what was used to automate the tests. This is why Scons is not currently supported. I tried to separate my CMake code as much as possible from the trunk's CMake code, but I had to modify two files. The first one is the main file. It is what creates the WITH_TESTS option, when true, it enables tests, and adds the extra subdirectories to the build. The second one was in the creator's CMake file. It is just a custom command for Unix/Mac/Windows. The reason I put those custom commands in there (rather than in the root testing folder, is because I couldn't get CMake to run those commands unless they were in there, and after spending a few days trying to figure it out, I thought it would be best to put it in there, at least for the time being, and move one. (This was at the beginning of the project). The custom commands simply copy the tests from the build directory, to the source directory. This was done because the tests (written in python), are essentially not affected by CMake, and I thought that it would be good for the build directory to work independent of the source directory.

The remainder of the CMake files are just tests. Many of the tests are generated by a loop which finds all of a certain type of file in it, this allows for easy adding of tests. However, it also creates a undesirable side effect, which is to say, sometimes there's a file in there that isn't supposed to be automated, as such, when the loop finds it (and adds it/runs it), the test fails. Thus, in the future, perhaps a folder for the tests generated by a loop would be a good idea. See each folder's CMakeLists.txt file to see how a test was added.

Each test also has it's own properties set. This is because PyUnit didn't seem to return anything special on pass/fail, thus, in order to determine if the tests passed or failed, regular expressions were used. For PyUnit, if OK is found, than the test passed, for the render tests, if All tests passed is found, than the tests passed. Also in the properties is a strict timeout (with the exception of the render tests, which require longer timeouts), this is because if the python script crashed, Blender still needs to close the window soon, rather than pointlessly leaving it open for 25 minutes (the default timeout).

Gui

Like the tests module, this is one of the more clunky parts of the test suite. However, unlike the tests module, I believe it should stay. The idea is that with some of the tools (especially the hash tool), it would be useful for a dev to have it right on hand, rather than needing to run it through a suite. Thus, there is a tests panel, which simply runs many of the tests operators. However, this was a very small portion of the project, and as such, it is not very elegant, very little thought was put into it's design, and it should likely be improved.

There is also a Tests menu for the main menu at the top of the screen. However, without editing that file directly, it was impossible to place it in the appropriate spot. Furthermore, with changes that occurred to the API and registering operators/ui elements/etc. The function that was used to add the menu to the bar at registration time could no longer be used. As such, it no longer appears (by default), in the GUI, but the item is still there.

The gui can be found in:

release/scrips/ui/space_tests.py

File Structure

Here is a brief layout of the file structure of the main modifications I made. I did make some modifications to the core of Blender a quarter of the way through the project, to bypass a few buggy things for me, but I think that by now, those modifications have dissipated, do to the merges I've done so far. For this reason, those changes will not be listed here.

For more details on what each folder is for, read the user documentation, or the README.txt file in each folder (that has one).

release/scripts/

This is the location of the tests module and operators

release/scripts/op

This is where the tests operators go, currently there are two, tests_hash.py and tests_render.py

release/scripts/modules/tests

This is where the tests module is. There are currently two submodules, tests.render, and tests.hashfile

tests

This is where the majority of the actual tests are located

tests/addon

This is where any additional python based test can be placed, and it will be added to the suite

tests/data

This is for tests that try to modify raw data for blender, without going through operators. (Currently these tests are disabled as they are unstable).

tests/export_import_testing

This is for testing import/export files. These tests were simply pulled from the old suite, as such, they're incomplete (as no one finished making them in the old suite).

tests/gameengine

This is for testing the game engine. Unfortunately, these tests have not been automated yet. So it's basically just a copied version of the old gameengine tests.

tests/manual

These are tests which are not well suited for automation. Mainly because they're for testing to ensure the UI works as expected. These currently must be run by hand.

tests/mesh_modeling

These are tests to ensure that modeling a mesh works properly. Note that currently the only thing that is done is a hash compare, thus it needs to be updated to also deal with any required actions.

tests/physics

This tests to ensure that Bullet works properly. It does this by baking a fluid (and comparing renders) in one case, as well as making a few modifications (and comparing renders), in another case. The rest are just hash tests, and may require further assistance in the future (such as the softbody test).

tests/python

These test the way the python scripts work. It could be said that it is a test to ensure the tests work properly, as the tests are made primarily with python. Their usage is talked about in detail in the previous section. In short, it compares the hashfile, makes changes, and compares again. Note that there are a few scripts there that are still using the old Blender Python 2.4x API.

tests/render

This section works both independently, and with the rest of the tests. It is a simple render comparison test. It is talked about in detail in the previous section.

tests/render/anim

This is the location for the animation files for the suite. Note that good animation renders would go in this folder, in subfolders that share the same filename as the blend file, without the .blend at the end.

tests/render/render

While currently not present, this is where the good image renders would go.

tests/sequence_editing

This tests the sequence editing to ensure that it works properly. Note that currently it only tests the hash of the scene, so these tests could use further assistance.

Other changes

Other changed files are:

CMakeLists.txt

CTestConfig.cmake

source/creator/CMakeLists.txt

and a few other source files listed in the logs which have likely been changed back now by merges.

Page Siblings