Tools/Tests/Adding New Tests

Introduction for Automated Testing Framework
The current framework supports tests for-
 * Mesh Modifiers
 * Generate
 * Deform
 * Physics
 * Curve Modifiers
 * Operators

Note-The scope of these tests is limited to modifiers.

There are some exceptions and limitations as each modifier is a bit different, some requires to switch modes (Edit <--> Object), some are just random and non-reproducible, and some output an Empty Mesh, hence its difficult or not possible to test them.

Note The current framework support is only for "mesh" and "curve" modifiers, compositor testing is not supported, nor are the new object types like Volume.

Need for Automated Testing
Automated testing improves software quality. It detects bugs very early in the development process and makes fixing them very cheap. It can also detect unintentional changes. Downside is it is more code to maintain and increases building time (e.g. if tests are ran as part of building in buildbot). So it is important to keep them as short and as fast as possible to minimize false alarms and therefore minimize maintenance effort.

How to get started with Automated Testing?
Follow the starting instructions on this page which shows how to run python tests- Python Tests

You can skip the `ctest` part for now if its a bit too much and come back to this page.

Quick Steps in case you couldn't follow : Note: `blender` should be set in your environment variables in the Path variable, the Path consists of the path/to/blender where your `blender.exe` file is located. OR
 * 1) Start blender by typing `blender` in the command prompt.
 * 1) Try typing `blender.exe` (method 1 is preferred).

Open the command prompt and the enter the following command.

Note- the path given here of the python test file is relative, you need to navigate using the `cd` command inside of `blender-git/blender`

blender --background -noaudio --python tests/python/bl_pyapi_mathutils.py -- --verbose

Or another example with absolute path would be

blender -b c:\blender-git\lib\tests\modeling\modifiers.blend --python c:\blender-git\blender\tests\python\curve_modifiers.py -- --run-all-tests -- --verbose

The Output would look something like this if they pass:

Comparing... Sucesss!

Addition of new tests
The current tests for Mesh Modifiers are located in `tests/python/modifiers.py` file. Navigate to this file and add another test for "Array" Modifier.

Steps to follow: SpecMeshTest("LearnTestingArray", "LearnTestCubeArray", "LearnExpCubeArray", [ModifierSpec('array', 'ARRAY', {})]),
 * 1) Navigate to the supporting blend files which are located in the `lib\tests\` folder.
 * 2) Open the file `modifiers.blend` which is located in the `modeling` folder.
 * 3) Open the blend file and add 2 `Cubes`.
 * 4) Rename them to "LearnTestCubeArray", and the second one "LearnExpCubeArray" respectively.
 * 5) Now open the python file and add this line in the tests list

Note: To have a better understanding for each parameter, navigate to `tests/python/modules/mesh_test.py` and look at the comments in "ModifierSpec" class, this will give a clearer idea of what's possible.

Different passed arguments

 * 1) --verbose : First set your environment variable. Type ```set BLENDER_VERBOSE=1``` in the command prompt (cmd). Note, it doesn't work for Windows Powershell.
 * 2) --run-test : To run a single test for visualization and comparison purpose in Blender.
 * 3) --run-all-tests : To run all tests.

How Automated Testing works?
Good question, the trick is you don't have to manually add and apply the modifier in Blender at all, on the "LearnExpCubeArray" that is the expected object, you can but then that is doing it manually. So here are the steps- Note - Failing of a test doesn't necessarily mean its a bug.
 * 1) After you add the test and expected object, no need to apply the modifier to the expected object.
 * 2) Set the `BLENDER_TEST_UPDATE` flag to 1 (or any value look at Stackoverflow on how variables are "set", 1 is just the simplest), and it will update the expected object for you.
 * 3) There is a good chance that the test will fail for the first time, if you look closely at the console, then get updated and then should pass.
 * 4) The file will get saved with the updated expected object, so the next time you run the test, it will pass in first go.

Changing Parameters of the Added Tests
Turn on the Python tool tip in the Preferences in Blender. You can now hover the setting you want to change and look at the attribute name. For example, let's increase the `count` in the `Array` modifier. Then the code will look like this - "LearnTestingArray", "LearnTestCubeArray", "LearnExpCubeArray", [ModifierSpec('array', 'ARRAY', {'count': 4})]],

Note- The setting name in Blender and actual setting might be different, that's where the tooltip will help you.

Contributing, Reviewing and Committing
For Contributing: The tests can be contributed by anyone.

For Reviewing: One module member of the affected area should be included, Jacques or Hans (Geometry Nodes), Bastien or Campbell (modifiers/operators), Jeroen Baker (Compositor) and Brecht for performance tests. Apart from the core members, volunteers to add for first pass and initial feedback:
 * Habib Gahbiche (zazizizou): Operators, modifiers, compositors and geometry nodes tests.
 * Himanshi Kalra (calra): Operators, modifiers and geometry nodes tests.

For committing: One requires commit access and id and password of projects.blender.org, I use SmartSVN to commit the test files. Like other patches, test files can also be committed on other's behalf.

Points to remember

 * Try to keep the meshes as simple as possible.
 * Make sure the tests pass after the `BLENDER_TEST_UPDATE` flag is un-set.

Future Scope
Adding tests for new modifiers and extending the framework for new object types.

What to do my test is failing?
Use `Ctrl+F` and find a similar test, look closely at the comments and see if its a corner case.

Its still failing? Let's dig deeper...
For Windows: Set the BLENDER_TEST_UPDATE flag=1 in the environment variable, so now when you run the test, the "expected object" gets updated automatically, if all's good it should pass.

If it still fails, you can manually investigate by running a single test using the `--run-test `, see the example below blender c:\blender-git\lib\tests\modeling\modifiers.blend --python c:\blender-git\blender\tests\python\curve_modifiers.py -- --run-test CurveWeld -- --verbose Here you will find a new object called "evaluated_object", you can try applying the modifiers to it and see if its the same as the expected object you added.

In case you are interested in testing operators, they follow a similar approach.

What to do when getting errors in console?
Try slowly reading through it, it could be a typo in parameter name, so the exception thrown would be an `AttributeError` and some common programming mistake.

Where are the Physics modifiers' tests located?
Each Physics modifier has been given a separate python file and blend file, the python file name usually starts with `physics_`, so you can test them as well, have a look at already added tests to see how to send the parameters, they use nested dictionary to send nested parameters. The supporting blend files are in `physics` folder.

Deform modifiers have a common blend file for Deform tests of both Mesh and Curves, it contains mostly the one that requires user input, which has been achieved by animating the object.