User:HimanshiKalra/Report

= GSoC 2020: Automated Regression Testing Frameworks =

Final Report
Over the course of summer, I implemented 2 of my 3 Deliverables. The main goal of my project was to extend the framework such that adding new tests for modifiers becomes easier and automated. So that developers can quickly check against the tests to see if the new changes break any existing features. It also gives developers the freedom to experiment more with less worrying as unwanted or unintentional changes will get caught.

There were quite some tests already with the existing framework but it missed testing of modifiers which were a little complicated. Overall, we can now say that testing of the mesh modifiers is more or less "wholesome" or "complete" and the main goal of the project was achieved. New tests can be added over the course of time to make the system more robust to changes.

Quick links to the Code
 * Diff from master
 * Commit History vs master

Some other helpful links
 * Weekly reports
 * Proposal (updated)
 * How to Test

Core functionalities added over the period
 * Supporting framework for Physics modifiers.
 * Added support Deform Modifiers.
 * Created a generic helper class for all Modifiers.
 * Dedicated class for Particle System.
 * Support for testing of Curves.
 * Extending framework to use Operator in Object Mode.
 * Overall generalization of the framework, so that the interface is similar across different tests.
 * Added script for generating test/expected object pair [D8674]

Work Done
The existing mesh testing framework lacked support for testing of various modifiers. The goal of my project was to extend support for the remaining modifiers and write tests for it. The current framework evolved to support most of the mesh modifiers at present (2020) in Blender:

Curve Modifiers - They were tested by converting the Curves to meshes.

Deform Modifiers - Some of the deform modifiers like `Warp` required user input, this was achieved by using animation and shape keys.

Physics Modifiers - Each of them had its own quirks, most important was baking them, although that support was already added by me outside Gsoc, I refactored parts of it to generalize the baking. I worked on to supporting nested parameters as they are widely present in Physics modifiers.

Particle System - Although it is a part of the Physics modifiers, it required special attention and has a dedicated class for it, the approach to testing it is -
 * setting the "render type" to "Object".
 * converting the "instance object" to mesh.
 * joining it with the `test object`
 * comparing the test and expected object.

New Added tests


Around 30 new tests were added to the existing tests, the following have been listed type-wise.

Curve Modifiers
I added tests for Curves Modifiers, I will be explicit and name them

From Generate Category:
`Array, Bevel, Build, Decimate, EdgeSplit, Mirror, Screw, Solidify, Subdivision Surface, Triangulate, Weld`

From Deform Category:
`Cast, ShrinkWrap, SimpleDeform, Smooth, Wave, Curve`

Deform Special Modifiers
Under a different file, especially for Deform Modifiers, the following tests were added:

Deform Mesh Modifiers:
`Surface Deform, Mesh Deform, Hook, Laplace, Warp`

Deform Curve Modifiers:
`Armature, Lattice, Hook, Mesh Deform, Warp`

Physics Modifiers
The following tests were added:

`particle system, particle instance, ocean, cloth, dynamic paint`.

An attempt was made to include Fluid tests, but they became non-reproducible after some time.

Note: Tests for `softbody` already exists, new tests for `Cloth` were added, the blend file contains some other tests as well which were removed due to non-reproducibility.

Stage 1
I started by adding unique test name to all the existing tests present - modifiers, bevel, boolean, operator. Updated the `MeshTest` class and `ModifierSpec` and `OperatorEditModeSpec` to support the same.

I faced some issues while testing of Deform Modifiers for Mesh, the existing class `OperatorSpec` only supported operators in Edit Mode.

So I added a new class `OperatorSpecObjectMode`, a helper class for holding operators in Object Mode. Another class, `DeformModifierSpec` was added to hold a list of different types of Modifier Class Objects. And to hold the test themselves and run them, another class `DeformModifierTest` was introduced. There was slight difference between the already existing `ModifierTest` class and `DeformModifierTest` class in the sense that the latter supported jumping to a given `frame number` and was directly interacting with the main `MeshTest` class. Whereas `ModifierTest` had its own list and extracted parameters from that to be sent to `MeshTest`

Their corresponding methods:
 * _apply_deform_modifier: To call add_modifier and keep it in "unapplied" state, to call _add_object_operator and then apply the modifier.
 * _add_object_operator: To use the operator in Object mode.

Stage 2
There is support for curves introduced, by converting the curves to meshes. I also worked on running an individual test by a "unique name" for inspection in case it fails.

Stage 3
After a slightly heated discussion, I went ahead with creating a separate class for each Physics modifiers. So there were a lot of new classes added.

There already existed `PhysicsSpec` which could at that time test SoftBody and Cloth modifiers. Although everything went smooth with `FluidSpec`, the tests became non-reproducible after some time. The following method helped greatly in generalizing the framework.

_set_parameters_util: It takes a nested dictionary as input and applies the parameters using Depth First Search.

Using this method, all the classes except `ParticleSystemSpec` was merged into `ModiferSpec`. And the helper class for running tests is also generalized to `ModifierTest`

For current state, please refer here:
The tests were added throughout all the three Stages as and when the framework was ready.

Work on Test Object Generator Script
During the first month I was busy with building a prototype for generating test and expected object in masses. I wanted to use this tool to actually create tests. The original prototype was aimed at a naive user and accepted the following parameters to generate an object:
 * Object name - name of the object like (MyCube)
 * Object type - type of the object like (Cube, Plane etc.)

A unique name for each test/exp object pair was used as the Collection name. Note: This is not included in the original diff link given above. Initially there was also a support for adding a vertex group, which is removed in the current version.

The current version uses Object Oriented Programming Style and is aimed at smart users, as the code in the original diff became redundant and repetitive. It takes the following as parameters -
 * Unique Collection Name- to keep the blend file neat and organized by categorizing by test name
 * Object name: Add object attribute call- for example `"MyCube": 'primitive_cube_add'`
 * Count- for example Count = 5, it will create 5 copies of test and expected object pair. test0, test1, test2... and similarly exp0, exp1, exp2...and so on.

The current version is in a bit incomplete state as a refactor needs to be done as per the review but is in "Working" condition and can be used by interested parties.

Future Scope
This was my 3rd Deliverable and I changed it in favor of having a more "complete" testing framework. Some work has already been done in this direction. Check out here
 * Automated Compositor Testing

I added it as a stretch goal, this has to done from scratch but I am pretty sure the New framework will be helpful.
 * Bone Constraints Testing

The test object generator script addresses the last comment of Brecht to have a setup to create mesh objects and collections. Check out here
 * Performance Testing

And far in the future when we modifiers for Volume object and Point Cloud object are available, they can be tested as well.
 * Tests for Other Object types like Grease Pencil

Acknowledgements
I would like to thank my mentor Habib Gahbiche (zazizizou) for bearing my throughout this three month long journey and my backup mentor Bastien Montagne (mont29) for answering my queries.

Special mention to Niobio Cappelli (NiCapp) for answering my all user-interface and related queries and Ray Molenkamp (LazyDodo) for fixing the build errors and guiding with various other stuff. And to WHOLE BLENDER COMMUNITY! <3

Thank you very much, it was a beautiful summer.