User:HimanshiKalra/Report

GSoC 2020: Automated Regression Testing Frameworks

Final Report

Over the course of summer, I implemented 2 of my 3 Deliverables. The main goal of my project was to extend the framework such that adding new tests for modifiers becomes easier and automated. So that developers can quickly check against the tests to see if the new changes break any existing features. It also gives developers the freedom to experiment more with less worrying as unwanted or unintentional changes will get caught.

There were quite some tests already with the existing framework but it missed testing of modifiers which were a little complicated. Overall, we can now say that testing of the mesh modifiers is more or less "wholesome" or "complete" and the main goal of the project was achieved. New tests can be added over the course of time to make the system more robust to changes.

Quick links to the Code

• Diff from master
• Commit History vs master

Some other helpful links

• Weekly reports
• Proposal (updated)
• How to Test

Core functionalities added over the period

• Supporting framework for Physics modifiers.
• Added support Deform Modifiers.
• Created a generic helper class for all Modifiers.
• Dedicated class for Particle System.
• Support for testing of Curves.
• Extending framework to use Operator in Object Mode.
• Overall generalization of the framework, so that the interface is similar across different tests.
• Added script for generating test/expected object pair [D8674]

Work Done

The existing mesh testing framework lacked support for testing of various modifiers. The goal of my project was to extend support for the remaining modifiers and write tests for it. The current framework evolved to support most of the mesh modifiers at present (2020) in Blender:

Curve Modifiers - They were tested by converting the Curves to meshes.

Deform Modifiers - Some of the deform modifiers like Warp required user input, this was achieved by using animation and shape keys.

Physics Modifiers - Each of them had its own quirks, most important was baking them, although that support was already added by me outside Gsoc, I refactored parts of it to generalize the baking. I worked on to supporting nested parameters as they are widely present in Physics modifiers.

Particle System - Although it is a part of the Physics modifiers, it required special attention and has a dedicated class for it, the approach to testing it is -

• setting the "render type" to "Object".
• converting the "instance object" to mesh.
• joining it with the test object
• comparing the test and expected object.

New Added tests

Around 30 new tests were added to the existing tests, the following have been listed type-wise.

Curve Modifiers

I added tests for Curves Modifiers, I will be explicit and name them

From Generate Category:

Array, Bevel, Build, Decimate, EdgeSplit, Mirror, Screw, Solidify, Subdivision Surface, Triangulate, Weld

From Deform Category:

Cast, ShrinkWrap, SimpleDeform, Smooth, Wave, Curve

Deform Special Modifiers

Under a different file, especially for Deform Modifiers, the following tests were added:

Deform Mesh Modifiers:

Surface Deform, Mesh Deform, Hook, Laplace, Warp

Deform Curve Modifiers:

Armature, Lattice, Hook, Mesh Deform, Warp

Physics Modifiers

The following tests were added:

particle system, particle instance, ocean, cloth, dynamic paint.

An attempt was made to include Fluid tests, but they became non-reproducible after some time.

Note: Tests for softbody already exists, new tests for Cloth were added, the blend file contains some other tests as well which were removed due to non-reproducibility.

Evolution of Modifier Test Framework

Stage 1

I started by adding unique test name to all the existing tests present - modifiers, bevel, boolean, operator. Updated the MeshTest class and ModifierSpec and OperatorEditModeSpec to support the same.

I faced some issues while testing of Deform Modifiers for Mesh, the existing class OperatorSpec only supported operators in Edit Mode.

So I added a new class OperatorSpecObjectMode, a helper class for holding operators in Object Mode. Another class, DeformModifierSpec was added to hold a list of different types of Modifier Class Objects. And to hold the test themselves and run them, another class DeformModifierTest was introduced. There was slight difference between the already existing ModifierTest class and DeformModifierTest class in the sense that the latter supported jumping to a given frame number and was directly interacting with the main MeshTest class. Whereas ModifierTest had its own list and extracted parameters from that to be sent to MeshTest

Old framework	New Framework
ModiferSpec	ModifierSpec, DeformModifierSpec
OperatorSpec	OperatorSpec, OperatorSpecObjectMode
ModifierTest	ModifierTest, DeformModifierTest

Their corresponding methods:

• _apply_deform_modifier: To call add_modifier and keep it in "unapplied" state, to call _add_object_operator and then apply the modifier.
• _add_object_operator: To use the operator in Object mode.

Stage 2

There is support for curves introduced, by converting the curves to meshes. I also worked on running an individual test by a "unique name" for inspection in case it fails.

Old Framework	New Framework
Splitting of _apply_modifier	Two new methods _add_modifier and _apply_modifier, converting to CURVE to MESH
Removing Run Test by Index	Introducing Run Test by Unique Name

Stage 3

After a slightly heated discussion, I went ahead with creating a separate class for each Physics modifiers. So there were a lot of new classes added.

Old Framework	New Framework
PhysicsSpec	FluidSpec, DynamicPaintSpec, ParticleSystemSpec

There already existed PhysicsSpec which could at that time test SoftBody and Cloth modifiers. Although everything went smooth with FluidSpec, the tests became non-reproducible after some time. The following method helped greatly in generalizing the framework.

_set_parameters_util: It takes a nested dictionary as input and applies the parameters using Depth First Search.

Using this method, all the classes except ParticleSystemSpec was merged into ModiferSpec. And the helper class for running tests is also generalized to ModifierTest

For current state, please refer here:

Previous State	Current Framework
ModiferSpec, PhysicsSpec, FluidSpec, DynamicPaintSpec, ParticleSystemSpec	All merged into ModifierSpec and separate for ParticleSystemSpec
ModifierTest, DeformModiferTest, OperatorTest	Merged into RunTest (Updated test files and run_test method to access MeshTest)

The tests were added throughout all the three Stages as and when the framework was ready.

Work on Test Object Generator Script

During the first month I was busy with building a prototype for generating test and expected object in masses. I wanted to use this tool to actually create tests. The original prototype was aimed at a naive user and accepted the following parameters to generate an object:

• Object name - name of the object like (MyCube)
• Object type - type of the object like (Cube, Plane etc.)

A unique name for each test/exp object pair was used as the Collection name. Note: This is not included in the original diff link given above. Initially there was also a support for adding a vertex group, which is removed in the current version.

The current version uses Object Oriented Programming Style and is aimed at smart users, as the code in the original diff became redundant and repetitive. It takes the following as parameters -

• Unique Collection Name- to keep the blend file neat and organized by categorizing by test name
• Object name: Add object attribute call- for example "MyCube": 'primitive_cube_add'
• Count- for example Count = 5, it will create 5 copies of test and expected object pair. test0, test1, test2... and similarly exp0, exp1, exp2...and so on.

The current version is in a bit incomplete state as a refactor needs to be done as per the review but is in "Working" condition and can be used by interested parties.

Future Scope

• Automated Compositor Testing

This was my 3rd Deliverable and I changed it in favor of having a more "complete" testing framework. Some work has already been done in this direction. Check out here

• Bone Constraints Testing

I added it as a stretch goal, this has to done from scratch but I am pretty sure the New framework will be helpful.

• Performance Testing

The test object generator script addresses the last comment of Brecht to have a setup to create mesh objects and collections. Check out here

• Tests for Other Object types like Grease Pencil

And far in the future when we modifiers for Volume object and Point Cloud object are available, they can be tested as well.

Acknowledgements

I would like to thank my mentor Habib Gahbiche (zazizizou) for bearing my throughout this three month long journey and my backup mentor Bastien Montagne (mont29) for answering my queries.

Special mention to Niobio Cappelli (NiCapp) for answering my all user-interface and related queries and Ray Molenkamp (LazyDodo) for fixing the build errors and guiding with various other stuff. And to WHOLE BLENDER COMMUNITY! <3

Thank you very much, it was a beautiful summer.