From BlenderWiki

Development of Blender's Non-Photorealistic Rendering Capabilities: Final Report

Author

Tamito Kajiyama

Date

24 October 2014

Introduction

This document reports the outcome of the Blender development project (April—September 2014) conducted by the Author based on the financial support granted by the Blender Foundation Development Fund. The development targets of the project are defined in the project proposal as follows:

1. Freestyle performance tuning in terms of memory consumption and rendering time.
2. Better artists experience during interactive line stylization exercises in Freestyle.
3. Technical support for the general development of Blender NPR rendering features.

Blender artists and developers were actively involved in the selection of specific priority targets along these development lines, through discussions in the bf-blender-npr mailing list as well as social networks (e.g., Twitter and Facebook).

This report briefly presents the results and deliverables of the project, followed by summary statistics of development activities during the project.

Results and deliverables

Subsequent sections highlight key improvements and new features according to the three development lines. Some of the results have already been described in detail in the first and second interim report. Full detail of development activities throughout the project can be found in the weekly reports.

Freestyle performance improvements

Relatively high memory consumption in the Freestyle rendering process is a known performance bottleneck which makes Freestyle less applicable to large input scenes with high polygon counts. Being a geometry-based line rendering system, Freestyle identifies feature edges of interest such as silhouettes and crease lines on the basis of input 3D mesh data in the camera space. The line rendering process can be roughly divided into six steps as follows:

1. Mesh loading: Input scene data is received from Blender and cleaned up.
2. Winged edge building: A graph data structure called winged edges is created for bidirectional traversing of edges.
3. Silhouette detection: Silhouette lines and other feature edges are identified based on winged edges.
4. View map building: A 2D representation of the input scene called view map is created from winged edges.
5. Stroke drawing: A set of stylization options is applied to the identified feature lines by means of Python scripting.
6. Stroke rendering: A raster image of stylized lines is rendered.

Individual rendering steps are implemented in a modular fashion, which permits component-wise performance optimization for space. Attempts made to reduce memory consumption in these rendering steps include:

• Internal switch from double to float in mesh loading and winged edge building.
• Optimization for space in the WXFaceLayer class used for silhouette detection.
• Concerning the mesh importing up to the silhouette detection, several class member variables were removed (at the cost of computing their values when necessary) or changed to a type of smaller memory footprint size.
• Remove unused tessellated forms of silhouettes intended for displaying in the UI.
• Internal switch from double to float in the FEdgeXDetector class (except for curvature computations) and grid data structures used for view map building.
• Avoid float-to-double conversion in the camera-to-grid space transformation.
• Compression of a set of pointers to 3D vectors in grid data structures used for line visibility computations.
• Faster and less memory-consuming stroke rendering by packing stylized strokes into the minimum number of mesh objects.

Some general optimization attempts were also made:

• Remove redundant instantiation of vector objects.
• Avoid copying vertex coordinates where possible.
• Use a pointer to a const char array instead of std::string to avoid string duplicates.

Preliminary performance tests of the documented optimization attempts using three test cases (Figure 1) confirmed no or only negligible visual differences in the rendering results. However, appreciable trade-offs were also observed between memory consumption and rendering time. Specifically, the following portions of the optimized code turned out to be run-time speed bottlenecks:

1. Using std::map (instead of std::vector) to compress a set of pointers to 3D vectors in grid data structures for line visibility computations.
2. Computing 3D bounding box coordinates of triangular faces on the fly when needed (instead of storing the minimum and maximum bounding box coordinates into vectors).

Experimental results below show performance comparisons among (a) the non-optimized baseline version, (b) faster optimized version excluding the time-consuming code changes listed above, and (c) less memory-consuming optimized version including all optimization attempts. Table 1 compares the peak memory consumption in units of megabytes, whereas Table 2 highlights differences in overall rendering time in seconds. All results concerned the full rendering of an input test scene including both Blender Internal and Freestyle render components.

	ID: ichiotsu 74641 vertices, 120592 faces
	ID: ramskull 370252 vertices, 740656 faces
	ID: lily 460330 vertices, 748683 faces

Figure 1. Test cases.

Table 1. Peak memory consumption in megabytes.

Test case	(a) Baseline	(b) Opt. faster	(c) Opt. less memory
ichiotsu	320.33M (100%)	291.45M (91.0%)	273.51M (85.4%)
ramskull	1616.9M (100%)	1409.3M (87.2%)	1309.0M (81.0%)
lily	1550.8M (100%)	1361.8M (87.8%)	1206.0M (77.8%)

Table 2. Overall rendering time in seconds.

Test case	(a) Baseline	(b) Opt. faster	(c) Opt. less memory
ichiotsu	6.93 sec (100.0%)	7.50 sec (108.2%)	7.93 sec (114.4%)
ramskull	35.6 sec (100.0%)	37.0 sec (104.1%)	41.6 sec (117.0%)
lily	14.5 sec (100.0%)	17.6 sec (121.3%)	18.8 sec (129.6%)

The coding results of Freestyle memory consumption optimization will be finalized and released in the forthcoming Blender 2.73 release. Some other performance optimization results have already been merged into Git master, including the optimized rendering of stroke segments (rB40520d549d50) that provides a significant speedup when Freestyle lines contain many small segments, for instance due to an extensive use of line splitting by dashed line patterns and the rectangular “Blueprint” geometry modifier (Figure 2).

Improved artists experience with Freestyle

Recovery of missing features as well as new and improved features were addressed during the project with a view to improved artists experience in Freestyle applications. Key achievements that give artists more control on NPR workflows include:

• Keyframe animation of Freestyle edge/face marks: As part of bug hunting efforts, the functionality of animated edge/face marks was recovered. Animated edge/face marks were possible in pre-BMesh release versions of Blender but not working in more recent versions.
• Freestyle integration into Cycles: In response to frequent requests from Blender artists, the Freestyle line renderer was made available in Cycles (Figures 3 and 4) in addition to the Blender Internal (BI). Most Freestyle options work in the same way for both Cycles and BI. Line textures in Freestyle for Cycles are defined by means of shader nodes, whereas Freestyle for BI instead relies on the conventional texture mapping and influence panels. The availability of Freestyle in Cycles will greatly simplify NPR workflows, without having to use BI merely for Freestyle lines through duplicated scenes and the compositor.
• Per-material Freestyle line colors for BI and Cycles: A new Freestyle-oriented color property named Freestyle line has been introduced in material ID data blocks. It provides users with a reliable way to specify line colors on a per-material basis, no matter what rendering engine is used for Freestyle renders.
• Freestyle line priority at material boundaries: Another per-material Freestyle property named line color priority has also been introduced in order to specify the ordering of competing line colors at material boundaries. A use case of the line color priority is detailed in a Freestyle development blog article (link).

All of the documented improvements and new features above have already been delivered to end users through the Blender 2.71 and 2.72 releases.

General Blender NPR development

Further development of Blender’s NPR capability was pursued in various directions from a more general perspective:

• Stroke textures in Freestyle: A proper support for textured strokes was one of long-awaited missing features that were available in the original stand-alone Freestyle program [7]. This feature was recovered in Freestyle for Blender in close collaboration with coder-artist Paolo Acampora (MAD Entertainment, Italy). Being fully configurable with standard mapping coordinates (e.g., global, view, and along stroke) and mapping targets (diffuse color and alpha transparency through different blending options), Freestyle textured strokes significantly extend the NPR line art capability in Blender (Figure 5).
• Freestyle Python API improvements: Freestyle offers a high programmability of line stylization rules using the Python scripting language. The Freestyle Python API hence plays an important role in increasing the productivity of Freestyle Python scripting in NPR workflows. Throughout the project duration the API underwent major revisions in collaboration with Folkert de Vries in terms of application code readability and run-time performance.
• Freestyle run-time stability improvements: Thanks to steadily incoming reports from Blender artists using Freestyle, the stability of the line rendering engine was greatly improved through fixes of crashing bugs and unwanted visual artifacts.
• Improved support for Windows and internationalization: Because of Windows being the main development environment as well as Japanese being the mother language of the Author, some work hours were also addressed to improvements of Windows support and Blender internationalization. Key contributions include fixes for MinGW compilers (good tools for testing code portability between GCC and Visual C++), Windows IME support for Asian languages, and proper handling of wchar_t file paths on Windows.

Summary statistics

Figure 6 shows the monthly number of commits pushed to Git master during the project. The total commit count amounts to 215.

Conclusion

The present project pursued the improvements of Blender’s NPR rendering capability by focusing on Freestyle run-time performance optimization, better artists experience in interactive Freestyle-oriented workflows, and general technical support for the development of Blender NPR solutions. Cartoon animation and other NPR applications have a solid consumer base in the computer graphics industry, and nowadays Blender has been getting more and more attentions from NPR artists in the open source sector as well as in commercial studios. It is the Author’s sincere hope that the documented project outcome contributes to improved creativity in production NPR workflows with Blender.

Acknowledgements

The Author is grateful for the financial and management supports offered to the project by the Blender Foundation Development Fund. Special thanks go to Ton Roosendaal and Anja Vugts-Verstappen for their dedicated assistance throughout the project duration.