From BlenderWiki

Proposal for the Development of Blender's Non-Photorealistic Rendering Capabilities

Author

Tamito Kajiyama

Date

5 March 2014

Overview

The development work presented in this proposal by the Author aims to enhance non-photorealistic (NPR) rendering capabilities of the Blender 3D computer graphics creation program. Development objectives acknowledge that Blender encompasses a variety of rendering functionalities applicable to the production of non-photorealistic (e.g., cartoon) still images and animation films, most remarkably including the Blender Internal (BI) ray tracer and the Freestyle line rendering engine recently integrated into Blender. In view of NPR scenes foreseen in the Gooseberry open feature film project (2014-2015) managed by the Blender Institute in support of the EU Media Fund, the NPR rendering capabilities of Blender deserve thorough enhancements focusing on real-world production workflows for unconventional NPR visualization styles.

Being a geometry-based line rendering system, Freestyle identifies feature edges of interest such as silhouettes and crease lines (both visible and hidden) on the basis of input 3D mesh data in the camera coordinate system. Feature edge detection represents a time- and memory-consuming process of geometry computations, whose time and space complexity is a function of the input data size. The detected feature edges are then transformed into stylized strokes in the 2D image space based on user-defined line stylization operations. Line styles are specified either through the built-in graphical user interface offering a fixed set of interactive style options (the Parameter Editor mode) or in the form of Python scripts referred to as style modules (the Python Scripting mode). Freestyle is currently available as part of the BI rendering pipeline through scene render layers and the compositor.

The scope of the proposal is the improvements of the BI and Freestyle rendering engines in terms of performance and users experience. Specifically, the following development objectives are pursued throughout the project:

1. Freestyle performance improvements in terms of rendering time and memory consumption.
2. Better artists experience during interactive line stylization exercises.
3. Technical support for the development of Blender NPR rendering features.

Tasks

The proposed project will address the following coding tasks throughout the project duration.

Task 1: Freestyle performance improvements

Memory consumption

Relatively high memory consumption in the Freestyle rendering process is a known performance bottleneck that makes Freestyle less applicable to large scene setups with a high polygon count. Input mesh data (i.e., a set of vlak instances) is first transformed into a data structure called winged edges that are bidirectional edges of a graph used for mesh topology analysis. The winged edges are then transformed into view edges that comprise a data structure named a view map used for line stylization by means of Python scripting. The following refactoring ideas are evaluated to reduce the memory consumption issue:

• There is some data redundancy inherent to the geometry computations, since the winged edges and the view map are different representations of the input mesh data. At the moment input mesh vertices are duplicated many times (at least once for a winged edge and once more for a view edge). An idea here for less memory consumption is to replace copies of the same vertex with a reference to it.
• Freestyle uses double precision floating-point numbers to represent vertices of winged edges and view edges. Recalling that Blender uses single precision floating-point numbers for the mesh data representation, using the same precision in Freestyle may lead to a significant reduction of memory usage without a major loss of visual quality.

Computation time

Line rendering with Freestyle requires a large amount of geometry-based computations. Two major compute-intensive steps are silhouette edge detection and line visibility testing. In principle these computations are performed on an edge-by-edge basis. Hence multi-threading could be applicable to boost the geometry analysis steps on multi-core shared-memory machines. Either OpenMP or POSIX threads will be considered to implement a parallelized version of the silhouette edge detection and line visibility testing.

Task 2: Freestyle workflow improvements

With the aim to improve artists' experience of NPR rendering workflows using Freestyle, the following topical elements (listed in the order of priority) are worked out:

• A frequently asked feature is a Freestyle pass to store RGBA pixels of rendered strokes, instead of merging them into the Combined pass as of this writing. There are both pros and cons of the requested feature. An upside is easier extraction of rendered Freestyle lines for post-processing in the compositor. To achieve the same effect, at the moment it is necessary to create a render layer with all Include toggles turned off except for the Include Freestyle switch, which some artists found bothering. A downside is a potential design issue in future Freestyle development due to a lot of duplicated passes. It is noted that the Freestyle pass if implemented will be a sibling of the Combined pass because both store RGBA pixels. Although Freestyle currently does not generate any other image channels such as Z depth and object/material index, Freestyle-specific passes of these channels could be implemented in the future in line with the Freestyle RGBA pass. This design problem will be addressed through careful discussions with Blender artists and core developers.
• View map caching is implemented to allow quicker style parameter tweaks by artists. At the moment Freestyle recalculates the view map from scratch to visualize the effects of altered stylization options even if the input mesh data is unchanged. The view map caching will significantly reduce the overall turn-around time for stylization trials and errors.
• A line style preview window is implemented to help artists identify the most preferable style settings more quickly.

Task 3: Support for Blender NPR capability development

Besides the aforementioned Freestyle-specific key tasks, general technical support is provided for the development of Blender NPR rendering functionalities as much as time permits. Specifically:

• Attempts will be made to address reported Freestyle bugs assigned to the Author. It is remarked that most problem reports have already been evaluated and left for future work since a proper solution is likely to take a long time.
• Aware of ongoing efforts on the Blender extended expressive rendering (BEER) proposal aiming at extended BI capabilities for NPR applications, coordinated joint work with the BEER development team is expected.
• Bug fixes and code review in general are foreseen through the Blender project management system (developer.blender.org).

Planning and Deliverables

The development work is scoped for 3 months from April 2014. The results of the planned coding tasks will be delivered in the form of Git revisions maintained by the Blender code repository (git.blender.org). New features will be documented through updates of the Blender User Manual (wiki.blender.org), as well as by release notes if applicable. The following milestone (MS) reports will be submitted to the Blender developer blog (code.blender.org) for general audience to inform of new features and functionality updates:

• MS1. Mid-term report, "Development of Blender's non-photorealistic rendering capabilities", 15 May 2014.
• MS2. Final report, "Enhanced Freestyle and Blender Internal functionalities for non-photorealistic animation film production", 30 June 2014.

Task 1 "Freestyle performance improvements" will be addressed until MS1, while Task 2 "Freestyle workflow improvements" will be executed after the completion of Task1 and no later than MS2. Task 3 "Blender Internal enhancement for NPR rendering" will be pursued throughout the project duration.

Appendix: Future work

Additional ideas for further Freestyle development are presented in a random order, with the aim to help evaluate a future project extension upon the completion of the present project.

• A better way for users to interact with Freestyle parameter controls within the compositor (e.g., immediate updates of stroke rendering results after parameter tweaks, without time-consuming view map re-calculation).
• Face-to-face and line-to-face intersections for feature edge detection. Some discussion of this topic has been done in the bf-blender-npr mailing list.
• Adding Z depth information to rendered Freestyle lines.
• Support for vertex groups in feature edge selection and line stylization.
• Presets for line stylization parameters.
• Built-in SVG export.
• Inclusion of curves and grease pencil strokes as feature edges.