Extensible Architecture Proposal

The goal of this proposal is to make Blender more extensible and object-oriented. The first step is to make it easier for Blender developers to implement new features and understand the code. If the interface is well-defined and stable it can then later also be exposed to Python or C++. However the focus of this document is only on improving the internal architecture.

Overview

Adding new features often requires changes to many files, as Blender was not initially designed to be extensible. When the codebase was small this worked well, but as it keeps growing it becomes harder and harder to manage.

Operators are an example of a case that works relatively well and was designed to be extensible from the start. Python add-ons can define their own operators that behave just like built-in operators. On the other hand, adding a new datablock requires changes across dozens of files. Modifiers are somewhere in between, they have a fairly well-defined interface, but still require changes across a few files and are not extensible through Python.

Ideally, implementing a datablock, node, modifier or constraint can be done in a single C/C++ file. That would include defining the user interface, DNA, RNA, .blend I/O, dependency graph nodes and relations, and any other interface specific to the data type. Some of those are relatively easy, some require changes to core Blender systems.

We can work towards this ideal incrementally.

Datablocks

There should be an IDTypeInfo interface, similar to for example ModifierTypeInfo. A significant part of this interface already exists, but is implemented through many switch statements on the datablock type. The first step would be to turn that into an IDTypeInfo struct with callback functions. (task done, Documentation)

The next step is identifying more areas in the code where the datablock type is checked, and turning them into new callbacks and flags where appropriate.

.blend I/O

For .blend file I/O, versioning likely will remain centralized. But direct and lib linking should become callbacks in the data type interface, using well documented blenloader API functions.

It may be possible to replace lib linking with lib query callbacks entirely, if those are extended to support deprecated members.

Patch

RNA

RNA is parsed as part of Blender compilation, and then stored in the executable. RNA can also be registered at runtime, but only for cases like operator properties and Python, not for wrapping DNA structures. We can make this either work fully at runtime (and hopefully it doesn't affect startup time too much), or we can find a way to statically process files throughout the source code instead of centralized in makesrna.

We expect runtime registration can be made fast enough, so that would be the first thing to work on. Modifiers and nodes would be a good first step here.

DNA

DNA could also be decentralized, once .blend file I/O and RNA are done.

We could have a way to mark structures as DNA with a macro, and then automatically find them throughout the source code. This still leaves us with our own parsing code which has limitations (like not supporting #defines for arrays sizes).

An alternative solution would be to unify DNA and RNA registration. There would need to be a way to add internal struct members that do not need to be used as RNA properties exposed in the API.

Nodes

Nodes have an interface, but it can be improved. Each node system generally converts the nodes to its own representation, which allows optimizations and scheduling. To avoid the associated code duplication, there are a few solutions.

At least for the new function node system, we should aim to define nodes in a single source and header file from the start. We can make it so that registering a function node in C++ will automatically create the associated C node type definition. Eventually, compositing and texture nodes should also be moved to use the new function system and take advantage of this.

Some things in the node sockets are only defined through Python, like socket types and node categories. There is also some distinction between built-in and custom nodes. This is a mistake, both C/C++ and Python should register the same data type interface, with callbacks being routed to either C/C++ or Python code.

Implementing a shader node in Cycles is complicated, due to the CPU-GPU abstraction and having both OSL and SVM shading systems. We can improve the system here by allowing mixed SVM/OSL nodes execution, and then removing the OSL node implementations. Further, we can automate the packing of SVM nodes, maybe with automatic code generation. Further, the connection to Blender shader nodes could be mostly automated.

Dependency Graph

The dependency graph node and relations creation can also be decentralized, at least per datablock and later more fine-grained as well. Here again, Modifiers are an example of how it can be done.

User Interface

There are many types of UI buttons and layout elements. This could also use a more object-oriented architecture, so that the UI is represented by a hierarchy of widgets with an interface for each widget type.

For modifiers and nodes, we could add a UI drawing callback to draw the properties, rather than having them in Python and drawnode.cc. C code is a bit more difficult to do UI layout in, but these properties are usually relative simple and it keeps the system extensible.