Source/Architecture/USD

= Universal Scene Description =

Differential Revision of the first introduction of USD exporting support: https://developer.blender.org/D6287


 * The USD libraries are built by `make deps`, but not yet built by install_deps.sh.
 * Only experimental support for instancing; by default all duplicated objects are made real in the USD file. This is fine for exporting a linked-in posed character, not so much for thousands of pebbles etc.
 * The way materials are exported is going to change to allow instancing of the objects that reference those materials.

Note that this document is to be considered an addition to the USD Exporter chapter in the manual.

The Building Process

 * USD is built as monolithic library, instead of 25 smaller libraries. We were linking all of them as 'whole archive' anyway, so this doesn't affect the final file size. It does, however, make life easier with respect to linking order, and handling upstream changes.
 * The JSON files required by USD are installed into datafiles/usd; they are required on every platform. Set the `PXR_PATH_DEBUG` to any value to have the USD library print the paths it uses to find those files.
 * USD is patched so that it finds the aforementioned JSON files in a path that we pass to it from Blender.
 * USD is patched to have a `PXR_BUILD_USD_TOOLS` CMake option to disable building the tools in its `bin` directory. This is sent as a pull request at https://github.com/PixarAnimationStudios/USD/pull/1048

Abstract Hierarchy Iterator
The core of the USD exporter is the Abstract Hierarchy Iterator. Its goal is to traverse the Objects in the scene, and to produce a hierarchy that is suitable for exporting. This takes care of handling dupliobjects (a.k.a. instances). It iterates over the dependency graph in much the same way as a render engine.

Class Structure
A concrete exporter, like the USD exporter, typically creates the following classes. This example describes a hypothetical Alembic exporter. (Historical note: by now Blender really has this `AbstractHierarchyIterator`-based Alembic exporter, but at the time of writing it did not.)


 * A class `AbstractAlembicWriter` that subclasses `IO::AbstractHierarchyWriter`. This contains all the format-specific code that's common to all the writers.
 * Concrete subclasses of `AbstractalembicWriter` for all different data types to write, like `AlembicMeshWriter`, `AlembicTransformWriter`, `AlembicLightWriter`.
 * A class `AlembicHierarchyIterator` that subclasses `IO::AbstractHierarchyIterator`, which will create the above writers for all objects that need to be exported.

The Export Process
The `IO::AbstractHierarchyIterator` class follows these stages. When multiple frames are exported, this happens for every frame.


 * 1) Determine export hierarchy: the dependency graph is iterated over, and it is determined which objects are exported, and what their parent objects will be in the exported file. It is left to the concrete subclass of `IO::AbstractHierarchyIterator` to perform filters here (like "visible objects only").
 * 2) Determine duplication references: if instancing is enabled, this step determines which object/mesh will be exported as a reference to another object/mesh.
 * 3) Write data: for each to-be-exported Object a concrete `IO::AbstractHierarchyWriter` instance is created. It is left to the concrete subclass, `AlembicHierarchyIterator` in the above example, to create this instance. The instances's `write` method is called to actually write the data. It is only at this point that the writer obtains an `Object *` for the object to be exported. This pointer should not be saved by the writer, as it can potentially change on every call to `write`.

The writer instances are created only once, namely at the first frame the to-be-exported object exists. This is usually the first frame of the export, but objects instantiated from a particle system can be spawned later.

Time
The administration of which timecode to use for the export is left to the file-format-specific concrete subclasses of `IO::AbstractHierarchyIterator`; the abstract iterator itself doesn't know anything about the passage of time. This will allow subclasses for the frame-based USD format and time-based Alembic format.

Meshes
Each UV map is stored on the mesh in a separate primvar. Materials can refer to these UV maps, but these are not yet exported by Blender. The primvar name is the same as the UV Map name. This is to allow the standard name "st" for texture coordinates by naming the UV Map as such, without having to guess which UV Map is the "standard" one.

Face-varying mesh normals are written to USD. When the mesh has custom loop normals those are written. Otherwise the poly flag `ME_SMOOTH` is inspected to determine the normals.

The UV maps and mesh normals take up a significant amount of space, so exporting them is optional. They're still enabled by default, though. For comparison: a shot of Spring (03_035_A) is 1.2 GiB when exported with UVs and normals, and 262 MiB without. We probably have room for optimisation of written UVs and normals.

The mesh subdivision scheme isn't using the default value 'Catmull Clark', but uses 'None', indicating we're exporting a polygonal mesh. This is necessary for USD to understand our normals; otherwise the mesh is always rendered smooth. In the future we may want to expose this choice of subdivision scheme to the user, or auto-detect it when we actually support exporting pre-subdivision meshes.

A possible optimisation could be to inspect whether all polygons are smooth or flat, and mark the USD mesh as such. This can be added when needed.

Particles
Particles are only written when they are alive, which means that they are always visible (there is currently no code that deals with marking them as invisible outside their lifespan).

Particle-system-instanced objects are exported by suffixing the object name with the particle's persistent ID, giving each particle XForm a unique name.

Lights
USD does not directly support spot lights, so those aren't exported yet. It's possible to add this in the future via the UsdLuxShapingAPI. The units used for the light intensity are also still a bit of a mystery.

Sparse writing
The export of animated data to USD is optimised as follows:


 * All data is written to the USD library on every frame.
 * By providing a default value for each property, USD automatically handles the case where that particular property doesn't actually change over time. Even though we write it on every frame, the USD library will write it to file as a static property. This also speeds up import & playback.
 * By using the `UsdUtilsSparseValueWriter` class, the USD library automatically uses runlength encoding for repeating data (so even when it is not equal to the default value).