Source/Architecture/Asset System/Asset Indexing

= Asset Indexing =

Asset indexing provides an optimized way to load asset representations into the asset system. With asset indexing, users have to spend significantly less time waiting for asset libraries to be loaded and ready to use.

Information about the asset like its asset meta-data (link), name and data-block type is extracted from asset files and stored in asset index files. Asset indexing can observe the content of asset libraries to perform minimal updates to the asset index files.

The asset index files are stored in a local cache of the user's operating system.

Glossary
Glossary used within the scope of this page.


 * Asset
 * Data-blocks that have been tagged by users to be an asset.


 * Asset File
 * Blend file which is located inside an asset library.


 * Asset Library
 * Entry point and container for asset representations and asset catalogs. https://wiki.blender.org/wiki/Source/Architecture/Asset_System/Back_End#Asset_Libraries


 * Asset Index File
 * JSON file containing information limited to selecting and browsing about assets contained inside a single asset file.


 * Asset Catalog
 * Catalogs where assets can be organized in.


 * Asset representation
 * An asset representation holds the necessary information about an asset so that the asset system can work with it, and the user interact with it as per design, without requiring the represented asset data to be available. It holds information like the asset’s name, traits and other asset metadata. Asset representations are organized into asset libraries. source.


 * File Browser
 * Editor in blender used for navigating file system and finding data-blocks in blend files. Used for linking, appending, file open, import, save and asset browsing functionalities.


 * Data-block
 * manual https://wiki.blender.org/wiki/Source/Architecture/ID/ID_Type

Problem statement
Assets are stored in a blend file as part of an asset library. In order to browse through the assets of an asset library Blender has to parse each asset file in order to know which data-blocks have been tagged as assets. Doing this each time when browsing assets and having large asset libraries extracting the assets can take a long time.


 * Blender loads asset files sequentially in a background thread, which is a slow operation.
 * Asset libraries can be located on network shares and version controlled repositories.
 * Asset libraries can have a large number of files.
 * Asset files can be large is size.
 * Asset files can be compressed.
 * Asset files may not actually contain assets.

Solution
To reduce the time needed to browse asset libraries we want to extract the assets representation from the asset files and store them in the asset index. When the asset file isn’t changed, the asset index file would contain the required information needed to be able to reconstruct the asset representation.

Requirements
Each asset file has an asset index file.

Each asset file has an asset index file even when the asset file has no assets. The reason is that asset files can be part of an asset library without containing assets. For example when an asset file contains materials which are linked from assets inside a different asset file, but those materials are not assets themselves. If there was no index, the file would be parsed every time.

Icons of assets are not part of the asset index file.

Compared to the rest of the asset representation, icons can use a lot of disk space. Blender already has an icon manager that utilizes the OS to store and cache icons. Therefore we chose not to include the icon inside the asset representation.

Empty asset files can be detected without opening the asset index file.

Opening, reading and parsing asset index files has latency. Detecting if there are actually asset representations stored inside the asset index file after parsing isn’t efficient. By checking the file size on disk, the asset indexer can already check if there are any asset representations inside the asset index file, removing the need for opening, reading and parsing the file.

The asset index is implemented as a file indexer. Browsing assets shares logic with the file browser editor. File browser uses file indexers to read data from folders on a file-system, but also to list data-blocks located inside blend-files or data-blocks from asset libraries.

The usage of the File browser for asset listing and  is planned to be refactored. (See )

Asset library loading process


The sequence diagram shown above describes an overview control flow of the asset indexer. When the asset browser needs to know the asset representations that are inside an asset library it uses a. The file list walks over the file inside the asset library and when an asset file is detected it invokes  on the. returns if the found asset representations or that the asset index file need to be updated for the given asset file.

If the asset file index requires an update, the file list extracts the asset representations from the asset file and invokes. Update entries would create update or create the asset index file with the given asset representations.

Asset Index
When browsing assets, asset indexes are created when they don’t exist or are out of date. The asset browser does not access asset files directly, but uses asset indexes.

Storage
The asset index files are stored in the local cache directory. Within the local cache directory the indexes are stores using the following pattern:.


 * Hash of the absolute file path of the asset library.
 * Hash of the absolute file path of the asset library.


 * Hash of the absolute file path of the asset file.
 * Hash of the absolute file path of the asset file.


 * Filename of the asset file. Not used by Blender, but is added for discoverability convenience.
 * Filename of the asset file. Not used by Blender, but is added for discoverability convenience.

Content

 * the version of the asset index file. It is used to identify the structure of the content. Asset indexes with a different version than used by Blender would be regenerated. Blender 3.1-3.4 expect version attribute to be .). Later versions might require to change it.
 * the version of the asset index file. It is used to identify the structure of the content. Asset indexes with a different version than used by Blender would be regenerated. Blender 3.1-3.4 expect version attribute to be .). Later versions might require to change it.


 * list of assets in the asset file related to the asset index.
 * list of assets in the asset file related to the asset index.


 * ID name of the asset, including its ID type prefix..
 * ID name of the asset, including its ID type prefix..


 * UUID of the catalog associated with the asset.
 * UUID of the catalog associated with the asset.


 * Name of the catalog associated with the asset.
 * Name of the catalog associated with the asset.


 * Human readable description of the asset.
 * Human readable description of the asset.


 * Information about the author of the asset.
 * Information about the author of the asset.


 * List of tags associated with this asset.
 * List of tags associated with this asset.


 * List of ID Properties associated with this asset.
 * List of ID Properties associated with this asset.

Performance
Performance is optimized for reading and parsing asset indexes.


 * Asset libraries can have blend files that doesn’t contain any assets. To reduce opening and parsing an asset index that doesn’t contain any entries the size on disk of the asset index is checked. Asset indexes smaller than  would not be parsed and assumed to contain no entries.
 * Asset indexes older than asset files are assumed to be out of date and should be regenerated. This can be determined without opening the asset index.
 * Asset indexes of asset files on network shares are stored on your local system to reduce latency. Although this could depends on how your OS has been configured.

Future expected changes
This section will contain a table with planned tasks and remark how it would change the asset indexing. To be added