User:Elubie/AssetBrowser

From BlenderWiki

Jump to: navigation, search
Note: This is an archived version of the Blender Developer Wiki. The current and active wiki is available on wiki.blender.org.

Asset Browser Design Document

Introduction

Library linking is a greeat feature of Blender, though I have the feeling that many artists don't use it to the full extent possible since the current workflow can certainly be improved. The fact that there is demand for a better solution im Blender, is that there have been already several attempts to include a Material Library and a Proposal for an Asset Manager has been made already.


For the first cut I would propose to start with quite a simple solution since I expect that suggestions will be more substantial once an initial implementation is present.

Current Situation

Problem 1: Linking Workflow

Currently library linking is sometimes quite tedious to use. It also puts quite some organizational burden on the Artist who needs to remember in which file everything is saved and where this file is.

The typical workflow for linking to an external ID block is the following:

  1. open file browse (file -> append/link)
  2. navigate to directory
  3. select .blend file and select data block(s) to link
  4. Hit enter or click on the Append/Link button

Solution Idea:

  • Asset Browser to quickly link in Objects or Assets from pre-defined .blend files or asset collections.

Problem 2: Editing Linked Library Data

When using a linked data block, it is not possible to quickly make changes to that data block. For example an animator links in a character and discovers a bad deformation, so he wants to quickly edit the weight painting. To do this he currently has to go into the file where the character geometry and the vertex groups are defined.

Solution Ideas:

  • Start new Blender?
  • Allow opening multiple .blend files with one Blender process running?

Open Questions:

  • How could this be done?
  • Is this possible through a new Blender bContext?

Problem 3: Additional Steps After Linking

During a larger production it is sometimes necessary to link in a series of groups and even python files for a certain character or set. Sometimes also Python files need to be executed after the linking has been done. This puts a large burden on the Artists as well, since they need to remember exactly which groups they need to include and which python scripts they need to execute.

Solution Idea:

  • Allow definition of Asset like in ReferenceDesk, this links additional py scripts and executes them.
  • Allow additional user defined dependencies? This would probably make it a bit more general, though we still would need to distinguish whether to exectute the script in case of a Python script.

Problem 4: Dependency Management

In a larger production it can get difficult to keep track where all the data that is used in a shot comes from. There are character .blend files, sets etc. and a character can link in materials, textures etc. from other .blend files. One issue for example is how to know which files to re-render if a certain object is changed. Another issue that is currently difficult is finding dead links that result from renaming an asset. Also renaming an ID block could lead to broken links if it is referenced from another file.

Comments:

  • This needs overview of all production files.
  • Debatable if this isn't really better done by an external tool?

Solution Ideas:

  • BlenderAid is one attempt to solve this.
  • BlenderAid does this by indexing a production folder, maybe this could be done by indexing an AssetList?
  • The question that needs to be answered here is 'Where is my Object, Group, Rig etc. used?'


Scope

To keep the scope and development effort reasonable for a first implementation, I propose to only support functionality initially that is already present in Blender as far as linking and appending is concerned. This means for the *initial* implementation:

  • no changes (except maybe bugfixes if found) are going to happen to how ID blocks are linked.
  • no fixes to the proxy system are also out of scope of the initial implementation
  • no dependency management yet like BlenderAid does

The main goal should be to make it easy to set up a library of assets and access them. This means linking the corresponding ID data blocks into your current scene file.

Definitions

Asset
I haven't really found a comprehensive and clear definition what an asset should be. From a practical standpoint I assume that an asset can be anything that the artist might want to re-use as an entity, independent of other things. This means that I prefer to keep the design flexible and open to allow fine grained assets, such as textures or materials and larger assets like whole sets or characters as well.
AssetCollection
An AssetCollection is a list of assets. The AssetCollection is writen to an AssetCollection file, which is a text file that contains the list of assets. (Likely in JSON format). The AssetCollection can also contain a list of .blend files, in which case the whole .blend file is seen as an 'Asset'.

Scenarios

Sharing Asset Collections in a larger Project:

When working on a larger project, the Artists can create one or several asset collections. A project manager or dedicated artist can then for example link to these asset collections from a central .blend file and thus generate a central point from which assets can be linked in. When the artist create a new .blend file and want to use any asset, they can just link in the asset collection(s) from that central .blend file.

Sharing Asset Collections for a larger Audience:

A nice idea that we could support for collections of .blend files would be better sharing of re-usable content. In the case of an Open Movie Project, there could be a central .blend file that contains the AssetCollection for the whole project. This would give people wanting to re-use for example characters from the move an easy way to access them. This would also be a nice way to show what content is there for re-use.

Artist Using Asset Collections :

To use an asset, the artist would link in the AssetCollection from a central .blend file and would then through the Asset Browser be able to quickly link in for example the prop “SintelKnife” without having to care that the object is actually linked in from weapons.blend.

Proposal

Workflow

Adding an Existing Asset Library

  1. Artist links in an AssetCollection datablock from an existing .blend file that defines the AssetCollection.
    1. File -> Append/Link ...
    2. Browse to .blend file containing the asset collection
    3. Select AssetCollection and hit 'Enter' or the 'Link/Append' Button
    4. Open Asset Browser Space
    5. Search for the desired asset
    6. Drag&Drop into 3D scene or hit 'Link' button.

Creating a New Asset Library

  1. From New Library File:
    1. Menu: Add → Asset Library → New Library.
    2. File Browser opens, you can select place and name of the new asset file. This will create a new AssetLibrary file.
  1. From Existing Library File:
    1. Menu: Add → Asset Library → From Asset File
    2. File Browser opens, you can select place and name of the new asset file – this parses a file (maybe JSON) and adds the assets from this file to the AssetCollection.

Adding a New Asset

  1. For Objects and Groups we can have a menu entry/Operator:
    1. Object → Create Asset From Object → Select Asset Library to add new Asset to
    2. Object → Create Asset From Group → Group Names → Select Asset Library to add new Asset to.
  2. For Materials, Textures etc. we might bring back the old “DataBrowse” mode and add an Operator to create an Asset for the selected ID block.
  3. Adding a New .blend File or Directory
    1. In the Asset Browser left Panel, a new .blend file can be added to the list
  4. Adding a new Directory to be scanned for .blend files
    1. In the Asset Browser left Panel, a new directory can be added to the list
  5. In the Asset Browser, the selected asset properties can be edited.


User Interface

Elubie asset browser ui v1.png


Implementation Notes

  • For the asset browser a new SpaceType is created, a mockup is shown above.
  • AssetCollection would be an ID block
  • The user can also link in an AssetCollection from his startup.blend, so he always has his favourite asset collections easily available
  • Updating an asset or asset collection could be done from the Property Panel (right panel) of the AssetBrowser Space.
  • For this we need to write back the asset library file, which needs to be synchronized if several instances of Blender are open (for example by checking if file has changed before writing to it).
  • This could possibly be improved with using a database, although when using SVN a text file probably is preferred to merge changes to the asset library from several artists.
  • Appending/Linking by Drag&Drop from the Asset Browser or with an Operator Button
  • Searching:
    • Search for an asset by name
    • Search for an asset by tag

Further Ideas:

  • Export AssetCollection together with all .blend files (and referenced external files like textures)?
    • copy all .blend files, preserving relative file structure
    • fix all links in the .blend files (make all links relative) - this is something that BlenderAid can do already
  • Add and remove assets from an AssetCollection
  • Check AssetCollection for consistency (do all .blend files exist?)
  • Pack AssetCollection into .blend file like packed images
  • make access to .blend files transparent regarding the protocol used. It should not matter whether the .blend file is read from local file system, ftp, http, svn etc.
  • Being able to save data to an external .blend file, like saving all materials in the current .blend file to an existing materials.blend
  • Export an Asset into a separate .blend file
    • Create temporary Main, link Asset into this Main
    • make everything local?
    • write this to new .blend file
  • Create an Asset for everything in the .blend file? For the whole .blend file?
  • ...

Data Structures

UML Diagram:

Elubie asset browser uml.png

Data Structures in C code: 

struct AssetCollection {
    ID id;
    ListBase assets; /* struct Asset */
    ListBase asset_files; /* struct AssetFile */
    ListBase asset_dirs; /* struct AssetDir */
    char filepath[256]; /* external asset collection file */
    struct PackedFile packedfile; /* if 0, AssetCollection is loaded from file */ --> compare with  packed images
}
 
struct AssetItem {
    char name[64];
    char lib[256];
    short instance_group;
}
 
struct AssetScript {
    char name[64];
    char lib[256];
    short execute;
}
 
struct AssetTag {
    char tag[128];
}
 
struct AssetFile {
    char filepath[256]
    char description[256];
    ListBase ListBase tags; /* AssetTags */
}
 
struct AssetDir {
    char dirpath[256]
    char description[256];
    ListBase ListBase tags; /* AssetTags */
}
 
struct Asset {
    ListBase items; /* AssetItem */
    ListBase scripts; /* AssetScript */
    ListBase tags; /* AssetTags */
    char description[256];
}
 
struct Main {
    ...
    ListBase gpencil;
    ListBase movieclip;
    ListBase asset_collections; 
}

Possible example of asset collection external file with JSON: 

{ "TubeProjectAssets": {
    "AssetList" : {
        "Gilgamesh" : {
            "items" : [ { "lib":"lib/models/chars/gilgamesh.blend" , "IDname":"gilga_HI",  "instance_group":True } , 
                        { "lib":"lib/models/chars/gilgamesh.blend" , "IDname":"gilga_PROX",  "instance_group":False } ,
                        { "lib":"lib/models/chars/gilgamesh.blend" , "IDname":"gilga_FACP", "instance_group":False }
                      ],
            "scripts" : [ { "lib":"lib/models/chars/gilgamesh.blend", "IDname":"tuberigUI.py", "exec":False },
                          { "lib":"lib/models/chars/gilgamesh.blend", "IDname":"__gilgamesh_post.py", "exec":True }
                      ],
            "tags" : [ "Characters", "Tube"],
            "description" : "This is the main character of the Tube project"
        },
        "Gilga Roach Close" : {
            "items" : [ { "lib":"lib/models/chars/gilga_roach.blend" , "IDname":"Roach_HI", "instance_group":True } 
                      ],
            "scripts" : [ { "lib":"lib/models/chars/gilgamesh.blend", "IDname":"tube_roach_rigui.py", "exec":False },
                          { "lib":"lib/models/chars/gilgamesh.blend", "IDname":"__groach_post.py", "exec":True }
                      ],
            "tags" : [ "Characters", "Tube"],
            "description" : "This is the main character of the Tube project"
        }
    },
    "FileList" : [
        { "file" : "lib/interior/furniture.blend",
          "tags" : ["Furniture", "Interior"],
          "description" : "This is a collection of furniture for Interior Shots"
        },
        { "file" : "lib/interior/windows.blend",
          "tags" : ["Windows", "Interior"],
          "description" : "This is a collection of windows for Architecture"
        },
    ]
  }
}

Open Questions:

  1. How do we create previews. We need to load a library block, parse DNA etc. without actually linking it to the current .blend file. And we need to probably define a preview scene that can be used to render those previews? Currently we can only show previews for Materials, Lamps, Textures and World since these can store a preview image in the .blend file. For the other ID types the preview is not yet created and stored.
  2. What functionality can/should we provide to actually manage the dependencies between different .blend files.
  3. How should we handle dependencies to external files like texture images, scripts etc...
  4. How should a nice UI for searching and grouping by tags look like?
Retrieved from "https://wiki.blender.org/index.php/User:Elubie/AssetBrowser"