User:Jon Denning/Reports/2022/Experiments

The following is a very quick and dirty write up of an experiment I ran.

notes:

• some of the findings from this experiment will go toward creating new retopo tools.
• i worked on some utility code to help facilitate quick implementation. some of these are seen in the code snippets below. this code will be posted soon
• my primary focus is to create retopo operators, gizmos, etc. secondarily, i want to find the shortcomings, hidden gotchas, missing docs, etc. for doing this type of work.

Useful Notes

Detecting when bpy.context.active_object changes

this was posted by @GaiaClary to #python link

key= bpy.types.LayerObjects, "active";
owner = Object()
bpy.msgbus.subscribe_rna(
  key= key,
  owner=owner,
  args=(""),
  notify=notification_handler,
  options={"PERSISTENT",}
)

Gizmo Matrices

a Gizmo object has several matrices that are used for positioning it for drawing, but the docs are not clear on how they're combined to form the final matrix. digging through the source, here is what i've found:

if(no_scale) {
    final = space * basis * offset
} else {
    // note: scale is a float that is treated as a 3x3 scaling matrix
    scale3x3 = [ scale 0     0     0
                 0     scale 0     0
                 0     0     scale 0
                 0     0     0     1 ]
    if(offset_scale) {
        final = space * basis * scale3x3 * offset
    } else {
        final = space * basis            * offset * scale3x3
    }
}

Detecting Changes Made Outside Operator

use bpy.app.handlers.depsgraph_update_post to register callbacks that can detect if some mesh data has changed. ex: if depsgraph.id_type_updated("MESH") returns True, then a mesh has been updated. it's still very crude, as cannot tell how the mesh was altered or even which mesh was changed, but it could be useful.

• changing co and select of vertex in Python does cause depsgraph to update
• note: changing normal of vertex does NOT trigger a depsgraph update. i wonder what all else does or does not. should changing vertex normal cause depsgraph update? also, ops to flip normals and recalculate normal do not trigger depsgraph update.

if operator makes changes to mesh, a simple deterministic is to set a flag (ex: IJustMadeAChange) when operator makes a change, then reset it every frame. in the depsgraph callback, check the flag to see if the change was due to our operator or something else. another approach is in the handler method of SnapWidgetCommon class in scripts/addons/mesh_snap_utilities_line/widgets.py. in there, a test checks the name of the last operator performed (i.e., last_operator = context.window_manager.operators[-1]) against the set of our operator names.

Detecting When Tool Becomes (Un)Selected

there seems to be no standard way to know when a tool becomes selected or unselected (without active polling).

one way by aburdin (stack exchange answer) is to add a NOP GizmoGroup to the WorkSpaceTool, since GizmoGroup objects know when they are selected. below is a copy-paste from the stack exchange answer.

class MYADDONNAME_TOOL_mytool(bpy.types.WorkSpaceTool):
    bl_idname = "myaddonname.mytool"
    bl_space_type='VIEW_3D'
    bl_context_mode='OBJECT'
    bl_label = "My tool"
    bl_icon = "ops.transform.vertex_random"
    bl_widget = "MYADDONNAME_GGT_mytool_activated"

class MYADDONNAME_GGT_mytool_activated(bpy.types.GizmoGroup):
    bl_label = "(internal)"
    bl_space_type = 'VIEW_3D'
    bl_region_type = 'WINDOW'
    bl_options = {'3D'}

    @classmethod
    def poll(cls, context):
        return True

    def setup(self, context):
        print("My tool activated!")

    def __del__(self):
        print("My tool deactivated!")

bpy.utils.register_class(MYADDONNAME_GGT_mytool_activated)
bpy.utils.register_tool(MYADDONNAME_TOOL_mytool, separator=True)

Call Flows

documentation and examples around Operators, Macros, Gizmos, etc. do not explain well the order of method / function calls. below are my attempts at diagramming the call flow.

note: i have not spent much time tweaking layout, readability, etc. these digraphs are generated directly from an online graphviz editor, Edotor. the original source behind graphs are embedded in the image link.

Operator

Below is a rough call-flow diagram of some of the functions around Operator. click the image to see the original source behind the diagram and to inspect the diagram better.

The docs are not clear on...

• what happens when invoke or execute return {"RUNNING_MODAL"} without registering the operator (ex: context.window_manager.modal_handler_add(self)).
• what happens when invoke or execute return {"PASS_THROUGH"}
• how multiple modal operators running at the same time work (ex: what exactly happens if modal returns {"PASS_THROUGH"})
• what happens if one operator (modal or not) invokes/executes another operator (modal or not)
• what happens if there are several operators are running modal, the top operator returns RUNNING_MODAL, and a lower operator returns FINISHED??
• what happens if an operator returns a combination of return values in the set (ex: {'PASS_THROUGH', 'FINISHED'})?

Caveats:

• poll -> __init__ and calls to __del__ are not quite correct. poll determines if an operator is valid in context. when Blender is drawing the UI, the UI code will specify to draw some operator. Blender will need an instance of that operator in order to call draw, but not if calling poll. if poll returns True, then Blender will call draw on operator. these drawn operators "stick around", so that when they are interacted with later (ex: clicked), Blender can call either invoke or execute. and this is especially important, because the code that drew the operator can set properties on the drawn operator that will affect the invoking or executing. in the end, however, this diagram is good enough to understand the general call flow. some unclear items:
  • when is an operator deleted (and __del__ is called)? ex: is the operator created every UI redraw? if operator is part of menu, is it created when menu is opened and then deleted when menu is closed?
  • call order of __init__ and poll?
  • call order of these methods when operator is executed from script / console (no drawing or invoking)?
• an operator is not removed from modal operator stack unless modal returns {'FINISHED'}

Macro

• docs are not clear on how macros work with modal operators

Questions

• what is GizmoGroup.mode? in context_mode_check() of mesh_snap_utitilies_line/widgets.py, there is a test of tool.mode == context.mode. but as far as i can tell, GizmoGroup.mode can only be empty set or {'DEFAULT'}. either way, i have no idea what's going on here.
• what are GizmoProperties and GizmoGroupProperties used for?
• why are there specialized collections (Gizmos, BMVertSeq, etc.) rather than using bpy_prop_collection (link)?
• functions in bmesh.ops invalidate references? (tried: bmesh.ops.bisect_edges)

Retopo Translate

I tried to duplicate the bpy.ops.transform.translate operator but with specific snapping settings set. ideally, the adjustment of snapping settings are made only temporarily. also, only absolutely necessary changes should be made to other Blender settings, scene, etc., if any at all are needed. finally, the operator should act like any other built-in operator (able to cancel, no visible indication of or delay from switching operators, able to undo, does not push excessively to undo stack, etc.).

Create a New Scene with Correct Snapping Settings

as the bpy.context.tool_settings is based on the scene, we could create a linked scene with the correct settings, and then switch between them.

not good.

Just Set It and Forget It

simply setting the snapping settings and then calling the operator works fairly well and requires very little code. however, this blows away all of the previous snapping settings, which isn't ideal.

class Retopology_Snap_Translate_JustSet(Operator):
    bl_idname = 'retopology.snap_translate_justset'
    bl_label = 'Translate with face project snapping (Just Set!)'
    def execute(self, context):
        ToolSettings.just_set(ToolSettings.project_face())
        bpy.ops.transform.translate('INVOKE_DEFAULT')
        return {'FINISHED'}

Macro

using bpy.types.Macro and Macro.define to chain together multiple operators is very quick to do and requires very little code (ignoring the additional operators for setting and resetting snapping settings). however, if one of the operators is cancelled, then there's no way reset the snapping settings. also, adjusting the operator's parameters after leaving modal will reapply transformation without snapping.

(while this doesn't work too well for snapping the translation, i think it might work well enough for other retopo tools)

class Retopology_Snap_Translate_Macro(Macro):
    bl_idname = 'retopology.snap_translate_macro'
    bl_label = 'Translate with face project snapping (Macro)'
    bl_options = {'REGISTER', 'UNDO', 'MACRO'}  # not sure about needing 'MACRO'

    @classmethod
    def registered(cls):
        cls.define('RETOPOLOGY_OT_snap_faceproject')
        cls.define('TRANSFORM_OT_translate')
        cls.define('RETOPOLOGY_OT_snap_reset')

Temporary Set

another approach is to set directly the settings, then call the operator, then reset the settings after (ex: using with). again, this doesn't allow for changing the operator parameters after leaving modal and still having the geometry snapped.

class Retopology_Snap_Translate_TempSet(Operator):
    bl_idname = 'retopology.snap_translate_quickchange'
    bl_label = 'Translate with face project snapping (Temp Set)'
    def invoke(self, context, event):
        with snap_faceproject():
            bpy.ops.transform.translate('INVOKE_DEFAULT')
        return {'FINISHED'}

Wrap in Another Modal Operator

one approach is to wrap the operator in another modal operator, where the snapping settings are set when started, the operator is called, then once the operator returns, reset the snapping settings in the modal callback. this really is just a sloppier and hackier version of the temp set option above, with the same drawbacks plus a small stutter. not good.

note: i also experimented with using _bpy.ops.call to call the "operator" directly and tried to understand bpy.ops.* and bpy.types.* better. I tried to see what it would take to extend some of the built-ins, but i didn't get too far.

# operator wrapping function / factory
def create_retopology_operator(op, label, toolsettings_args, *, bl_description=None, bl_options=None, register=True):
    from _bpy import ops as _ops

    op_category, op_pyname = op.split('_OT_')
    op_category = op_category.lower()

    class Retopology_Operator(Operator):
        bl_idname = f'retopology.{op_category}_{op_pyname}'
        bl_label = f'Retopology {label}'
        #bl_description = f'{description}'
        bl_options = _ops.get_bl_options(op)

        @classmethod
        def poll(cls, context):
            return _ops.poll(op)

        def invoke(self, context, event):
            # SET UP
            self.ts = ToolSettings(toolsettings_args)

            # call op
            C_dict = None
            kwargs = {}
            C_exec = 'INVOKE_DEFAULT'
            C_undo = False
            _ops.call(op, C_dict, kwargs, C_exec, C_undo)

            context.window_manager.modal_handler_add(self)
            return {'RUNNING_MODAL'}

        def modal(self, context, event):
            # TEAR DOWN
            self.ts.restore_all()

            return {'FINISHED'}

    Retopology_Operator.__qualname__ = f'retopology.{op_category}.{op_pyname}'
    Retopology_Operator.__name__ = f'retopology.{op_category}.{op_pyname}'
    #Retopology_Operator.__doc__ = f'Retopology. {op.__doc__}'

    if not register: return Retopology_Operator
    return registerable(Retopology_Operator)

create_retopology_operator(
    'TRANSFORM_OT_translate',
    'Translate with face project snapping (Wrapper)',
    ToolSettings.project_face(),
    bl_description='Translate with face project snapping (Wrapper)',
)

A bpy.context.temp_override_tool_settings Method

I created a temp_override_tool_settings Context method that works similarly to the temp_override Context method (note: temp_override does *not* work to temporarily override tool_settings, as it is not specially handled and temp_override does not recurse). it allows the scene's tool settings to be stashed, overwritten, then restored. i took a couple attempts at this. while I think this work could be useful, I'm going to pause/abandon this work for now. this approach does not allow for tool parameters to be modified after leaving modal with snapping.

Add Additional Snapping Arguments

This version requires making changes to the C code. in particular, adding extra params to the transform operator. this option does everything I need it to do, except that snapping is enabled if snap=True is passed. still working on finding a workaround for this.

As far as duplicating the translate operator, this option is by far the best. it doesn't push additional and unnecessary actions to undo stack. it allows for adjustment of operator parameters after leaving modal mode.

in my patch, I also exposed the snapping settings in the operator parameter panel.

class Retopology_Snap_Translate_Args(Operator):
    bl_idname = 'retopology.snap_translate_args'
    bl_label = 'Translate with face project snapping (Arguments)'
    def execute(self, context):
        bpy.ops.transform.translate(
            'INVOKE_DEFAULT',
            snap=True,
            snap_elements={'FACE'},
            use_snap_project=True,
        )
        return {'FINISHED'}

GeoPen Tool

Here are some notes from working on a basic geopen tool

• When extending bpy.types.Macro, ...
  • any bug in Operator in the macro causes a crash, but the macro continues on calling the next operator.
  • if any of the modal operators are cancelled, the macro stops execution not sure about this actually... see note on modal operators
    • it might be nice to have a callback for success and cancellation (similar to Gizmo().exit

• BMesh issues
  • no method to select all / deselect all. must loop through all geometry and set manually
  • no method for knowing how many or which verts / edges / faces are currently selected. must loop through all geometry

• Face projection (original face snapping method):
  • Geometry that does not project to a face is simply translated (as if snapping is turned off). how to handle this situation?
    • use face nearest as fallback? does this make sense? worth making this change regardless of using for handling this case?
    • modify face snapping to remember last successful snapping location, and use that if unable to project? (could add option to enable this)
    • modify face snapping to remember last successful snapping location, and translate off that location (instead of original location) if unable to project? this seems less useful

• keymap
  • impossible to have create-new-geometry alongside select-geometry where both use LMB (for example). adding a keyboard modifier to create-new-geometry will bleed into and interfere with the translate followup modal operator.
    • possibly add additional args that cause translate to ignore modifiers??? if artist wants constraints or precision, they can switch to a translate tool for those options?
  • forcing artists to switch tools or use shortcut to select geometry is non-ideal.

modal operators

i tried creating a Macro that chained together an immediate operator (add new vertex), a modal operator (translate), and another immediate operator (select), but the final select operator was not getting called, but i'm not entirely sure why. the reason for the need to chain is to that the tool always creates a new vertex, but that new vertex is bridged to existing geometry based on selection (selection is very obvious and clear, and it's exposed through UI already; as opposed to using a tag, property, or other marking; artist can quickly set up context).

i think the reason for this behavior is that Macros will blaze through the operators, and invoking a modal operator will call the invoke method right away, but modal will happen in next frame (lazy first call).

geopen hiccups

as of 2022.05.23, I have a working prototype for geometry pen (tweet with video), but there are several "hiccups" in the design.

• the creation and translation of vertices both push to undo, which means that artists must undo twice. it seems calling bpy.ops.transform.translate('EXEC_DEFAULT', False, value=(0,0,0)) pushes to undo stack even though i've specified the False to prevent this. I'll need to ask some questions on #blender-coders or dig in the source.

• after selecting a vertex when nothing is selected, Blender does not register that the mouse cursor is now hovering a Gizmo. the standard Move tool does work, but mine does not.

• when Vertex and/or Edge snapping is enabled, the grabbed vertex will snap to vertices and/or edges of the source (edited) mesh (which is correct) and of the target (non-edited) mesh (which might not be correct). presently, there is no way to have Vertex/Edge snapping work on source mesh while having Face Project snapping work on target.

• knifing into existing geometry does work if auto merge / auto split are enabled, but only after committing to grab (does not merge/split with initial creation of vertex). this is a decent behavior for most work, but there are times when it is cumbersome. (perhaps an actual knife is appropriate)

• presently, Blender's auto merge and auto split are combined as one property, auto_merge_and_split.

• newly created vertices do not have a normal (actually, they have zero normal) if they aren't connected to a face. should the face project snapping set their normal to be that of the normal of face at projection? right now, i'm assuming that the normal should be pointing toward the view. this assumption works well even if the artist is working on targets with inconsistent or inward pointing normals, but it doesn't work well if artist is working on the backside of the target.

• there is no pre-viz of action right now, so the artist will know what will happen only after they take the first action. i've experimented with hiding / showing individual Gizmo objects in a GizmoGroup, but more work needs to happen. i do have concerns about performance... see next note

• i'm iterating through all verts and edges of edit mesh to test for selection. ideally, this would happen in C/C++ (new feature), or i could use the selection history (not sure this actually works). as the target gets larger, this O(Nv + Ne) operation can impact performance.

• geometry can be moved off the target. this is non-ideal. should probably make Face Nearest snapping be fall-back if the vert cannot project to a face.

• there is no way to turn off/on snapping while in the middle of grab.

• Vertex and Edge snapping only works correctly if there is one vertex grabbed. if a grabbed edge is moved, then the source "target" is the only thing that is snapped, and only if the mouse cursor is hovering the vertex/edge. an ideal snapping method would be to allow individual vertices to snap to nearby vertices. an example of another ideal snapping method that presently isn't possible: if the grabbed edge is hovering another source edge, the verts of grabbed edge snap to the nearest verts of hovered edge, allowing for quickly merging two separated faces (as an example).

• sometimes, vertex snapping does not work when working on vertices that are connected to the grabbed vertex via an edge or two.

• right now, auto merging and splitting happens based on world-space distance. this really should be screen-space distance instead. vertex and edge snapping is all done in screen space, so this disconnect could result in different behaviors (in screen-space the geo is too far apart to snap, but in world-space they are close enough to merge, so the geometry is surprisingly merged after releasing grab even though visually it wasn't snapped).

• vertex and edge snapping distances are not exposed anywhere.

Gizmos and WorkSpaceTools

Currenly, there is no real documentation for the Gizmo, Gizmos, GizmoGroup, GizmoProperties, GizmoGroupProperties, and WorkSpaceTool classes. There are a few small examples that come with Blender and some archived docs (Custom Manipulators), but none of these go into much detail. I poked the #blender-coders channel, and Falk David (@filedescriptor), Jesse Y (@deadpin), and JulianEisel (@julianeisel) filled in some of the missing details. I plan to write up a document with more extensive examples, but below are the highlights in a rough form.

Gizmos

• used by GizmoGroup as a very simple collection of Gizmo objects. this is similar to BMesh having a BMVertSeq containing a bunch of BMVert objects.
• not really needed or used other than through GizmoGroup().gizmos

GizmoGroup

• a logical grouping of Gizmo objects ("all these Gizmo objects do / work on related things")
• responsible for setting up each of the Gizmo objects
• can be visualized / rendered:
  • at any point if not associated with a WorkSpaceTool (whenever its poll returns True). if the GizmoGroup is not rendered, none of its Gizmo objects will be rendered. individual Gizmo objects can opt to not render
  • (optionally) when an associated WorkSpaceTool is active. effectively, the WorkSpaceTool wraps the poll method of GizmoGroup, returning False if associated WorkSpaceTool is not active. If active, poll returns whatever the GizmoGroup.poll returns.

Gizmo

• can be thought of as a visualized, interactive version of Operator (calls other Operator objects or manipulates Operator parameters using target_set_ methods.
• a single instance of Gizmo can belong to exactly one GizmoGroup. if 2+ GizmoGroup objects need a Gizmo, they each need to create their own instances of the Gizmo.
• do not have a poll method, but visualization and interaction of Gizmo can be controlled by using the hide property or by performing NOP in the draw and draw_select methods.

WorkSpaceTool

• the widget property should really be named gizmogroupname or something similar. the "widget" term is an older term ("manipulator" is another).
• a WorkSpaceTool can have exactly one GizmoGroup (set at start up)

GizmoProperties and GizmoGroupProperties

• similar to Gizmos class, just a collection of properties.
• documentation is extremely limited here

To Do

• rename "widget group" to "gizmo group" in WindowManager.gizmo_group_type_ensure and WindowManager.gizmo_group_type_unlink_delayed
• rename "widget" to "gizmo group name" in WorkSpaceTool.widget
• rename GizmoGroup examples to use "GizmoGroup" instead of "WidgetGroup"
• rename Mesh.total_face_sel to Mesh.total_polygon_sel (there are several uses of face)
• check that total_*_sel is always correct (change selection w/o toggling edit mode) and update docs if not
• add use_snap_selectable to bpy.ops.transform.translate
• make snap on bpy.ops.transform.translate not change tool settings!
• add ability to snap to vertex of active/edit but not non-edit
• add option for updating normal of snapped geometry
• add screen-space auto merge option?
• separate auto_merge_and_split arg for bpy.ops.transform.translate into auto_merge and auto_split
• polybuild bugs
  • Ctrl+LMB does not respect snapping settings!
  • Ctrl visualizations does not respect object's matrix_world
  • holding Cmd does not have previz like holding Ctrl
• calling with bpy.ops.transform.translate with INVOKE_DEFAULT will ALWAYS push to undo stack, even if False is passed to prevent undo push