User:Jon Denning/Reports/2022/Experiments

The following is a very quick and dirty write up of an experiment I ran.

notes:


 * some of the findings from this experiment will go toward creating new retopo tools.
 * i worked on some utility code to help facilitate quick implementation. some of these are seen in the code snippets below. this code will be posted soon
 * my primary focus is to create retopo operators, gizmos, etc. secondarily, i want to find the shortcomings, hidden gotchas, missing docs, etc. for doing this type of work.

= Useful Notes =

Detecting when `bpy.context.active_object` changes
this was posted by @GaiaClary to #python link

`Gizmo` Matrices
a `Gizmo` object has several matrices that are used for positioning it for drawing, but the docs are not clear on how they're combined to form the final matrix. digging through the source, here is what i've found:

Detecting Changes Made Outside Operator
use `bpy.app.handlers.depsgraph_update_post` to register callbacks that can detect if some mesh data has changed. ex: if `depsgraph.id_type_updated("MESH")` returns `True`, then a mesh has been updated. it's still very crude, as cannot tell how the mesh was altered or even which mesh was changed, but it could be useful.


 * changing `co` and `select` of vertex in Python does cause depsgraph to update
 * note: changing `normal` of vertex does NOT trigger a depsgraph update. i wonder what all else does or does not.  should changing vertex normal cause depsgraph update?  also, ops to flip normals and recalculate normal do not trigger depsgraph update.

if operator makes changes to mesh, a simple deterministic is to set a flag (ex: `IJustMadeAChange`) when operator makes a change, then reset it every frame. in the depsgraph callback, check the flag to see if the change was due to our operator or something else. another approach is in the `handler` method of `SnapWidgetCommon` class in `scripts/addons/mesh_snap_utilities_line/widgets.py`. in there, a test checks the name of the last operator performed (i.e., `last_operator = context.window_manager.operators[-1]`) against the set of our operator names.

Detecting When Tool Becomes (Un)Selected
there seems to be no standard way to know when a tool becomes selected or unselected (without active polling).

one way by aburdin (stack exchange answer) is to add a NOP `GizmoGroup` to the `WorkSpaceTool`, since `GizmoGroup` objects know when they are selected. below is a copy-paste from the stack exchange answer.

Call Flows
documentation and examples around Operators, Macros, Gizmos, etc. do not explain well the order of method / function calls. below are my attempts at diagramming the call flow.

note: i have not spent much time tweaking layout, readability, etc. these digraphs are generated directly from an online graphviz editor, Edotor. the original source behind graphs are embedded in the image link.

Operator
Below is a rough call-flow diagram of some of the functions around `Operator`. click the image to see the original source behind the diagram and to inspect the diagram better.



The docs are not clear on...


 * what happens when `invoke` or `execute` return `{"RUNNING_MODAL"}` without registering the operator (ex: `context.window_manager.modal_handler_add(self)`).
 * what happens when `invoke` or `execute` return `{"PASS_THROUGH"}`
 * how multiple modal operators running at the same time work (ex: what exactly happens if `modal` returns `{"PASS_THROUGH"}`)
 * what happens if one operator (modal or not) invokes/executes another operator (modal or not)
 * what happens if there are several operators are running modal, the top operator returns `RUNNING_MODAL`, and a lower operator returns `FINISHED`??
 * what happens if an operator returns a combination of return values in the set (ex: `{'PASS_THROUGH', 'FINISHED'}`)?

Caveats:


 * `poll` -> `__init__` and calls to `__del__` are not quite correct. `poll` determines if an operator is valid in context.  when Blender is drawing the UI, the UI code will specify to draw some operator.  Blender will need an instance of that operator in order to call `draw`, but not if calling `poll`.  if `poll` returns `True`, then Blender will call `draw` on operator.  these drawn operators "stick around", so that when they are interacted with later (ex: clicked), Blender can call either `invoke` or `execute`.  and this is especially important, because the code that drew the operator can set properties on the drawn operator that will affect the invoking or executing.  in the end, however, this diagram is good enough to understand the general call flow.  some unclear items:
 * when is an operator deleted (and `__del__` is called)? ex: is the operator created every UI redraw?  if operator is part of menu, is it created when menu is opened and then deleted when menu is closed?
 * call order of `__init__` and `poll`?
 * call order of these methods when operator is executed from script / console (no drawing or invoking)?
 * an operator is not removed from modal operator stack unless `modal` returns `{'FINISHED'}`

Macro

 * docs are not clear on how macros work with modal operators

= Questions =


 * what is `GizmoGroup.mode`? in `context_mode_check` of `mesh_snap_utitilies_line/widgets.py`, there is a test of `tool.mode == context.mode`.  but as far as i can tell, `GizmoGroup.mode` can only be empty set or `{'DEFAULT'}`.  either way, i have no idea what's going on here.
 * what are `GizmoProperties` and `GizmoGroupProperties` used for?
 * why are there specialized collections (`Gizmos`, `BMVertSeq`, etc.) rather than using `bpy_prop_collection` (link)?
 * functions in `bmesh.ops` invalidate references? (tried: `bmesh.ops.bisect_edges`)

= Retopo Translate =

I tried to duplicate the `bpy.ops.transform.translate` operator but with specific snapping settings set. ideally, the adjustment of snapping settings are made only temporarily. also, only absolutely necessary changes should be made to other Blender settings, scene, etc., if any at all are needed. finally, the operator should act like any other built-in operator (able to cancel, no visible indication of or delay from switching operators, able to undo, does not push excessively to undo stack, etc.).

Create a New Scene with Correct Snapping Settings
as the `bpy.context.tool_settings` is based on the scene, we could create a linked scene with the correct settings, and then switch between them.

not good.

Just Set It and Forget It
simply setting the snapping settings and then calling the operator works fairly well and requires very little code. however, this blows away all of the previous snapping settings, which isn't ideal.

Macro
using `bpy.types.Macro` and `Macro.define` to chain together multiple operators is very quick to do and requires very little code (ignoring the additional operators for setting and resetting snapping settings). however, if one of the operators is cancelled, then there's no way reset the snapping settings. also, adjusting the operator's parameters after leaving modal will reapply transformation without snapping.

(while this doesn't work too well for snapping the translation, i think it might work well enough for other retopo tools)

Temporary Set
another approach is to set directly the settings, then call the operator, then reset the settings after (ex: using `with`). again, this doesn't allow for changing the operator parameters after leaving modal and still having the geometry snapped.

Wrap in Another Modal Operator
one approach is to wrap the operator in another modal operator, where the snapping settings are set when started, the operator is called, then once the operator returns, reset the snapping settings in the modal callback. this really is just a sloppier and hackier version of the temp set option above, with the same drawbacks plus a small stutter. not good.

note: i also experimented with using `_bpy.ops.call` to call the "operator" directly and tried to understand `bpy.ops.*` and `bpy.types.*` better. I tried to see what it would take to extend some of the built-ins, but i didn't get too far.

A `bpy.context.temp_override_tool_settings` Method
I created a `temp_override_tool_settings` Context method that works similarly to the `temp_override` Context method (note: `temp_override` does *not* work to temporarily override `tool_settings`, as it is not specially handled and temp_override does not recurse). it allows the scene's tool settings to be stashed, overwritten, then restored. i took a couple attempts at this. while I think this work could be useful, I'm going to pause/abandon this work for now. this approach does not allow for tool parameters to be modified after leaving modal with snapping.

Add Additional Snapping Arguments
This version requires making changes to the C code. in particular, adding extra params to the transform operator. this option does everything I need it to do, except that snapping is enabled if `snap=True` is passed. still working on finding a workaround for this.

As far as duplicating the translate operator, this option is by far the best. it doesn't push additional and unnecessary actions to undo stack. it allows for adjustment of operator parameters after leaving modal mode.

in my patch, I also exposed the snapping settings in the operator parameter panel.

= GeoPen Tool =

Here are some notes from working on a basic geopen tool


 * When extending `bpy.types.Macro`, ...
 * any bug in `Operator` in the macro causes a crash, but the macro continues on calling the next operator.
 * if any of the modal operators are cancelled, the macro stops execution not sure about this actually... see note on modal operators
 * it might be nice to have a callback for success and cancellation (similar to Gizmo.exit


 * BMesh issues
 * no method to select all / deselect all. must loop through all geometry and set manually
 * no method for knowing how many or which verts / edges / faces are currently selected. must loop through all geometry


 * Face projection (original face snapping method):
 * Geometry that does not project to a face is simply translated (as if snapping is turned off). how to handle this situation?
 * use face nearest as fallback? does this make sense?  worth making this change regardless of using for handling this case?
 * modify face snapping to remember last successful snapping location, and use that if unable to project? (could add option to enable this)
 * modify face snapping to remember last successful snapping location, and translate off that location (instead of original location) if unable to project? this seems less useful


 * keymap
 * impossible to have create-new-geometry alongside select-geometry where both use LMB (for example). adding a keyboard modifier to create-new-geometry will bleed into and interfere with the translate followup modal operator.
 * possibly add additional args that cause translate to ignore modifiers??? if artist wants constraints or precision, they can switch to a translate tool for those options?
 * forcing artists to switch tools or use shortcut to select geometry is non-ideal.

modal operators
i tried creating a `Macro` that chained together an immediate operator (add new vertex), a modal operator (translate), and another immediate operator (select), but the final select operator was not getting called, but i'm not entirely sure why. the reason for the need to chain is to that the tool always creates a new vertex, but that new vertex is bridged to existing geometry based on selection (selection is very obvious and clear, and it's exposed through UI already; as opposed to using a tag, property, or other marking; artist can quickly set up context).

i think the reason for this behavior is that `Macro`s will blaze through the operators, and invoking a modal operator will call the `invoke` method right away, but `modal` will happen in next frame (lazy first call).

geopen hiccups
as of 2022.05.23, I have a working prototype for geometry pen (tweet with video), but there are several "hiccups" in the design.


 * the creation and translation of vertices both push to undo, which means that artists must undo twice. it seems calling `bpy.ops.transform.translate('EXEC_DEFAULT', False, value=(0,0,0))` pushes to undo stack even though i've specified the `False` to prevent this.  I'll need to ask some questions on #blender-coders or dig in the source.


 * after selecting a vertex when nothing is selected, Blender does not register that the mouse cursor is now hovering a `Gizmo`. the standard Move tool does work, but mine does not.


 * when Vertex and/or Edge snapping is enabled, the grabbed vertex will snap to vertices and/or edges of the source (edited) mesh (which is correct) and of the target (non-edited) mesh (which might not be correct). presently, there is no way to have Vertex/Edge snapping work on source mesh while having Face Project snapping work on target.


 * knifing into existing geometry does work if auto merge / auto split are enabled, but only after committing to grab (does not merge/split with initial creation of vertex). this is a decent behavior for most work, but there are times when it is cumbersome.  (perhaps an actual knife is appropriate)


 * presently, Blender's auto merge and auto split are combined as one property, `auto_merge_and_split`.


 * newly created vertices do not have a normal (actually, they have zero normal) if they aren't connected to a face. should the face project snapping set their normal to be that of the normal of face at projection?  right now, i'm assuming that the normal should be pointing toward the view.  this assumption works well even if the artist is working on targets with inconsistent or inward pointing normals, but it doesn't work well if artist is working on the backside of the target.


 * there is no pre-viz of action right now, so the artist will know what will happen only after they take the first action. i've experimented with hiding / showing individual `Gizmo` objects in a `GizmoGroup`, but more work needs to happen.  i do have concerns about performance... see next note


 * i'm iterating through all verts and edges of edit mesh to test for selection. ideally, this would happen in C/C++ (new feature), or i could use the selection history (not sure this actually works).  as the target gets larger, this O(Nv + Ne) operation can impact performance.


 * geometry can be moved off the target. this is non-ideal.  should probably make Face Nearest snapping be fall-back if the vert cannot project to a face.


 * there is no way to turn off/on snapping while in the middle of grab.


 * Vertex and Edge snapping only works correctly if there is one vertex grabbed. if a grabbed edge is moved, then the source "target" is the only thing that is snapped, and only if the mouse cursor is hovering the vertex/edge.  an ideal snapping method would be to allow individual vertices to snap to nearby vertices.  an example of another ideal snapping method that presently isn't possible: if the grabbed edge is hovering another source edge, the verts of grabbed edge snap to the nearest verts of hovered edge, allowing for quickly merging two separated faces (as an example).


 * sometimes, vertex snapping does not work when working on vertices that are connected to the grabbed vertex via an edge or two.


 * right now, auto merging and splitting happens based on world-space distance. this really should be screen-space distance instead. vertex and edge snapping is all done in screen space, so this disconnect could result in different behaviors (in screen-space the geo is too far apart to snap, but in world-space they are close enough to merge, so the geometry is surprisingly merged after releasing grab even though visually it wasn't snapped).


 * vertex and edge snapping distances are not exposed anywhere.

= Gizmos and WorkSpaceTools =

Currenly, there is no real documentation for the `Gizmo`, `Gizmos`, `GizmoGroup`, `GizmoProperties`, `GizmoGroupProperties`, and `WorkSpaceTool` classes. There are a few small examples that come with Blender and some archived docs (Custom Manipulators), but none of these go into much detail. I poked the #blender-coders channel, and Falk David (@filedescriptor), Jesse Y (@deadpin), and JulianEisel (@julianeisel) filled in some of the missing details. I plan to write up a document with more extensive examples, but below are the highlights in a rough form.

`Gizmos`

 * used by `GizmoGroup` as a very simple collection of `Gizmo` objects. this is similar to `BMesh` having a `BMVertSeq` containing a bunch of `BMVert` objects.
 * not really needed or used other than through `GizmoGroup.gizmos`

`GizmoGroup`

 * a logical grouping of `Gizmo` objects ("all these `Gizmo` objects do / work on related things")
 * responsible for setting up each of the `Gizmo` objects
 * can be visualized / rendered:
 * at any point if not associated with a `WorkSpaceTool` (whenever its `poll` returns `True`). if the `GizmoGroup` is not rendered, none of its `Gizmo` objects will be rendered.  individual `Gizmo` objects can opt to not render
 * (optionally) when an associated `WorkSpaceTool` is active. effectively, the `WorkSpaceTool` wraps the `poll` method of `GizmoGroup`, returning `False` if associated `WorkSpaceTool` is not active.  If active, `poll` returns whatever the `GizmoGroup.poll` returns.

`Gizmo`

 * can be thought of as a visualized, interactive version of `Operator` (calls other `Operator` objects or manipulates `Operator` parameters using `target_set_` methods.
 * a single instance of `Gizmo` can belong to exactly one `GizmoGroup`. if 2+ `GizmoGroup` objects need a `Gizmo`, they each need to create their own instances of the `Gizmo`.
 * do not have a `poll` method, but visualization and interaction of `Gizmo` can be controlled by using the `hide` property or by performing NOP in the `draw` and `draw_select` methods.

`WorkSpaceTool`

 * the `widget` property should really be named `gizmogroupname` or something similar. the "widget" term is an older term ("manipulator" is another).
 * a `WorkSpaceTool` can have exactly one `GizmoGroup` (set at start up)

`GizmoProperties` and `GizmoGroupProperties`

 * similar to `Gizmos` class, just a collection of properties.
 * documentation is extremely limited here

= To Do =


 * rename "widget group" to "gizmo group" in WindowManager.gizmo_group_type_ensure and WindowManager.gizmo_group_type_unlink_delayed
 * rename "widget" to "gizmo group name" in WorkSpaceTool.widget
 * rename `GizmoGroup` examples to use "GizmoGroup" instead of "WidgetGroup"
 * rename `Mesh.total_face_sel` to `Mesh.total_polygon_sel` (there are several uses of face)
 * check that `total_*_sel` is always correct (change selection w/o toggling edit mode) and update docs if not
 * add `use_snap_selectable` to `bpy.ops.transform.translate`
 * make `snap` on `bpy.ops.transform.translate` not change tool settings!
 * add ability to snap to vertex of active/edit but not non-edit
 * add option for updating normal of snapped geometry
 * add screen-space auto merge option?
 * separate `auto_merge_and_split` arg for `bpy.ops.transform.translate` into `auto_merge` and `auto_split`
 * polybuild bugs
 * `Ctrl+LMB` does not respect snapping settings!
 * `Ctrl` visualizations does not respect object's `matrix_world`
 * holding `Cmd` does not have previz like holding `Ctrl`
 * calling with `bpy.ops.transform.translate` with `INVOKE_DEFAULT` will ALWAYS push to undo stack, even if `False` is passed to prevent undo push