User:Sergey/UndoStackImprovement

Current implementation

In current undo system implementation we've got:

• Global undo stack where full copy of scene is storing
• Local undo stacks in several spaces (edit mode, text space, sculpt mode, image space). Suck stacks are storing only changes, made to specific space so each undo step needs much less memory than global undo step

There are some confusing things caused by such approach. Here are some examples:

• Stroke in image editor calls global push, but when you're trying to undo stroke when your mouse is out of image space no changes would be made to image (but undo node would be pop-ed out from global stack)
• Edit text in text space and move object just after this. Then undo moving. Text will disappear
• You're unable to undo changes made to modifiers' stack when you're in edit or sculpt mode.
• And I suppose there would be more mess when you'll try to work simultaneously in different spaces which has got their own stacks -- there would be quite difficult to predict behavior of undo operator.

Suggested solution

I've got an idea of expanding a bit global undo stack. It could store nodes of different type with specification of it's type, pointer to undo function and some data which will be passed to undo function:

struct UndoNode {
  short type;
  (void*) handler(void *data);
  void *data;
} UndoNode;

Maybe some kind of additional context specification is necessary to be stored in this structure.

Behavior

• When global undo push is necessary new UndoNode would be created and type would be something like UN_GLOBAL and data will store MemFile, handler will be a pointer to function which will load this MemFile
• When local push is necessary new UndoNode would be created with type like UN_LOCAL, handler would be a pointer to function which loads local data (this function will be different for different type of spaces) and data will contain everything needed for local undo (UndoMesh/UndoCurve/etc for push from edit mode, SculptUndoNode for push from sculpt mode and so on)
• When undo operation is necessary handler from the stack pointer would be called with data stored in this node. (Btw, pointer to current undo node will be necessary in undo stack, but this is a nuance)
• And finally, before executing operator, which could be undone, there would be necessary to ensure latest node in undo stack is global and make global push otherwise. So there would be no problems with text disappearing, latest action will be always undone even if you've moved mouse out of space area.

Postscript

Here is description of idea -- I haven't tried to implement this yet s maybe there will be some small changes in logic.

Maybe this isn't a perfect solution, but maybe in discussing this idea we'll find perfect one and then implement it.