User:Yiming/LANPR GP New Solutions

= LANPR GP New Solutions =

After some discussions on the current status on LANPR, here's what we are gonna do with it:

Mechanism
LANPR will now mainly act as data provider to Grease Pencil Object instead of a separate engine. This way we can benefit from the style functionality that Grease Pencil provides, and don't need to maintain a separate line style module within LANPR. Also, this allows any composition with existing engines like EEVEE or cycles.

Here is how the simplest way it's gonna work:


 * 1) The user activates LANPR. They select GP stroke targets for desired collections and objects.
 * 2) Bake the stroke. LANPR will calculate the strokes and send the data to selected GP targets.

As a lot of things are broken in LANPR when the recent merge happens, so above functionality will be the first priority to achieve.

The next step is to restore real-time preview function.

Currently the plan for this is to hack some GP drawing function to directly accept LANPR data, contrary to drawing using actual GP objects in the scene for strokes. Then we lay the "fake" strokes over the viewport. This way the performance impact will be less significant.

The GP team will try to arrange a meeting with Clement to discuss how go about this live preview GP API at a later date.

Problems

 * 1) We need tagging for GP object when viewport rotates. The way to do it in the depsgraph is still not settled. I'm not familiar with it. Whether to use a notifier or deps tag? Is there any precautions on using copied data from depsgraph?
 * 2) * Sebastian suggested that we need to have protected time interval to prevent too many tagging that slow down the UI. This is good practice.
 * 3) Hacking the GP draw engine to accept LANPR data might not be the best idea. Needed more discussion.

0605
LANPR is running again in a new branch `lanpr-under-gp`(not synced yet), Engine is removed, associated memory operations now moved into Editor's init and deinit.

0607
（汉语用户可以在这里和这里查看进度描述）

Now working on those bugs:


 * Scene should recalculate after modification rather than each frame.
 * [Done] Chaining should always be enabled for GP. Remove "disable" option.
 * [Done] Duplicated points in generated GP strokes. This might be a bit hard due to the nature of how LANPR runs. Maybe more efficient way exist but not it's fast enough that this doesn't have any notable impact to performance.
 * Might induced some "small segment lost" problem.
 * [Done in 0610] Ctrl-Z Problematic.
 * [Done] Grease pencil object data should tag for update.
 * [Done] Normal will invert on the geometry if do a calculation. weird.

0608

 * [Done] merging errors in `readfile.c`.
 * [Done] Occasional crashes in duplicate points detection during chain connection stage.
 * Line layer should have another solution, maybe should put into `CollectionLANPR` and `ObjectLANPR`. Pending research.
 * And current "Add Line Layer" code produce errors.
 * Because we are using GP and it's 3D, so we can just not do occlusion calculation in viewport editing, just use depth occlusion.
 * Provide an option for toggling occlusion calculation (for viewport speed).
 * Don't allow geometry count greater than certain amount being processed in the viewport, value should be configurable.
 * [Done 0610] Deleting GP object but keeping Collection Target selected will result in crash.
 * [Done 0610] Baking isn't working properly.
 * [? See 0625] Also, for baked strokes, need 2 updates to get rid of the wrong ones. Please investigate.

Things to throw into Grease Pencil: (Which means LANPR no longer care about it)


 * Depth controlled style (With nodes)
 * Taper (Thickness modifier improvements)
 * Dot dash presents (again, nodes)

0610

 * [Done] Ctrl-Z Behavior.
 * [Done] Task pool freeing has problem.
 * [Duplicated] Deleting Grease Pencil Object who has already been an selected target cause UI to crash.
 * When above is fixed, use pointer for GP target material and layers as well.
 * [See 0611] Baking function is not threaded, and is interfering with viewport update rendering status.

0611
Me and Antonioya/Sebastian/Mattias are talking about UI and workflow stuff. /
 * [Done?] Baking finished flag not set.
 * [? See 0625] Possible rendering status flag missing somewhere else, after current fix should no longer cause "unable to quit" behaviour, but keep testing.

0625
I'm up again today and let me see if I can sort something out.

We have a pretty decent proposal on GP Modifier design for LANPR. Appears to be easy to port with current structure. Please note that after GP-NPR-modifier change, the old LANPR test files you have will not work under new mechanism. I'm not implementing a transfer.

Regarding remaining tasks listed above:


 * Strokes not deleting may resulted from either the misuse of dg or simply a misplaced stroke flag, but when using modifier, this problem may or may not exist anymore. Will see after implementation.
 * Rendering status flag is messy at the moment, I need to remove "canceling" flag, and implement later when cleaned up code path.

Tasks today:


 * [Done] Hard surface model with boolean operations appears to have wrong line segments. Looking now.
 * [Done] Appears to have been solved, but `max_occlusion` value seems incorrect from GP settings.
 * [Done 0627] Make GP modifier and move line layer and selection stuff there.
 * [Done 0627] With correct file RW.
 * [Done 0627] Pointer RNA for SRC.
 * [Done] Crease threshold should not go to value less than zero.

0627

 * [Done] LANPR rename to LineArt, marcos will be using `LRT`, function names change to `lineart`, struct will be using `LineArt`, compiler option will be using `WITH_LINEART`.
 * [Done] `readfile.c` API changes.
 * Remove target settings in `CollectionLineart` and `ObjectLineart`, and preserve only `usage` flag for now.
 * [Done] Panel display logic and name error.
 * Occlusion level 1/2/3... will have few wrong chain segments here and there, probably due to chain splitting or reduction.

Please note, since LANPR was never in Master, so there won't be a legacy compatible loader for this DNA struct name change. your old LANPR file would need to be reconfigured. (`SceneLANPR` is now `SceneLineArt`)

Solution for auto update:


 * 1) When enable auto update, a special `needs_update` flag is set in `SceneLineart`.
 * 2) When doing frame update, function checks `needs_update` and do the calculations, and use depsgraph to tag all the GP objects that has Lineart modifier, then reset `needs_update`.
 * 3) The GP Object gets tagged will fetch line data from now complete LANPR buffer, and update themselves, cause another scene update event, but due to lack of `needs_update`, when again into frame update function, LANPR will not trigger again to prevent the endless loop.
 * 4) Add relationship between GP objects that has Lineart modifiers between visible objects in scene, then once the object changes, the modifier's `depsgraphUpdate` callback will be called, and in this callback we only set `needs_update` flag in scene so when the next frame update happens, LANPR is running again, providing updated result for those GP objects.

0628

 * [Done] Functional Modifier-driven stroke generation.
 * [Done] Auto update using a separate flag and DG.
 * [Done 0715] Instance collection still not working, might be related to object iterator.
 * Freestyle edge mark is not working again.(If mesh is using modifier)
 * [Done?] Can't append anything. Probably shouldn't use `BLO_expand lineart->target`, need to know how `BLO_expand` works.
 * [Done] Old file with same procedure can't produce line, probably due to line filter process. Caused by line->picked flag not cleared after chaining.
 * [Done] Modifier apply doesn't work.
 * Code naming style fixes.
 * [Done] RNA properties.
 * Function name classification.
 * [Done 0703] Modifier value change should update view.

By using modifier, and due to the way modifiers are evaluated, currently you are not able to first pick lines from an object and then pick it's parent collection's without picking lines from this object twice. We may come up with another solution in the future, but this could also lead to some interesting results. Maybe a switch to enable/disable double picking, such mechanism would be something like LANPR calculator marking object-sourced lines before allowing DG to update the modifier.

0629
Not very likely to do much today... We'll see.


 * [Done 0703] Collection usage mark reportedly not working.

0703

 * [Done] Modifier value change triggers view updates.
 * [Done] Depsgraph tagging for updates when scene changes.
 * [Done] Background calculation should notify the UI about it's completion.
 * [Done] Collection usage mark reportedly not working.

Discovered that `id` and `orig_id` can't be used in a mix. Should be fixed in the future.

Occasional crashes observed during adjusting occlusion level. Later found out that it was caused by a wrong usage flag.

Very few memory leaks when opened a lot of files before closing, but not always occurring, might be related to Line Art internal state.

0704

 * [Done] F12 should re-evaluate related objects and also strokes (blocking, not background updates).
 * [Done] Remove redundant UI.
 * [Done] Master thickness and strength in LineArt panel.

Should change modifier depsgraph relationship to all visible objects instead of scene parameter. Some scene property would cause re-evaluation, do Line Art tagging there.

0705
Just cleaned up some function names.

0707

 * [Done] Line Art panel should show up in all engines.
 * [Done] Don't add frames in GP modifiers.
 * [Done] Process reporting
 * Cursor process: use `WM_cursor_time` to do cursor number, and `WM_progress_set` to set task bar progress in Windows.
 * Use `WM_cursor_set` to set cursor icon but it said there's a reset function but there doesn't seem to have any?
 * [Done] Use `LISTBASE_FOREACH` to do the loop.
 * [Done] Stroke baking panel and functionality.

0714

 * [Done] Bake/Applying reportedly not working. Caused by misuse of `id` and `orig_id` in source reference. Now we all use `orig_id` for consistency.

0715

 * [Done] Baking should return original state.
 * [Need some work] When the source collection is hidden, some calculation is still going on.
 * Currently, I made Line Art calculation return early when no actual geometry is loaded. Maybe a prescan is needed to determine whether run the calculation or not.
 * [Done] Refreshing when properties in the main panel changes.
 * [Done] Overwrite flag isn't effective at the moment.
 * [Done] Exact global back-transform.
 * Alternative method without extensive matrix calculation made it fast enough so this doesn't impact the performance.
 * [Done?] Clicking "Apply" in Line Art modifier will cause lock up when Auto Update isn't turned on. Need investigation.
 * Currently we just skip applying when Auto Update isn't turned on, unless better solution comes up.
 * [Done] Move thickness and strength value into the modifier, and combine the rest few global properties into one panel.
 * [Done] Debug log cleanup.
 * [Done] Remove `LRT_ENABLED` flag.
 * [Done] View layer flag is not working.
 * This is also related to instanced collection. After using `DEG_OBJECT_ITER_FOR_RENDER_ENGINE_BEGIN`, this problem seems to have been resolved.
 * [Done] Only one modifier gets to show the result when baking strokes.
 * This was caused by multiple cleaning of a single frame.

Idea: Set occlusion Only mode implicitly for all non-referred source objects to speed up calculation.

Baking button design not clear yet, three modes may be considered:


 * 1) All physics/simulations/line art at once.
 * 2) All line art modifiers (current, but the modifier was not removed when finished)
 * 3) Bake single line art modifier (for all frames instead of "Applying to" existing frames).

0716
Some discussions has been made with users from a few studios, there are some requirements they brought to the table, some features are easy to achieve:


 * 1) Smooth group border.
 * 2) Configurable material separation lines.
 * 3) Make an mesh object to only produce intersection lines on other objects but not show up in the rendering.
 * 4) Smoother lines.
 * 5) Some sort of semi-automatic LOD for distant objects.

Some features are hard to achieve:


 * 1) Texture alpha or other channel's border. (If not style-wise demanding, do a masked filtering in composition stage can give a "line", but if to generate strokes, this is more problematic)
 * 2) Animating the stroke deformation between two frames on a geometry. (Maybe doing a dynamic boolean and use the intersection line would be OK.)

Other reports including:


 * Chaining doesn't really produce lines that smooth, especially for intersection lines on characters. (Smooth contour modifier by Sebastian would be a lot helpful!)
 * After some tests, I found it's likely due to overlapping threshold in Line Art internal settings, need to find an optimal value to match the numeric feature of the calculated result.
 * F12 and viewport preview has different line thickness. (Which is weird...)

0717 0718
Literally pulling teeth

0721
Creating an document to clarify the code in line art module. Also cleaning up the code. Line Art Code Structure


 * [Done] Function names clean-up. Now in `_lineart_category_operation` format.
 * [Done] Completely remove legacy LineLayer structure..
 * By testing the Bi-plane file, found some very weird bug, like the operator doesn't update intersection checkbox...
 * Turns out it's a layer configuration problem. Suggestion would be in Dope sheet, Grease Pencil mode, make the "+", "-" icons into "layer +", "layer -" to distinguish "add new frame" operation which is basically insert keyframe. In this context it's easy to confuse "add a layer" with "add a frame", because in dopesheet, frame information is probably more dominant.
 * Intersection checkbox not responding is baked strokes being shown with the moving object.
 * Received some request that require Grease Pencil should have its own compositor pass.

Sebastian suggested we keep a "list of desired functionality" here on the wiki so here it is: Desired Functionality

We also discussed how smooth contour algorithm could help achieve better chaining quality. Currently the problem is contour quality and line type switches along a seemingly smooth feature (Line Art explicitly split strokes at these points so that you can select the line types you would like to display).


 * [Done 0722] Providing a switch to allow "chaining intersection lines and contour lines together" would solve some of the problems here.
 * 0722: Experiment found that fuzzy chaining is doable, and it produces quite nice effect over organic models. The problem of fuzzy chaining is that you potentially can not select specifically those "fuzzed" line types, if we allow all types to chain together, you may not be able to select any line types, and intersection line would stay in the object that is first chained in sequence (which is not predictable, but if using the same line style, should be hard to notice).
 * Also: Breaking chains at sharp points should be done at Line Art stage not at the modifier for speed. (Can provide a threshold to select).

0722
Specifically investigated chaining artefacts and its cause. Looks like 32bit floating point data as chained data is just lack a little bit of precision that caused some problems in short segment separation stage. Currently using a increased threshold (1e-5, hard-coded) for detecting overlapped vertex in image space, log didn't make much sense, but will have to now.


 * [Done] Fuzzy chaining options.
 * Also: re-categorize intersection lines. Please note that if there's a floating intersection line which has not being chained onto other contour lines, it can only be selected by collection, because there's no object reference.
 * Some segments are chained wrongly. Overall good enough. Need to print log for specific points to see if the wrong connection is due to missing link in the bounding area.
 * Chaining is accelerated using bounding area method, but no near-area comparator implemented. So two end points could result in very closely visually but be linked to 2 bounding areas thus resulted in not chaining. may optimize this later, but so far it's good enough.
 * [Done] Remove that "Use" text in modifier UI.

0724

 * [Done 0725] After merging with 2.91 the collection tab is gone (used to select collection line art configurations). Need to bring back.
 * [Done 0725] Object one just put it inside the object tab.
 * The usage flag should actually be associated with View Layer rather than Collection/Object. Do this in the futrure.
 * Cursor callback would (very rarely) trigger a X Window Bad Cursor error, not sure the situation on Windows, the API may have some problems (probably is because the process updating being called from another thread).
 * [Done 0725] Chaining bug: Current implementation does not necessarily pick up the nearest point pair when connecting chains. The mechanism is that I didn't check both end-points for nearest connection to other chains.
 * Used a hack to stop chains from propagating once the other end point is closer to the testing line.

Remaining tasks for this stage of development

 * [Done 0728] Limitations on big mesh when allocating memory.
 * [Done 0726] Sampling before outputting stroke data into GPencil modifier to lighten the load on the modifiers.
 * [Done 0729] Angle splitting after chaining, Thus avoid using the modifier.
 * Code style clean-ups and removing obsolete stuff.
 * [Done 0729] Non-standard near clipping still produce some problems. But plan save this for later stages of development as it doesn't really occur that frequently if not specifically triggered.
 * (I'll add when it occurs to me)

--

0725

 * [Done] Chaining loop bug. (For some cases yes, some cases no. Use algorithm below to eliminate the bug)
 * [Implemented] Need a new chain-connection algorithm at the moment, should get closest point pair instead of propagating.
 * [Done] Dual stage chaining should not remove `cre` node from bounding areas.
 * [Done 0726] Should provide a main line type flag when fuzzy chaining option is on, so we can still select line types.
 * [Done 0726] Re-mark line type as contour when the chain is fuzzy.

0726

 * [Done] Pre-sample strokes before finishing the modifier.

Did some performance optimizations: (Thanks to Sebastian's hint of using Hotspot program)


 * 1) Intersection line calculation use geometry bound box instead of image space bound box, quicker and save more memory than BVH.
 * 2) Per thread occlusion task line count set to 1000 instead of 10000, in the future there will be further optimizations for automatic count set. (This works like cycles tile size, it's just Line Art works with a list of line). This made a lot more scenes utilize multi-thread with visible performance gain. (But the major performance drag is still in the intersection generation process，takes like 40-60% of the time)

The LANPR core is already pretty much performance tuned back when it was a stand-alone program using VS's profiler. There are sure still some things I can improve upon.

0727
Idea: For a lot of times we want contour to be thicker than inner feature lines, maybe I can provide an option to adjust the ratio directly so the user don't have to add two modifiers to slow down the calculation.

Discussed about code review plans for this stage of development, at the moment, no new major features would go into Line Art module other than usability and workflow optimizations. Any functional implementations would happen after current code getting reviewed and made into `master`.

0728

 * [Done] Strokes are retained when Line Art master switch (`auto_update`) is disabled.
 * [Done] Memory allocation now allows large mesh (as large as client memory can take).
 * [Done] Quick menu to create line art configurations in one click.
 * [Done Also see 0729] NDC to viewport bug.
 * So this problem is that I have not applied NDC to viewport transformation for Z buffer, hence mapping z from (-1,1) to (0,1). If not having done so, when a result line crosses NDC Z=0 plane, the image to global back transformation formula isn't correct, thus creating crazy exploding lines at NDC Z=0 plane. This problem doesn't exist anymore, but near clipping pre-processing might have some bugs, which are producing erroneous clipped triangles.

0729
More about back transform bug: after re-mapping Z, the problem still exists, that's because (0-1) Z buffer is non-linear, or at least not the same linear ratio as NDC. So I used W instead of Z as distance reference for 2D to 3D back-transform, This time is correct, and won't produce errors observed yesterday. But near clipping still has some slight issues, with lines poking out right on the near plane. Need to take a closer look.
 * [Done] NDC to viewport more bug.

Also Did some code clean-ups.

Found out the reason why that wrong intersection line appearing at the near clipping plane (as adjacent triangle line) is that when clipping, each adjacent triangle gets duplicated shared edge and vertex. but `rl->tr， rl->tl` is correct adjacent data. Using this hack can get rid of the rouge intersection, but the topology is in-fact incorrect. In the future we should have another algorithm that handles this kind of situation.
 * [Not really done] Intersection lines appear on near clipping plane position.
 * So I found out that my current method of clipping triangles simply can't keep correct adjacent data. This function needs to be re-written. Currently a hack is used to dicard rouge lines on clipped triangle pairs.


 * [Done] Long line sometimes not registering in occlusion tasks.
 * This is due to a wrong bounding area priority edge detection method.
 * [Done] Angle splitting before give line results.
 * [Done] Applying modifier doesn't show the calculated result.

Did some memory optimizations:


 * Saved 32 bytes per triangle.
 * Saved 16 bytes per vertex.
 * Saved 4 bytes per edge.
 * Saved 2 bytes per occlusion cut on one edge. (1 byte might be added back for future "transparent window" functionality)

These should have some visible gain on Line Art memory usage.

0803
Did some code clean-ups, updated on https://developer.blender.org/D5442, but this old diff is already filled up with comments and history. I may need to create a new diff. Also, `arc` doesn't seem to work very well with `proxychains`.

0804 - 0809
Mainly code clean-ups.

Let intersection line "belong" to one of the two objects it came from, so that you can select intersection lines that are fully contained within a collection, inter-collection selection is not reliable, we will have this as a future function.

Very short segments in linked mesh may be under-cut (erroneous occlusion data), producing visual artfacts. Now fixed.

When using EEVEE engine, it's scene update code path may interfere with Line Art internal sync state, when rendering animation without baking, it will cause some crashes. Now the problem is fixed.

Memory sync flag used to have small problems, solved now.

0813
Selection of linked collection now fully supported as I implemented a new "instanced" function for finding child objects in link placeholders.

0814

 * [Done] File r/w pointer error.
 * [Done] Garbage lines within the text object.

Added an option to enable/disable duplicated object instances in Line Art, which includes face/vert duplication and particle instances. Toggle this switch if viewport performance is of concern.

Added curve type object support. Now Curves/Surfaces/Text Objects/Metaballs can be rendered in Line Art without converting to mesh.

0816
Added transparency material functions, now you can use up to combination of 8 bits to create transparency masks in Line Art, and select different parts to assign different material/styles. This is particularly useful for "behind the glass" kind of scene, where you can render the glass surface, and at the same time show what's inside.

Fixed some compiler warnings.

0820

 * [Done] Camera shifting would result in wrong image border.
 * This is caused by wrong FOV, shift values also take account of aspect ratio.

Added edge-split capability. Now overlapping edges won't cause flicker or undefined result. This is espetially useful for models that comes from `step` format, where the converter may not take the overlapping edge into account.

Code cleanups.

0821
Added an "intersection only" flag in collection and object, so that you can use them to only generate occlusion lines and it won't affect the final render.

0826
Prevent crashes in linux (`XWindow bad cursor error`) by not showing cursor progress in bakcground thread when there's a g/s/r modal operator running.

Disabled viewport selection of Line Art object by default. So this doesn't interfere with your viewport manipulation.

0827
Prompt on baking finish. If confirm the message, the Line Art is turned off, because all the strokes are already genereated.


 * Should have a 2 option menu, one for "disable Line Art", one for "remove Line Art modifiers", so you can quickly move on to add other modifiers if you want.

Prevent infinite tile splitting in dense triangle scenarios. This makes it good for calculating head-on views of industrial model, where a large amount of faces may align perfectly together.

0828
Feature line compatible near-clipping. No artefacts and all the feature lines that are potentially cut by the near clipping plane is preserved perfectly.

Improved quick canceling performance, now viewport manuverability is very good, and Line Art will always calculate based on latest state of the scene, thus there won't be ghost lines when you move stuff around.

0829
Slope throwaway threshold increased to 1e-9, otherwise would cause K=inf error in line intersecting.

0831
Problems reported from twitter `@AversionReality`, below are some stuff. Note that these are from version 2020-08-26, I just did an update on viewport task cancelling and that version may happened to be based on an unstable commit.


 * "Some crashes". May related to cursor progress reporting, but not very likely because my situation is an `XWindow Bad Cursor` error.
 * Duplicated GP object will stop everything from showing. (Not replicated today)
 * No lines from sub collections. (Not replicated today and using fresh file seems to have solved it)
 * Freestyle edge mark not available when there are mesh modifiers.
 * Possible memory leaks?

Other preferred stuff, some of them are also mentioned in earlier "future functions" page.


 * Baking single frame.
 * Per-object crease.
 * External contour (see below).
 * Excluded object can be included by using the modifier alone so you don't have to create a collection just for that.
 * Material as source (not sure based on current algorithm, may add memory).
 * Vertex group stuff.

Cleaned up some stuff and made a revision here on developer page: https://developer.blender.org/D8758

New line intersection function based on `isect_v2_v2`, appears to be stable. Also fixed `ratiod` usages.

Development logs starting from September is continued in this new page.