User:Hypersomniac/Selection Debug

=Selection Debugging with RenderDoc= When experiencing a selection bug, it's preferable to see the selection buffer and debug each calls one by one. Using RenderDoc it can be really simple to spot the error.

Requirement

 * RenderDoc: It is a opensource third party application that records Graphic Library calls in order to help debugging. You can download it here.
 * You need to have compiled blender without ASan (adress sanitizer).
 * Disable continuous viewport update (Viewport sample to 1 in Eevee render settings & Viewport quality to 0 inside Viewport Preferences).

Step By Step

 * Open RenderDoc.
 * Go to the "Launch Application" tab.
 * Set the "Executable Path" to the blender binary.
 * Enable "Queue Capture" and set Frame to something like 30 or more depending on the steps you have to do before the capture. (you can also capture using F12 but that will trigger the render operator under default keymap)
 * Launch blender using the launch button and be sure to use selection on the exact frame you told RenderDoc to capture. You should see the frame being capture on the renderdoc overlay.
 * Close blender.
 * In renderdoc, the capture should automatically load if it is a single capture. If that's not the case double click on the capture's thumbnail.
 * Open the "Event Browser" and the "Texture Viewer".
 * The first drawcalls in the Event Browser should be from the selection drawing. Double click one to view the state of the framebuffer after this call. It should look something like what you see in the screenshot.
 * Without going into all the details of graphic debugging, the most important tabs to look at are Texture viewer (to see the final output), Pipeline State (to check state error: i.e. backface culling) and Mesh Output (to detect either vertex shader issues or VBO/IBO issues)

Coloring Shader
If you are debugging a selection pass that outputs selection IDs, you can view them a bit better by using a custom shader.


 * In the "Texture Viewer" tab, change the "Channels" to "Custom" and enter a custom name in the text field after the RGBA buttons.
 * Hit the create shader button (the green "+" icon) and paste the provided shader in the newly opened text editor. Hit the "Refresh" button.
 * Set the *_START defines accordingly to the offsets used to render the select id pass. You can find them by finding the specific draw calls, double click it, go to "Pipeline State" tab, click "Vertex Shader" or "VS" and finally double click the "Uniforms" row at the bottom.
 * You can further modify the shader from here following renderdoc documentation.

Note: The shader is saved by renderdoc you don't need to create it each time.

Here is the custom shader code: