Note: This is an archived version of the Blender Developer Wiki (archived 2024). The current developer documentation is available on developer.blender.org/docs.

User:Hypersomniac/Selection Debug

Selection Debugging with RenderDoc

When experiencing a selection bug, it's preferable to see the selection buffer and debug each calls one by one. Using RenderDoc it can be really simple to spot the error.

Requirement

  • RenderDoc: It is a opensource third party application that records Graphic Library calls in order to help debugging. You can download it here.
  • You need to have compiled blender without ASan (adress sanitizer).
  • Disable continuous viewport update (Viewport sample to 1 in Eevee render settings & Viewport quality to 0 inside Viewport Preferences).

Step By Step

Renderdoc step8.png
  • Open RenderDoc.
  • Go to the "Launch Application" tab.
  • Set the "Executable Path" to the blender binary.
  • Enable "Queue Capture" and set Frame to something like 30 or more depending on the steps you have to do before the capture. (you can also capture using F12 but that will trigger the render operator under default keymap)
  • Launch blender using the launch button and be sure to use selection on the exact frame you told RenderDoc to capture. You should see the frame being capture on the renderdoc overlay.
  • Close blender.
  • In renderdoc, the capture should automatically load if it is a single capture. If that's not the case double click on the capture's thumbnail.
  • Open the "Event Browser" and the "Texture Viewer".
  • The first drawcalls in the Event Browser should be from the selection drawing. Double click one to view the state of the framebuffer after this call. It should look something like what you see in the screenshot.
  • Without going into all the details of graphic debugging, the most important tabs to look at are Texture viewer (to see the final output), Pipeline State (to check state error: i.e. backface culling) and Mesh Output (to detect either vertex shader issues or VBO/IBO issues)

Coloring Shader

Renderdoc final.png

If you are debugging a selection pass that outputs selection IDs, you can view them a bit better by using a custom shader.

  • In the "Texture Viewer" tab, change the "Channels" to "Custom" and enter a custom name in the text field after the RGBA buttons.
  • Hit the create shader button (the green "+" icon) and paste the provided shader in the newly opened text editor. Hit the "Refresh" button.
  • Set the *_START defines accordingly to the offsets used to render the select id pass. You can find them by finding the specific draw calls, double click it, go to "Pipeline State" tab, click "Vertex Shader" or "VS" and finally double click the "Uniforms" row at the bottom.
  • You can further modify the shader from here following renderdoc documentation.

Note: The shader is saved by renderdoc you don't need to create it each time.

Here is the custom shader code:

#version 420 core

/* Change to whatever your face, vert, edge count is.  */
#define FACES_ID_START 1
#define EDGES_ID_START 501 /* Face count + 1 */
#define VERTS_ID_START 1521 /* Face count + Edge count + 1 */

layout (location = 0) in vec2 uv;
layout (location = 0) out vec4 color_out;
layout (binding = 2) uniform sampler2D tex2D;

float integer_noise(int n)
{
	n = (n >> 13) ^ n;
	int nn = (n * (n * n * 60493 + 19990303) + 1376312589) & 0x7fffffff;
	return (float(nn) / 1073741824.0);
}

float linearrgb_to_srgb(float c)
{
	if (c < 0.0031308)
		return (c < 0.0) ? 0.0 : c * 12.92;
	else
		return 1.055 * pow(c, 1.0 / 2.4) - 0.055;
}

float srgb_to_linearrgb(float c)
{
	if (c < 0.04045)
		return (c < 0.0) ? 0.0 : c * (1.0 / 12.92);
	else
		return pow((c + 0.055) * (1.0 / 1.055), 2.4);
}

int decode_id(vec4 id)
{
	uint id_decode = 0u;
	id_decode |= uint(linearrgb_to_srgb(id.x) * float(0xFF) + 0.5);
	id_decode |= uint(linearrgb_to_srgb(id.y) * float(0xFF) + 0.5) << 8u;
	id_decode |= uint(linearrgb_to_srgb(id.z) * float(0xFF) + 0.5) << 16u;
	//id_decode |= uint(id.w * float(0xFF)) << 24u;
	return int(id_decode);
}

void hsv_to_rgb(vec4 hsv, out vec4 outcol)
{
	float i, f, p, q, t, h, s, v;
	vec3 rgb;

	h = hsv[0];
	s = hsv[1];
	v = hsv[2];

	if (s == 0.0) {
		rgb = vec3(v, v, v);
	}
	else {
		if (h == 1.0)
			h = 0.0;

		h *= 6.0;
		i = floor(h);
		f = h - i;
		rgb = vec3(f, f, f);
		p = v * (1.0 - s);
		q = v * (1.0 - (s * f));
		t = v * (1.0 - (s * (1.0 - f)));

		if (i == 0.0) rgb = vec3(v, t, p);
		else if (i == 1.0) rgb = vec3(q, v, p);
		else if (i == 2.0) rgb = vec3(p, v, t);
		else if (i == 3.0) rgb = vec3(p, q, v);
		else if (i == 4.0) rgb = vec3(t, p, v);
		else rgb = vec3(v, p, q);
	}

	outcol = vec4(rgb, hsv.w);
}

void main()
{
	vec4 id_encoded = texture(tex2D, uv);
	int id_decoded = decode_id(id_encoded);

	if (id_decoded >= FACES_ID_START) {
		/* Random coloring ids */
		float hue = integer_noise(id_decoded + 5652881);
		float val = integer_noise(id_decoded + 25647) * 0.5 + 0.5;
		if (id_decoded < EDGES_ID_START) {
			hsv_to_rgb(vec4(hue, 0.9, val * .2 + .1, 1.0), color_out);
		}
		else if (id_decoded < VERTS_ID_START) {
			hsv_to_rgb(vec4(hue, 0.9, val * .5 + .5, 1.0), color_out);
		}
		else {
			hsv_to_rgb(vec4(hue, 0.9, val * .2 + .8, 1.0), color_out);
		}
	}
	else {
		color_out = vec4(0.0);
	}
}