User:Ankitm/GSoC 2020/Weekly Reports

Week 12 August 17 - 21

 * Fix overlapping nodes in the nodetree.
 * Vertex normals:
 * Parse vertex normals: vn lines.
 * Decouple getting smooth groups from export parameters
 * Export vertex normals if smooth shading is enabled.
 * Refactor:
 * Move file opening errors to constructors.
 * Add a public member function in MaterialWrap.
 * Move object creation code from constructor.
 * Remove unused uv vertex index offset.


 * For polygons with holes:
 * Initial commit for ngon_tessellate: it is untested and will very likely crash.
 * Fix tessellate crash due to wrong Vector size.
 * Add newly creates faces to imported faces and dissolve edges. As of this commit, ngons are tessellated but the holes are also filled up since it keeps falling back to triangle-fan method. Need to debug and fix it to use scan-fill.
 * Edit FaceElem list outside polygon creation code.


 * Parsing
 * Use `blender::StringRef` instead of `std::string_view`.
 * Use remove_prefix for easy transition to StringRef.
 * Read multiple lines with "\\n" line breaks.
 * Silence normal being zero warnings in mesh validate's verbose mode.
 * Silence mesh_validate verbose warnings.

Week 11 August 10 - 15

 * Catch out of range exceptions for stoi/stof..
 * Add vertex offset due to Curves' vertices also..
 * Use const& instead of pointers in Mesh class..
 * Accept 3D UV vertex coordinates.
 * Fix loose edge export & import, and offsets..
 * Fix vertex offsets for MEdges and MLoops..
 * Fix several material import bugs, remove wrong assert..
 * Fix usemtl line not being written.
 * Use function pointer; remove redundant variables.
 * Export packed image file names as if they're in current directory.
 * Use const; move constructors to implementation file.
 * Fix several material import bugs, remove wrong assert.
 * Apply axes transform to exportable Nurbs.
 * Move MTL writer code near OBJ writer code.
 * Skip parsing extra texture map options, move MTLWriter from Objects code.
 * Clean up: use const, header cleanup.
 * Move file open errors to constructors.
 * Add a public member function in MaterialWrap.
 * Cleanup : clang format; cout > cerr

Week 10 August 4 - 7
Committed material creation prototype & fixed several issues later on.
 * Initial material creation support..
 * Add created material to the object.
 * Use Map's lookup instead of direct pointer to MTLMaterial.
 * Fix bsdf_ being null; asserts for SOCK_RGBA.
 * Don't add null Image* to the p-BSDF node..
 * Load image and add it to image texture node.
 * Cleanup: Renaming, comments..
 * Use pragma once as per style guide.,.
 * Refactors:
 * Replace enum and its functions with Map for maps
 * Move index offset from parsing to Object creation
 * Use std::string_view for string splitting. This gave 33% speedup.	 OBJ_import_using_string_view.png
 * Create new Geometry instance at the start.


 * Accept 3D UV vertex coordinates
 * Fix wrong material exporter assert added recently.

Week 9 July 27 - 31
I could not work for the whole week productively.
 * Updated design doc: https://docs.google.com/document/d/17Uzl47OljjoKgaMbukiLHUVGQP220lPTmPS-atb65mw/edit
 * Add fallback value in string to number conversions.
 * Add MTL parser.
 * Cleanup: Use MutableSpan, add asserts & make exporter's material code similar to importer's.
 * Initial material creation support.
 * Off-topic (Master) Enable header auto-complete suggestions in Xcode. This turned into a standardised way of dealing with Xcode's include file paths.
 * Off-topic: Respond to a new developer on a bug fix and the devtalk thread
 * Off-topic: Tested ClangBuildAnalyzer on Blender (lite build).

Week 8: July 20 - 25

 * Move vertex (and UV vertex) list outside `OBJRawObject` so that objects that are defined (`o curvename`) after the list of vertices is written can access them using relative indices.
 * Support NURBS curve import . Got help from Howard Trickey for fixing a visibility bug.
 * Fix build errors: avoid `uint` with Vector's `size` function. . Style guide was updated to discourage `uint` usage and `{}` initialisation doesn't allow narrowing conversions. There's still a lot of code in OBJ exporter that uses `uint`, that's for a Friday. This commit was only for removing `uint` with `Vector.size`.
 * Worked on Refactor: Conform to style guide, add documentation..
 * Move Object creation code in Mesh creation class. . This helps adding deform groups and materials to the `Object` in one place and away from the code that adds that `Object` to a Collection.
 * Rename `exportable_meshes` to `exportable_as_mesh` . (This was long due.)
 * Support importing vertex deform groups..
 * Use VectorSet instead of vector for def_group. . The commit used `std::vector` which is slow due to linear searching. Jacques Lucke suggested using `VectorSet` for faster lookups. Also `StringRef` is used to avoid copying.
 * Cleanup: silence clang-tidy warnings; comments..
 * Wrote an intial draft of importer's design document. T68936#982751. Still need to incorporate mentors' feedback on the document. Google doc for Importer design

Next Week plans

 * Fix design doc
 * MTL and material grouping.
 * Port a portion of mesh editing tools written in python to C++: `bpy_utils.mesh.ngon_tessellate` (in `release/scripts/modules/bpy_extras/mesh_utils.py`
 * Add error handling for bad meshes or fix them.

Week 7: July 13 - 17

 * Break code into mesh, objects, and reader files.
 * Directly create Mesh without using intermediate BMesh . It lowered the peak memory usage by half. Also made mesh creation faster.
 * Use `stoi` and `stof` instead of `>>` for faster parsing. . Time taken in string to float conversion is less now as compared to `>>` operator. But the cost is in splitting the string and allocations made for several substrings. Overall this is faster. OBJ_file_import_using_stoi.png
 * Fix UV crashes and redundant loops in v1//vn1 case.
 * Support meshes with edges with or without polygons.
 * Importer: Support smooth shading for polygons.
 * Importer: Fix misaligned UV vertices of a mesh
 * Exporter: fix vertex normals not being exported when smooth shading is enabled. Calculating smooth groups before exporting normals was the solution.

Week 8 planned tasks

 * Due to curves, the parser needs some changes in how it stores the vertices. There can be vertices that belong to no object. And then later on an object refers to them with relative indices. So there needs to be another copy of all vertices which can be referred to by curves.
 * Grouping
 * MTL

Week6: July 6 - 10
Side note: an interesting diff is under review: Use mmap IO for reading uncompressed .blends
 * Setup clang tidy with Ninja. On macOS, had to build llvm + clang 11 + tidy to get the executable. Also setup a build with clang 11 + leak sanitiser since Xcode's asan doesn't provide the functionality.
 * Import a mesh successfully in the viewport. It took a long time, trying to avoid Mesh, BMesh asserts, seg-faults and memory leaks due to un-freed meshes.
 * The mesh has vertex data only, so far. Texture coordinates are to be added.
 * The parser takes a long time. ~70% of the import process. Making a Blender object from the parsed data is the rest ~30%. So need to optimise it.
 * The importer is still 3-4 times faster than the python one.

Week 7 planned tasks

 * Texture coordinates, normals and curves.
 * Vertex Grouping, object grouping.
 * Material library and grouping.

Week 5: June 29 - July 4

 * Support smooth groups and bit flags, per face
 * Support vertex groups per face
 * Support modifier's viewport and render properties
 * Review update to adhere to style guide & comments update.
 * Null Checks: material, image , UV map , deform vert.

Ticked exporter checkbox in the status tracker task.

Week 4: June 22-27

 * Review update and cleanup in the OOP refactor:.
 * Support NURBS being exported in parameter form, optionally. So far, they were exported as meshes..
 * Support NURBS surfaces to be exported as a regular mesh. (same as above )
 * Broke down files to create new ones for `OBJMesh` and `OBJNurbs`. Later on, `MTLWriter` will also have its own implementation file.
 * Support material library export. We decided against using the existing python writer, since there were no benefits of using it except saving some time now. To extend support, one would have to edit Node Shader Utils file first. Also debugging it would be a trouble. Since nodes are already in C/C++, we have more control there, instead of in the API.
 * Export of single material per object: . Support for multiple materials in one object:
 * Changes in the UI:
 * Object groups:
 * Material groups:
 * Export selected objects only:.
 * Bug fixes:
 * Duplicate texture coordinates:
 * Show red colored warning if overriding an existing file while exporting (it was a wrong flag)
 * Fix socket vector being read as socket RGBA

Remaining tasks

 * Smooth groups (and bitflags) & Vertex groups
 * Start with importer. I'm already lagging here User:Ankitm/GSoC_2020_Proposal_IO_Perf. But since the refactor I had planned for week 7 (along with profiling) won't take long, I feel that I have a good buffer to work in.

Week 3: June 15-19
This allowed us to have minimal memory overhead with no timing cost. UV vertices, however, have to be allocated on heap and thus show a peak while exporting.
 * Animation: Allow all frames to be exported, instead of 0-1000. Also filenames are also edited with frame number only if there are multiple frames.
 * Worked on exposing more settings in the UI:
 * triangulating a mesh,
 * export UV vertices, export normals
 * axes transform (up, forward axes).  Scaling factor
 * Add support for curves to be exported after converting them to meshes.
 * Refactor the code upto now to object-oriented style. (commit [todo], in review )

Unfortunately, I couldn't finish some goals at time: Material library, NURBS and grouping settings. So the goal for next week is to finish them as soon as possible and start with importer.

Week 2: June 8-13
Here's a table of comparison of a single object to one file which includes UV coordinates too.
 * The OBJ exporter now exports Texture Coordinates. . Not the MTL file yet. It's a goal of Week 3.
 * Since fstream is not adding any benefits and is slower too, it was removed.
 * It also exports multiple objects in the same file. But so far only `OB_MESH` object types are supported..
 * Multiple frames can also be exported now, in individual files..
 * Progress is now shown on per-file basis in Terminal and timing in the end. It should be improved to act on per-object basis when there is a single frame.
 * The windows build to test is up: https://blender.community/c/graphicall/Pmbbbc/

Planned for Week 3

 * Curves, Axes transform, Scale transform.
 * Object Grouping, Indices (absolute is already there, will discuss about relative.)
 * Materials, Triangulation.

Week 1: June 1-6

 * Received a lot of input from the mentors regarding coding style, comments, documentation, and file structures. Spent quite some time on cleaning the hastily committed . See current version at Diffusion
 * Currently the exporter exports vertices, face normals, and face indices. In world coordinates.
 * It's lacking in (1) multiple objects, (2) texture coordinates (3) Axes transformation. If you compare obj txt files written with python & C++, you'll see differences in vertex order, and some coordinates with opposite sign. But the shape is correct.
 * Did a comparison between the old & new exporters. Release configuration, lite build + opensubdiv, default cube with 9 subsurf modifiers. See images: F8586987 & F8586984. We're checking the time taken by `wm_handler_fileselect_do` function.

Planned for Week 2

 * Textures, multiple objects in the same file.
 * Modifiers, Curves, Multiple frames, Axes transform, Progress logging

Week 0: before June 1

 * Updated the goals to move OBJ in the beginning & PLY in the end. See timeline:
 * Set up the branch `soc-2020-io-performance` : See on diffusion
 * Project status tracker and design discussion task on Phab:
 * Get familiar with gperftools, see gperftools installation and use
 * Set up UI and file-handler API for both exporter & importer (OBJ).

Planned for Week 1

 * Export vertex coordinates, vertex normals, faces & texture coordinates.