User:MingXu/GSoC2022/Proposal

Project Title

Waveform Drawing Improvements

Name

Ming Xu

Contact

Email: [email protected]
Blender id: xsmingerfan(the same as blender chat)

Synopsis

This project focuses on waveform drawing improvements in VSE, which will better the workflow of video editing. The main goal is to optimize the waveform drawing process so that it can be parallelized and parts of the waveform will be drawn if they are available. As well, the waveform can be cached on local disk to avoid waveform regeneration when the project starts up next time.

Benefits

If artists need to view the waveform after starting the blender, they have to wait, which is a waste of time. This project is expected to improve this experience. The waveform will be drawn dynamically according to the position of the timeline playhead, and multithreading will accelerate this process. These will lead to a smoother workflow in video editing.

Deliverables

1 Faster waveform drawing

2 Waveform drawing system which tracks timeline playhead
3 Optional cache file for waveform

Project Details

User interface

1.The progress bar will be canceled, instead the available portion of the waveform can be drawn as it loads just like figure (a).Waveform will be processed in segments. Those near the timeline playhead should be guaranteed to be drawn preferentially, which makes sure that artists view waveform “in real-time” as drawing is being processed.

2.A new check box can be added in the Source panel of Property like Figure(b). This checkbox named “Waveform Disk Caching” provides an option for users to choose whether cache is enabled to accelerate waveform drawing.

General idea

Waveform can be cached as an array, which is contiguous in memory. So, it can be divided into many pieces by index, whose length is based on the waveform. Each piece will be processed by a different thread. These threads read the sound, calculate the waveform and send data to the main thread to draw the actual waveform. A doubly linked list(List A) will act as a bridge between processing and drawing. The element of the list is a structure(Struct A) containing data such as:

1 Start index & end index: start and end index of the (waveform)array piece

2 Actual data that can be directly used in drawing function (such as the iterator for draw_waveform function in sequencer_draw.c)

3 Etc

As shown in Figure(c), a processing thread will save data in struct A and the structure will be appended to List A after the task has been accomplished. Drawing thread traverses the linked list and picks each actual data in the list as an input of the drawing function. A Mutex may be used in List A to ensure thread safety. Processing thread has two different “priorities”(I’m not sure whether it can be realized through BLI threads), for there are two thread types which I call them “playhead relatedly part” and “normal part”. Threads in “Playhead relatedly part” deal with drawing near timeline playhead. These should have a higher priority to assure that the waveform near the playhead is drawn first. Threads that belong to the "normal part" dispose segments one by one from the start of audio, which has a lower priority.

In Figure(d), the left number in a block is the start index and the right one is the end index. The linked list should regularly be traversed and if the end index of a block meets the start index of the previous one, they will be integrated with each other. The integration shows in (iii). Block A and B will be deleted and the index of new block A’ will be replaced by the start index of A and end index of B. Actual data in A and B will be copied to A’s but how to migrate these data depends on the data structure. Integration will also occur when two blocks are “close enough”. If the start index of a block minus the end index of the previous one is less than the size of a piece, a request will be dispatched to ask for processing the gap between two pieces. After that, C, D and the (requested) block will be integrated. To avoid pointless Linked list traversal, algorithms illustrated in Figure(d) may be executed in drawing thread, which reuses “pick data traversal”. When waveform processing has been completed, all the blocks except the final one will be deleted and actual data will be cache on the disk. This is an optional operation controlled by the “Waveform Disk Caching” checkbox.

Potential difficulties & problems

1.Will multithreading actually speed up the waveform drawing? Mutex on Linked list may lead to additional overhead. I am not sure whether this overhead offsets the benefits of multithreading.

2.Can Sound reader support multithreading? Disk IO and decoding may be two of the most significant performance bottlenecks when disposing of a waveform. Whether the Sound reader supports multithreading is critical to the performance of my project. If not, the entire architecture should be refactored.

3.Timeline Playhead moves when playing the video. How to track playhead and draw waveform should be considered seriously, as the context of processing thread may be different from drawing thread. Therefore, processing thread may not be able to get the playhead position directly.

4.Disk cache for waveform may be out of date. If the audio is changed outside the blender, the waveform read from the cache may be out of date. A feasible solution is to provide a reload option for users. Detecting this change automatically is a selective method as well, but it may be much more complicated.

Project Schedule

I will have my final test during the GSoC time period for which I have to allocate 2 weeks to review my course during which I may have no time to work on this project.

Details	Time
Learn more about blender source code	2 weeks
Implement multithreading processing	3 weeks
Implement segmented drawing system	2 weeks
Implement disk caching for waveform	1-2 weeks
Reserved time period	2 weeks

Bio

My name is Ming Xu, an Electronic Engineering Student studying in Shenzhen University, China. I learned some video production software such as After Effects, Premiere Pro and Photoshop etc. in senior high school in my spare time and became interested in film and television work since then.

Though I have always wanted to get involved in 3D modelling and animations, pitifully it remained a dream that I didn’t really realize. Fortunately, recently I got to learn something about the blender, whose powerful functions really encouraged and inspired me to go deeper in these fields. I have a lot of experiences concerned. Since grade 10, I have been coding in C/C++. From then on, I have got involved in many C/C++ embedded projects like Smart Watch, so I have a good knowledge of the syntax of the language. Besides, I also wrote a game just like “flappy birds” in python which I learned two years ago. Another advantage I must mention is that I worked on blender coding(see T95012/ D14624) and report a bug in VSE(see T97453) this April. As a result, I am familiar with the blender coding workflow.

GSoC is a chance for me to dive deep in the blender. I sincerely hope you can offer me a chance to participate in GSoC to make my contributions and create more possibilities.