User:Gandalf3/GSoC 2017/Reports

From BlenderWiki

Jump to: navigation, search
Note: This is an archived version of the Blender Developer Wiki. The current and active wiki is available on wiki.blender.org.

Week 1

What I did this week:

Less than I'd like, as I'm currently in the midst of school exams. I did catch up on a few preliminary things; my proposal is now on the wiki: https://wiki.blender.org/index.php?title=User:Gandalf3/GSoC_2017/Proposal

Questions

I assume we are to keep branch regularly updated on dev.b.o. Are their any special guidelines around keeping this branch "stable" or anything like that?

Plans for next week

My plan is to resume poking around the UI code in the user preferences. It may not necessarily be the best thing to start with, but as I have somewhat of a start in that direction already (from back in may), I may as well continue.

As school doesn't get out for another week an a half, I'll try setting a modest goal: get a few buttons/ui elements added to the preferences, with the idea of testing asynchronous operations with them later.


Week 2

What I did this week:

Again, not very much. On a happy note, finals are next week. This means I should be able to get going properly soon!

However, I did get my branch pushed, and I added tab added to the preferences. This turned out to involve more code than I expected (lots of rna changes), and it crashes when starting with my user prefs.

Questions

Is there documentation on RNA/DNA and how it interfaces with python?

Plans for next week

I'm hoping to start implementing a way to run asynchronous tasks. I think my favorite candidate for this so far is a subprocess, as it seems there is some overhead to running an asyncio loop inside blender. Also, a separate process could potentially be made to work without running the rest of blender.


Week 3

What I did this week:

  • Found that crash was due to compiling with incorrect python version (thanks to Moguri)
  • Implemented a simple operator which calls a function as a subprocess (i.e. without blocking blender).
  • Also, my reports are now on the wiki.

Questions

  • Since the plan seems to be to implement everything in an addon (at least to start with), how should I handle including it on my branch? Should I create my own branch on the Blender Addons repo?

Plans for next week

Start thinking about package format/repo format design; implement a "repo" from which blender can list and install packages.


Week 4

What I did this week:

The metadata stored in the index is still subject to change, depending on what exactly a "package" ends up being. For now there are a number of unknowns, e.g. Do all packages have bl_info? Or just addons? Do they have an explicit "type" which should be stored? The answers to these questions will hopefully become clearer as the rest of the design falls into place.

Questions

My plan to use subprocesses for asynchronous operation remains more or less unchanged. That said, exactly how best to communicate with these subprocesses on the blender end is not entirely clear. So far it seems the most promising possibility is to use a modal operator to busy-wait for messages from the subprocess, and limit the time spent busy-waiting as much as possible. If I'm not mistaken, it looks like the cloud addon took this approach. However, especially as I'm not restricted to an addon, I'd be interested in hearing about other ways this might be accomplished.

Plans for next week

Start on networking stuff, namely repository and package fetching. Also begin UI work; get the aforementioned fetching operations working from within blender.


Week 5

What I did this week:

Questions

Where should the local package list be stored? Is $BLENDER_USER_CONFIG a good place?

On a related note, should package files be kept around even after installation? For instance, pacman keeps every package ever downloaded in /var/cache/pacman/pkg This allows for fairly painless rollbacks, which is particularly helpful as the arch repositories don't keep old versions of packages around. On the other hand, this can easily eat up disk space. Pacman provides tools for cleaning the cache up, but they must be manually invoked.

Does this approach seem appropriate for Blender? Would it be better to clean up such a cache automatically? (e.g. deleting based on age, or by number of versions behind the current version) Or would it be preferable to simply not keep a cache at all? (perhaps making it the repository's responsibility to keep old versions)

Plans for next week

Clean up and extend repository generation and fetching. Implement package fetching (continued from last week). Get started on real UI.


Week 6

What I did this week:

  • Code cleanup: restructuring and refactoring
  • Added simple repository list UI

I didn't make as much progress as I would've liked this week, most of my time was spent writing tests and doing code cleanup.

Questions

None

Plans for next week

Repository and package fetching (again continued from last week). Start package list UI.


Week 7

What I did this week:

  • Added an install function.
  • Started on gui. Basic browsing and filtering of packages is possible, as well as (to some extend), installing.

In order to add a operator for collapsing/expanding of package entries I ended up adding a hashed ID for each package. I was hoping to avoid using any kind of unfriendly-to-humans identifier in the operator interface, but it seems that the only way to "pass" a custom class instance to an operator is to store it globally and, if there are many to choose from, give it something it can identify one uniquely with. I also started defining a property group to describe a package with the idea of doing all internal storage, but I'm not sure if that's a good approach either; it makes it harder to store arbitrary extra flags such "is expanded".

Instead, I'm thinking that packages with the same name should be "merged", in the sense that only one entry is presented in the GUI. That entry could then have some extra display of alternate versions which may be installed. This approach does the make risky assumption that no actually different addons will ever share the same name, but as blender doesn't allow same-name addons already, perhaps this assumption is acceptable?

Questions

Plans for next week

Add tracking of installed packages and an uninstall operator, update README with testing instructions.

Also address some smaller UX-improvement things:

  • Allow `repo.json` to be omitted from repository URLs.
  • Improve responsiveness/add progress displays, listen for abort signals on longer-running subprocesses.
  • Prevent subprocesses operators from being started multiple times.


Week 8

What I did this week:

  • Added tracking of installed packages, and an uninstall operator.
  • Added installation/setup instructions to the readme.
  • Put a package repository containing blender's official addons up on http://blendermonkey.com/bpkg/ for testing.

To try it out:

  1. Clone blender and checkout the soc-2017-package_manager branch.
    2. git clone git://git.blender.org/blender.git
cd blender
git checkout soc-2017-package_manager
  2. Build.
  3. Clone the package manager addon repository:
    4.  git clone git://git.blender.org/blender-package-manager-addon.git
  4. Install the addon. Copy (or symlink) the package_manager directory contained within the cloned repository into /path/to/blender/build/bin/2.78/scripts/addons/
  5. Run blender
  6. Open the user preferences, enable the addon and set the repository
  7. Refresh, and switch to the "Packages" section.

The default repository can currently be changed in the addon's preferences section (currently only accessible in the "Addons" section).

A local repository can be generated with the generate_repository script found in the addon repository. Example usage: 

 ./generate_repository /path/to/packages --baseurl 'http://localhost/'

This will produce a repo.json file in the current directory, which can then be copied to the server. The baseurl is prepended to the filename of each package to form the package's url (so for example, http://localhost/node_wrangler.py).

Questions

  • I'm not sure how best to disable asynchronous operators while they are running. The trick is, I want to allow multiple instances same operator to be run at once, but not with the same parameters. This is so that a user can click install on multiple different packages without waiting for any one to complete, but double-clicking on one will not spawn two install processes. I believe operators are only instantiated when they are executed, so I can't store an instance variable to track the running state. I thought of adding a class variable which holds a list of the parameters of running operators. The operator poll() method could then perform a check to see if the current operator's parameters are present in this list, and if so, return false. However, surely there's a better way?
  • Putting the package manager addon in a python package puts the addon itself under a subdirectory, so blender won't recognize it when the whole repository is placed in a submodule in scripts/addons. The blender cloud addon also seems to have the same issue, how was this solved there?
  • ideasman42: I was directed to ask you regarding this, but I haven't managed to catch you in irc.. How do info header progress bars work? The only thing I've found which seems connected to them is this undocumented `template_running_jobs()` method.
  • I noticed that my `bpkg` subpackage isn't getting reloaded when pressing F8, despite getting passed to `importlib.reload()`. Does anyone know why this might be?

Plans for next week

  • More code cleanup.
  • Add addon preferences, enabling/disabling, and more filtering.
  • Multiple repository support.

Continued from last week:

  • Improve responsiveness/add progress displays, listen for abort signals on longer-running subprocesses.
  • Prevent subprocesses operators from being started multiple times.

Week 9

What I did this week:

  • Lots of code cleanup
  • Added filtering based on install state and support level
  • Added enabling/disabling
  • Made a demo video:

I ran into an issue with multiprocessing on windows. It seems that windows doesn't have a nice system call like `fork` for *nix, so python has to create a new python interpreter from scratch. This is okay, except multiprocessing uses `sys.executable` as the thing to execute to get a new python interpreter. In blender's embedded python, `sys.executable` points to blender's executable. This means that starting a subprocess starts a new blender instance instead! It seems the reason `sys.executable` points to blender is due to the `Py_SetProgramName` call made on line 244 of bpy_interface.c. If this is removed, sys.executable points to python, and it appears to work. Whether such an alteration to blender's python environment is acceptable is another matter (perhaps some other things expect `sys.executable` to point to blender?), will investigate further.

Plans for next week

Iron out various bugs which have been discovered (thanks to GiantCowFilms from stackexchange), then begin looking into progress display. Currently it seems the only method for doing this from python just changes the cursor, which I don't think is really optimal for the package manager (it doesn't indicate what the progress is for, can get in the user's way, etc). I'll investigate how the global progress "job" system might be exposed to python.


Week 10

What I did this week:

  • Set up a windows box and got blender building on windows (this took an annoying amount of time, spontaneous system updates interrupting downloads didn't help..)
  • Investigated progress display
    • I want to expose an api for doing real progress bars from python, but on the advice of ideasman42 I've decided to hold off for now and use existing wm.progress* methods.
  • Looked into multiprocessing issue on windows (the 'spawn' method).
    • I solved the problem from last week by telling multiprocessing to use bpy.app.binary_path_python as the interpreter. However, I encountered subsequent issues:
      • Multiprocessing re-executes the module which spawns the subprocess in the new process. This is a problem if that module uses bpy, as (of course) bpy cannot be imported outside of blender. As we at least need bpy to determine the interpreter path, we have to make a special bpy-free module and pass the interpreter path to it. This seems like an awkward workaround to say the least.. And that only revealed another issue:
      • Multiprocessing then dies with a FileNotFoundError: '/path/to/blender/dir/<blender string>'. I'm not sure what causes this or if it's worth finding out (this might not even be the last issue in the chain)

After talking with ideasman42 (who provided a working multiprocessing-less subprocess example), I'm planning to avoid using multiprocessing inside blender and spawn subprocesses directly.

Plans for next week

Begin work on multi-repository support. Currently the addon just stores a list of packages, to support multiple repositories it should deal with repository objects. Refactor multiprocessing code to spawn subprocesses directly and communicate with json serialized objects.


Week 11

What I did this week:

  • Added initial multi-repository support (it hasn't been properly tested yet, and likely introduced a number of issues)
  • Started tweaking UI (using suggestions from Pablo Vazquez)
  • Moved code around a bit. I was approaching things with the idea that the package manager could eventually be used outside of blender, but properly allowing for that adds more complication than I think I have time for at this point.

I didn't get as much done this week as I would've liked, I've just felt a bit off lately. Perhaps its the heat, I don't know. (it's supposed to cool down for a while, so if that's what it is, next week should be better ;)

Plans for next week

Write up user documentation, do bug fixing/cleanup, and address phabricator tasks. Also add a simple browser/web interface to the repository.

Week 12

What I did this week:

  • Fixed multiprocessing issue on windows by catching bpy import errors and assuming that means we are in a new process created by multiprocessing (thanks dr. Sybren).
  • Addressed other minor platform-specific bugs discovered while testing the above fix.
  • Added preferences display (so replacing the addon section with the package section won't lose functionality)
  • Fixed some miscellaneous error handling issues

Plans for next week

Continue fixing and cleanup, catch up on feedback, write final report.

Week 13

What I did this week:

  • Moved code out of addon and into scripts/startup and scripts/modules. This involved more modifications than I expected. For instance, I used a global to store a definitive list of all packages. I didn't really like this to begin with, and splitting operators and panels into separate modules broke it. I worked around it by moving the global to a commonly accessible module, but I'm still not sure this is ideal.
  • Wrote user documentation (no pictures yet, as the UI may change)
  • Observed solar eclipse (okay, that wasn't on the TODO, but it was pretty spectacular)

Plans for next week

Final presentation, continue shrinking TODO (but save larger changes for after the final)

Retrieved from "https://wiki.blender.org/index.php/User:Gandalf3/GSoC_2017/Reports"