Doc:2.5/Manual/Render/Engines/Netrender

From BlenderWiki

[edit] Description

Network renderer from inside Blender

Goals:

• Transparency
• Flexibility

[edit] Download

In 2.5 (svn trunk)

[edit] Instructions

[edit] GUI

[edit] Master

On one machine, start a Master server.

• Start Blender, switch Render Engine to Network Render using the dropdown in the Info window header (next to Scene)
• Select Master as mode of operation
• Optional Specify the IP address of the interface to listen on as well as the port. Leave at [default] if you want the server to listen on all network interface on the specified port.
• Press Start (it will open a blank render window). The render status line will reflect the actions of the server
• The Master will run until stopped by pressing Esc, as if canceling a normal render.

[edit] Slaves

On other machines, start render slaves

• Start Blender, switch Render Engine to Network Render
• Select Slave as mode of operation
• Optional Specify the IP address of the master server as well as the port. Leave at [default] if you want the slaves to automatically detect the master from its broadcast.
• Press Start (it will open a blank render window). The render status line will reflect the actions of the slave
• The Slave will run until stopped by pressing Esc, as if canceling a normal render.

[edit] Client

To send job to the cluster, from your workstation:

• Open the blend file to be rendered. Confirm your render settings (size, etc)
• Save the file (it sends the last saved file at this point)
• Select Network Render as Render Engine
• Select Client as mode of operation
• One of the following:
  • Specify the IP address of the master server as well as the port.
  • Press the Refresh button underneath the address to automatically detect the Master server from its broadcast.
• Press Send Job to dispatch the animation job to the Master server
• Whenever you want, Render the Animation (Ctrl-F12) to gather the finished frames. Finished frames with "appear" automatically while it will pause on ongoing frames.
• You can also hit Render on any frame of the animation and it will fetch the result from the Master.
• In the simplest example, you can just press "Animation on network" and wait for the frames to come in. Total render time should be close to inverse proportional to the number of slaves (minus transfer times).

[edit] Command Line

• Configure master and slaves as described previously. Save configurations to separate blend files (ie: master.blender and slave.blend).
• Use background rendering to start the master and slaves like so:
  • blender -b master.blend -a
  • blender -b slave.blend -a
• Master and Slaves can be stopped with Ctrl-C (it is recommended to stop the Slaves before the Master).

[edit] Extra

Full multilayer render results are used, so the final results should be exactly the same as a local render. You don't have to specify this as output in the original file, it's done on the slaves automatically.

More info for testers here: http://blenderartists.org/~theeth/temp/netrender_log.txt

Testers are invited to contact me on irc (#blendercoders) or by email.

[edit] Settings

The Render Engine drop down is located in the Info window at the top of the Blender window. This is where you select Network Render to access NetRender features.

[edit] Master

• Network Settings
  • Start Service: Starts the Master Server
  • Path: Where the Master will save job files, results, logs and others. It will create a new directory there of the form master_<pid> where <pid> is the process ID of the Master server.
  • Server Address: Address of the network interface that the Master will listen on. [default] means listen on all available network interfaces.
  • Port: Port that the Master will listen on.
  • Open Master Monitor: Open a browser to the Web-based Master monitor. Enabled when the Master is running.
• Master Settings
  • Broadcast: Broadcast the Master's Address and Port on its local network (every 10s).
  • Clear on exit: Remove the directory created in Path when the Master is stopped.

[edit] Slave

• Network Settings
  • Start Service: Start the Slave node.
  • Path: Where the Slave will save job files, results and logs. It will create a new directory there of the form slave_<id> where <id> is the Slave ID assigned by the Master server.
  • Server Address: Address on which the Master listens.
  • Port: Port on which the Master listens
  • Refresh: Listen to the Master's broadcast to determine its Address and Port (can take up to 20s).
  • Open Master Monitor: Open a browser to the Web-based Master monitor. Enabled when the Master's address is valid.
• Slave Settings
  • Clear on exit: Remove the directory created in Path when the Slave is stopped.
  • Generate thumbnails: Create thumbnails of the render result on the Slave (they are otherwise created on demand by the Master).
  • Threads: How many threads should the Slave use for rendering.

[edit] Client

• Network Settings
  • Path: Where the Client will save its temporary render result file.
  • Server Address: Address on which the Master listens.
  • Port: Port on which the Master listens
  • Refresh: Listen to the Master's broadcast to determine its Address and Port (can take up to 20s).
  • Open Master Monitor: Open a browser to the Web-based Master monitor. Enabled when the Master's address is valid.
• Job Settings
  • Animation on network: Send the current file as job to the Master and waits for results (other than the rendering taking place elsewhere, this works like a normal Render Animation).
  • Send Job: Send the current file as job to the Master. The job ID returns becomes the current job ID.
  • Get Image: Request the current frame in the current job from the Master (waits until the result is available if it isn't already). The result is the same as a normal Render Frame.
  • Get Animation: Request all frames in the current job from the Master (waits when encountering a frame that hasn't been rendered yet). The result is the same as a normal Render Animation.
  • Name: Name of the job. [default] uses the name of the blend file.
  • Category: Category of the job, Optional. Jobs on the Master are also balanced by Categories.
  • Priority: Priority of the job. The Priority level is a multiplier that makes the Master count the job as if if were X jobs (ie: balancing between a priority 1 and a priority job will make them take 33% and 66% of the workload respectively).
  • Chunks: How many frames are dispatched to a Slave as part of the Chunk of work.
• Slaves Status
  • List: List of all Slaves connected to the Master.
  • Refresh: Refresh the Slaves information from the Master
  • Remove: Move the selected Slave to the Blacklist.
• Slaves Blacklist
  • List: List of all Blacklisted Slaves.
  • Remove: Remove the selected Slave from the Blacklist.
• Jobs
  • List: List of all jobs on the Master.
  • Refresh: Refresh the Jobs information from the Master
  • Remove: Remove a Job from the Master.
  • Remove All: Remove all Jobs from the Master.
  • Get Results: Get all available frames from the selected Job. Results are downloaded as multilayer EXR into the current output directory.

[edit] Notes and Known Bugs

• No shared network space required between nodes
• You can dispatch many different files, all results can be retrieved independently (save the file after the dispatch if you want to close it and retrieve later).
• There is very little network error management, so if you close the master first, stuff will break. Same if you enter an invalid address.

YES, I know the current workflow is far from being ideal, especially from a professional render farm point of view. I expect Matt to whip me and suggest better stuff. Optimally, I'd like if users could just press "Anim on network", it would automatically dispatch to the network and wait for results, like a local render. All "pro" features should be optional.

[edit] Load Balancing

Primary balancing is done by calculating usage of the cluster every 10s for each job, averaged over time. The next job dispatched is the one with lowest usage (the one that is using the lesser number of slaves). The priority of a job acts as a divisor, so a job of priority 2 would use a percentage of the cluster as if it were 2 jobs and not just one (ie: a job of priority 1 and one of priority 2 sharing slaves will use respectively 33% and 66% of the processing power). On top of that, there's a set of exceptions and first priority rules.

Exceptions:

• A single job cannot use more than N% of total slaves, unless it's the only job. That prevents slow job from starving faster ones. This is set at 75% for now, but should be customizable.

First Priorities. Jobs that fit in the following

• Less than N frame dispatched (prioritize new jobs). The goal of this is to catch errors early.
• More than N minutes list last dispatch. To prevent high priority jobs from starving others.

[edit] To do

• Send job from memory
• Don't depend on render engine choice for visibility
• "Expert" render manager
• Better defined communication protocol
• The option to calculate simulations (cloth, smoke, ...) on a node which would then send point cache to server for dispatch to render
• Pack textures on upload
• Dispatch single frame as tiles
• Create web interface to control and monitor jobs

[edit] Technical Details

Out of date, read the code and put info here.

[edit] Feature List

• support paths instead of files
• client-server-slave: restrict job to specific nodes
• client-server-slave: view node machine stats
• client-server-slave: reporting error logs back to manager (all stdout and stderr from nodes)
• Cancel jobs
• Restart error frame
• Disable crash report on windows
• Dispatch more than one frame at once (a sequence of frames)
• Blacklist slave that errors on frame after reset
• Multiple paths on job announce
• Delay job until all files accounted for
• Frame range restrictions (ie: send point cache files only when needed for the range of frames)
• Send partial logs to master
• TODO: Set slaves to copy results on network path
• TODO: client-master: archive job (copy source files and results)
• TODO: master-slave: restrict jobs based on specs of slaves.

[edit] API Feature Wishlist

This is a list of blender code I would need to make netrender better. Some of them are bugs, some are features that should (hopefully) eventually be there.

• API access to jobs, to be able to run masters and slaves in the background as well as render job notifiers on the client.
• Render result from multilayer image in memory
• Render and load tiles in render results

If you have any question about this page topics, please leave a message in the discussion page.
Please report any problems editing or creating these pages: Missing images in this page, Broken links in this page. Thanks!