From BlenderWiki

Jump to: navigation, search

Some notes on the sun grid renderfarm

  • Systems run Solaris 10 64-bit, AMD-Processors, 4gig ram. [[1]]
  • Internet access is available (inside a solaris zone[[2]], only 1 processor can be accessed)

See Building Blender on solaris

Interactive Session

When running an interactive session on suns you need to have a script that connects to your system. This script is the script that runs as a part of a job.

Our internet can disconnect so I added a infinate loop that reconnects every 2min. this way you can only stop the job be canceling through the web interface. - XXXXXXX replace with the ip/domain name.

if [ -f $HOME/.profile ]; then
  . $HOME/.profile
while true; do
	if [[ -f ~/session.lock ]]; then  
		echo "Session running fine"
		# session must re-create so we know its running
		# while true; do touch ~/session.lock; echo "keepalive"; sleep 20; done
		rm ~/session.lock 
		echo "There is no xterm session running, restarting"
		qsh -display
	sleep 120

Packaging Blender for the sun grid

The directory structure is as follows

./bundle/* - archive to upload to suns grid, will be zipped into an 'Application' resource.
./blender/* - blenders source code. (svn checkout)
./lib/solaris-2.10-x86_64/* - Solaris libs, these are used for building but OpenGL libs from here are also needed to run blender on the renderfarm.

The Files for the package are as follows

Blender binary (statically linked)


Python so blender can load modules like math


Scripts (contense listed below)


Scripts to run the renderfarm

./bundle/ - Master script that runs every line in a text file, and retries if the command fails.

if [ -f $HOME/.profile ]; then
  . $HOME/.profile
ifile=$1 # Use first argument
[ ! -f $ifile ] && exit
while read line_iter
  echo "Running" $line_iter
  while [ ! "$EXITSTATUS" -eq 0 ]
    # This line should contain the qsub command!
    # Deal with failed attempts to add a job.
    if [ ! "$EXITSTATUS" -eq 0 ]
      echo "Failed to add job, sleeping 10sec and try again..."
      sleep 10
done < $ifile
echo "done"

./bundle/ - Called from, renders each line in "".

 #$ -N Blender
 #$ -cwd
 export OUT=${NAN_BLENDERHOME}/render
 export LD_LIBRARY_PATH=/usr/local/lib:/usr/X11/lib:${NAN_BLENDERHOME}/lib
 export PYTHONHOME=${NAN_BLENDERHOME}/pythonhome
 mkdir $OUT
 echo	$NAN_BLENDERHOME/bin/blender -b $@
 	$NAN_BLENDERHOME/bin/blender -b $@
'''./bundle/''' - Only for running blender locally and testing.
 export LD_LIBRARY_PATH=/usr/local/lib:/usr/X11/lib:${NAN_BLENDERHOME}/lib
 export PYTHONHOME=${NAN_BLENDERHOME}/pythonhome

Updating The Package

./ - update and repackage the application to run on sun grid nodes.
(note, required for the renderfarm, just makes updating blender_app.tar.bz2 easier)

rm $PACKAGE.tar.bz2
svn update ./lib/solaris-2.10-x86_64
svn update ./blender
cd blender # blenders svn repo
cd ..
cp ./blender/obj/solaris-2.10-x86_64/bin/blender ./bundle/bin/blender
strip ./bundle/bin/blender
# Update python from solaris lib
rm -rf ./bundle/pythonhome
svn export ./lib/solaris-2.10-x86_64/python ./bundle/pythonhome
# Copy libGL
rm -rf ./bundle/lib
mkdir ./bundle/lib
cp /usr/lib/X11/mesa/amd64/ ./bundle/lib/
cp /usr/lib/X11/mesa/amd64/ ./bundle/lib/
cp ./lib/solaris-2.10-x86_64/python/lib/ ./bundle/lib/
# Remove some unneeded python files for a smaller package
rm -rf ./bundle/pythonhome/include/
rm -rf ./bundle/pythonhome/bin
rm -rf ./bundle/pythonhome/man
rm -rf ./bundle/pythonhome/lib/python2.5/bsddb
rm -rf ./bundle/pythonhome/lib/python2.5/compiler
rm -rf ./bundle/pythonhome/lib/python2.5/config
rm -rf ./bundle/pythonhome/lib/python2.5/ctypes
rm -rf ./bundle/pythonhome/lib/python2.5/curses
rm -rf ./bundle/pythonhome/lib/python2.5/distutils
rm -rf ./bundle/pythonhome/lib/python2.5/email
rm -rf ./bundle/pythonhome/lib/python2.5/encodings
rm -rf ./bundle/pythonhome/lib/python2.5/hotshot
rm -rf ./bundle/pythonhome/lib/python2.5/idlelib
rm -rf ./bundle/pythonhome/lib/python2.5/lib-tk
rm -rf ./bundle/pythonhome/lib/python2.5/logging
rm -rf ./bundle/pythonhome/lib/python2.5/plat-sunos5
rm -rf ./bundle/pythonhome/lib/python2.5/site-packages
rm -rf ./bundle/pythonhome/lib/python2.5/sqlite3
rm -rf ./bundle/pythonhome/lib/python2.5/test
rm -rf ./bundle/pythonhome/lib/python2.5/wsgiref
rm -rf ./bundle/pythonhome/lib/python2.5/xml
# Create with no subdirectory
cd ./bundle/
tar -cvvf ../$PACKAGE.tar *
cd ..
bzip2 -9 $PACKAGE.tar

Example .job file

This file is really really simple, it needs to be uploaded with any job you submit.

01_intro-04.job - Render's 6 frames at once!

qsub -N 01_intro-04_0001 -cwd ./production/scenes/01_intro/04.blend -P ./production/scripts/ -o 01_intro-04_ -f 1
qsub -N 01_intro-04_0002 -cwd ./production/scenes/01_intro/04.blend -P ./production/scripts/ -o 01_intro-04_ -f 2
qsub -N 01_intro-04_0003 -cwd ./production/scenes/01_intro/04.blend -P ./production/scripts/ -o 01_intro-04_ -f 3
qsub -N 01_intro-04_0004 -cwd ./production/scenes/01_intro/04.blend -P ./production/scripts/ -o 01_intro-04_ -f 4
qsub -N 01_intro-04_0005 -cwd ./production/scenes/01_intro/04.blend -P ./production/scripts/ -o 01_intro-04_ -f 5
qsub -N 01_intro-04_0006 -cwd ./production/scenes/01_intro/04.blend -P ./production/scripts/ -o 01_intro-04_ -f 6
qsub -N 01_intro-04_0007 -cwd ./production/scenes/01_intro/04.blend -P ./production/scripts/ -o 01_intro-04_ -f 7

Run "blender --help" for details on other arguments you can use.

Render Presets - Use when you want to render many files with some of the same settings.

# run with...
# blender -b myblendfile.blend -P -a
from Blender import Scene, Object
import sys
def main():
	# Was a camera name given as a python arg???
	# eg...
	# -- SomeCameraName
	camera = None
	argv = sys.argv
	if argv:
			camera = Object.Get(argv[argv.index('--')+1])
			print "ERROR, Camera as argument could not be set '%s'" % argv[0]
	# Settings for all scenes
	for sce in Scene.Get():
		rend = sce.render
		rend.freeImages = True
		rend.saveBuffers = True
		rend.compositeFree = True
		rend.oversampling = True
		# rend.OSALevel = 8 # We dont want to set this now.
		rend.threads = 1
		rend.motionBlur = False
		rend.fieldRendering = False
		if camera: = camera
	# Settings for render scene
	sce = Scene.GetCurrent()
	rend = sce.render
	rend.extensions = 1
	rend.renderwinSize = 100
	rend.compositor = True
	rend.sequencer = False
	rend.sizeX = 1920
	rend.sizeY = 1080
	rend.xParts = 6
	rend.yParts = 6
	rend.fps = 24
	rend.zbuf = False
	rend.halfFloat = True
	rend.preview = False
	rend.touch = False
	rend.noOverwrite =False
	rend.imageType = Scene.Render.OPENEXR
if __name__ == '__main__':

Packaging Libs

These commands make a package with a binary and all its dependencies

 mkdir $MYDIR
 export IFS=$'\n'
 MYLIBS=$(for f in `ldd -d $MYBIN`; do echo $f | cut -f3; done)
 export IFS=$'\n\t '
 for f in $MYLIBS; do cp -L $f $MYDIR/ ; done
 tar -cvfh svnlibs.tar $MYDIR/

EDIT, setting the library path did not seem to work, used a single solaris svn binary from...

SSH On Sun's Grid

Port 22 is blocked from the sun grid, so a default ssh configuration wont be able to call out from an interactive session. This is easy to work around by using port 80 (usually used for http).

This will only work if your server is not currently using http of course.


On a linux system edit /etc/ssh/sshd_config Change "Port 22" to "Port 80"

then restart the sshd server

 sudo /etc/init.d/ssh restart


When using ssh to do access the command prompt just add -p 80, for eg..

 ssh user@biserver -p 80

When using svn you need to set the SVN_SSH environment variable

 export SVN_SSH="ssh -p 80"

Now you can do an rsync copy through ssh with port 80

For rsync

 rsync -ave ssh ~/test.txt


 rsync -ave 'ssh -p 80' ~/test.txt