Org:Institute/Open projects/Peach/Renderfarm

From BlenderWiki

Some notes on the sun grid renderfarm

Systems run Solaris 10 64-bit, AMD-Processors, 4gig ram. [[1]]
Internet access is available (inside a solaris zone[[2]], only 1 processor can be accessed)

See Building Blender on solaris http://wiki.blender.org/index.php/BlenderDev/BuildingBlender/Solaris

Interactive Session

When running an interactive session on suns network.com you need to have a script that connects to your system. This script is the script that runs as a part of a job.

Our internet can disconnect so I added a infinate loop that reconnects every 2min. this way you can only stop the job be canceling through the web interface.

interactive_session.sh - XXXXXXX replace with the ip/domain name.

#!/bin/bash
#
if [ -f $HOME/.profile ]; then
  . $HOME/.profile
fi
 
while true; do
	if [[ -f ~/session.lock ]]; then  
		echo "Session running fine"
		# session must re-create so we know its running
		# while true; do touch ~/session.lock; echo "keepalive"; sleep 20; done
		rm ~/session.lock 
	else
		echo "There is no xterm session running, restarting"
		qsh -display blender3d.dyndns.org:1
	fi
 
	sleep 120
done

Packaging Blender for the sun grid

The directory structure is as follows

./bundle/* - archive to upload to suns grid, will be zipped into an 'Application' resource.
./blender/* - blenders source code. (svn checkout)
./lib/solaris-2.10-x86_64/* - Solaris libs, these are used for building but OpenGL libs from here are also needed to run blender on the renderfarm.

The Files for the package are as follows

Blender binary (statically linked)

 ./bundle/bin/blender

Python so blender can load modules like math

./bundle/pythonhome/

Scripts (contense listed below)

./bundle/blenderBatch.sh
 ./bundle/blenderBatchItem.sh
 ./bundle/blenderTestRun.sh

Scripts to run the renderfarm

./bundle/commandBatch.sh - Master script that runs every line in a text file, and retries if the command fails.

#!/bin/bash
#
if [ -f $HOME/.profile ]; then
  . $HOME/.profile
fi
 
ifile=$1 # Use first argument
[ ! -f $ifile ] && exit
 
while read line_iter
do
  echo "Running" $line_iter
  EXITSTATUS="1"
  while [ ! "$EXITSTATUS" -eq 0 ]
  do
    # This line should contain the qsub command!
    $line_iter
 
    # Deal with failed attempts to add a job.
    EXITSTATUS="$?"
    if [ ! "$EXITSTATUS" -eq 0 ]
    then
      echo "Failed to add job, sleeping 10sec and try again..."
      sleep 10
    fi
 
  done
 
done < $ifile
 
echo "done"

./bundle/blenderBatchItem.sh - Called from blenderBatch.sh, renders each line in "blender.in".

 #!/bin/bash
 #$ -N Blender
 #$ -cwd
 export NAN_BLENDERHOME=$PWD
 export OUT=${NAN_BLENDERHOME}/render
 export LD_LIBRARY_PATH=/usr/local/lib:/usr/X11/lib:${NAN_BLENDERHOME}/lib
 export PYTHONHOME=${NAN_BLENDERHOME}/pythonhome
 
 echo $LD_LIBRARY_PATH
 echo $NAN_BLENDERHOME
 
 mkdir $OUT
 echo	$NAN_BLENDERHOME/bin/blender -b $@
 	$NAN_BLENDERHOME/bin/blender -b $@
 
'''./bundle/blenderTestRun.sh''' - Only for running blender locally and testing.
 #!/bin/bash
 export NAN_BLENDERHOME=$PWD
 export NAN_BLENDEROUT=${NAN_BLENDERHOME}/render
 export LD_LIBRARY_PATH=/usr/local/lib:/usr/X11/lib:${NAN_BLENDERHOME}/lib
 export PYTHONHOME=${NAN_BLENDERHOME}/pythonhome
 
 echo $LD_LIBRARY_PATH
 echo $NAN_BLENDERHOME
 
 $NAN_BLENDERHOME/bin/blender

Updating The Package

./update.sh - update and repackage the application to run on sun grid nodes.
(note, required for the renderfarm, just makes updating blender_app.tar.bz2 easier)

#!/bin/bash
PACKAGE="blender_app"
rm $PACKAGE.tar.bz2
 
svn update ./lib/solaris-2.10-x86_64
svn update ./blender
 
cd blender # blenders svn repo
gmake NAN_NO_KETSJI=true NAN_NO_OPENAL=true WITH_BF_BLENDERPLAYER=false
cd ..
cp ./blender/obj/solaris-2.10-x86_64/bin/blender ./bundle/bin/blender
 
strip ./bundle/bin/blender
 
# Update python from solaris lib
rm -rf ./bundle/pythonhome
svn export ./lib/solaris-2.10-x86_64/python ./bundle/pythonhome
 
# Copy libGL
rm -rf ./bundle/lib
mkdir ./bundle/lib
cp /usr/lib/X11/mesa/amd64/libGL.so.1 ./bundle/lib/
cp /usr/lib/X11/mesa/amd64/libGLU.so.1 ./bundle/lib/
cp ./lib/solaris-2.10-x86_64/python/lib/libpython2.5.so.1.0 ./bundle/lib/
 
# Remove some unneeded python files for a smaller package
rm -rf ./bundle/pythonhome/include/
rm -rf ./bundle/pythonhome/bin
rm -rf ./bundle/pythonhome/man
rm -rf ./bundle/pythonhome/lib/python2.5/bsddb
rm -rf ./bundle/pythonhome/lib/python2.5/compiler
rm -rf ./bundle/pythonhome/lib/python2.5/config
rm -rf ./bundle/pythonhome/lib/python2.5/ctypes
rm -rf ./bundle/pythonhome/lib/python2.5/curses
rm -rf ./bundle/pythonhome/lib/python2.5/distutils
rm -rf ./bundle/pythonhome/lib/python2.5/email
rm -rf ./bundle/pythonhome/lib/python2.5/encodings
rm -rf ./bundle/pythonhome/lib/python2.5/hotshot
rm -rf ./bundle/pythonhome/lib/python2.5/idlelib
rm -rf ./bundle/pythonhome/lib/python2.5/lib-tk
rm -rf ./bundle/pythonhome/lib/python2.5/logging
rm -rf ./bundle/pythonhome/lib/python2.5/plat-sunos5
rm -rf ./bundle/pythonhome/lib/python2.5/site-packages
rm -rf ./bundle/pythonhome/lib/python2.5/sqlite3
rm -rf ./bundle/pythonhome/lib/python2.5/test
rm -rf ./bundle/pythonhome/lib/python2.5/wsgiref
rm -rf ./bundle/pythonhome/lib/python2.5/xml
 
# Create with no subdirectory
cd ./bundle/
 
tar -cvvf ../$PACKAGE.tar *
cd ..
 
bzip2 -9 $PACKAGE.tar

Example .job file

This file is really really simple, it needs to be uploaded with any job you submit.

01_intro-04.job - Render's 6 frames at once!

qsub -N 01_intro-04_0001 -cwd blenderBatchItem.sh ./production/scenes/01_intro/04.blend -P ./production/scripts/render_settings.py -o 01_intro-04_ -f 1
qsub -N 01_intro-04_0002 -cwd blenderBatchItem.sh ./production/scenes/01_intro/04.blend -P ./production/scripts/render_settings.py -o 01_intro-04_ -f 2
qsub -N 01_intro-04_0003 -cwd blenderBatchItem.sh ./production/scenes/01_intro/04.blend -P ./production/scripts/render_settings.py -o 01_intro-04_ -f 3
qsub -N 01_intro-04_0004 -cwd blenderBatchItem.sh ./production/scenes/01_intro/04.blend -P ./production/scripts/render_settings.py -o 01_intro-04_ -f 4
qsub -N 01_intro-04_0005 -cwd blenderBatchItem.sh ./production/scenes/01_intro/04.blend -P ./production/scripts/render_settings.py -o 01_intro-04_ -f 5
qsub -N 01_intro-04_0006 -cwd blenderBatchItem.sh ./production/scenes/01_intro/04.blend -P ./production/scripts/render_settings.py -o 01_intro-04_ -f 6
qsub -N 01_intro-04_0007 -cwd blenderBatchItem.sh ./production/scenes/01_intro/04.blend -P ./production/scripts/render_settings.py -o 01_intro-04_ -f 7

Run "blender --help" for details on other arguments you can use.

Render Presets

render_settings.py - Use when you want to render many files with some of the same settings.

# run with...
# blender -b myblendfile.blend -P render_settings.py -a
from Blender import Scene, Object
import sys
 
def main():
 
	# Was a camera name given as a python arg???
	# eg...
	# -- SomeCameraName
	camera = None
	argv = sys.argv
	if argv:
		try:
			camera = Object.Get(argv[argv.index('--')+1])
		except:
			print "ERROR, Camera as argument could not be set '%s'" % argv[0]
 
 
	# Settings for all scenes
	for sce in Scene.Get():
		rend = sce.render
		rend.freeImages = True
		rend.saveBuffers = True
		rend.compositeFree = True
 
		rend.oversampling = True
 
		# rend.OSALevel = 8 # We dont want to set this now.
		rend.threads = 1
 
		rend.motionBlur = False
		rend.fieldRendering = False
 
		if camera:
			sce.camera = camera
 
	# Settings for render scene
	sce = Scene.GetCurrent()
	rend = sce.render
 
	rend.extensions = 1
	rend.renderwinSize = 100
 
	#rend.enableSky()
	#rend.enablePremultiply()
	#rend.enableKey()
 
	#rend.enableBWColor()
	rend.enableRGBColor()
	#rend.enableRGBAColor()
 
	rend.compositor = True
	rend.sequencer = False
	rend.sizeX = 1920
	rend.sizeY = 1080
 
	rend.xParts = 6
	rend.yParts = 6
 
	rend.fps = 24
 
	rend.zbuf = False
	rend.halfFloat = True
	rend.preview = False
 
	rend.touch = False
	rend.noOverwrite =False
 
	rend.imageType = Scene.Render.OPENEXR
 
if __name__ == '__main__':
	main()

Packaging Libs

These commands make a package with a binary and all its dependencies

 
 MYBIN=/opt/csw/bin/svn
 MYDIR=./bundle
 mkdir $MYDIR
 export IFS=$'\n'
 MYLIBS=$(for f in `ldd -d $MYBIN`; do echo $f | cut -f3; done)
 export IFS=$'\n\t '
 for f in $MYLIBS; do cp -L $f $MYDIR/ ; done
 cp $MYBIN $MYDIR/
 tar -cvfh svnlibs.tar $MYDIR/

EDIT, setting the library path did not seem to work, used a single solaris svn binary from...

http://osdir.com/ml/version-control.subversion.rapidsvn.devel/2006-10/msg00020.html

SSH On Sun's Grid

Port 22 is blocked from the sun grid, so a default ssh configuration wont be able to call out from an interactive session. This is easy to work around by using port 80 (usually used for http).

This will only work if your server is not currently using http of course.

Server

On a linux system edit /etc/ssh/sshd_config Change "Port 22" to "Port 80"

then restart the sshd server

 sudo /etc/init.d/ssh restart

Client

When using ssh to do access the command prompt just add -p 80, for eg..

 ssh user@biserver -p 80

When using svn you need to set the SVN_SSH environment variable

 export SVN_SSH="ssh -p 80"

Now you can do an rsync copy through ssh with port 80

For rsync

 rsync -ave ssh ~/test.txt myname@url.org:/home/myname/

Becomes...

 rsync -ave 'ssh -p 80' ~/test.txt myname@url.org:/home/myname/