User:Ideasman42

Log in / create account

From BlenderWiki

Jump to: navigation, search

Subpages

SVN Masked Copy (script)

When syncing with trunk I found Id sometimes run into conflicts or have trouble merging when I really just wanted a simple copy of trunk (with the exception some directories and files that I wanted to stay unchanged for my own manual editing and merging)

The script below manages this by doing a copy from 1 branch to another, excluding 'masked' files and runs the needed svn commands to add and remove files. 

import os

#----------------------------#
# About...
#
# This script only runs in linux, and requires python and svn.
# 
# I wrote it to allow updating one branch from another,
# when you only care about a small set of files (mask'ed files)
# and you just want the other files to stay in sync with 'trunk' for example.
# 
# This script matches your source tree with those files and adds/removes
# files on your disk and through SVN.
# 
# The script prints bash commands to the stdout,
# so you'll need to redirect the output see below.
#----------------------------#


#----------------------------#
# Usage...
# backup your source tree
# 	tar -cvvf" ../tree_back.tar .
# 
# run the script and direct teh outout into a shell script
#	python svn_masked_copy.py > ../tree_update.sh
# 
# run the generated script
# 	bash ../tree_update.sh
#
# commit the changes
#	svn commit .
#----------------------------#


#----------------------------#
# Config...
src = '/blender_rel/blender/'				# main repo to mirror here
dst = '/blender/pyapi_devel/pyapi_devel/'	# main repo to mirror here (except masked files)
#  ignore files/dirs containing this
ignore_containing = ('.svn', '.pyc')

# files will be masked and directories recursively
# file locations are relative to 'dst'
mask =[\
'svn_masked_copy.py',\
'.sconsign.dblite',\
'blender_scons',\
'epy_doc_gen',\
'source/blender/python/',\
'source/blender/python/BPY_extern.h',\
'source/blender/python/BPY_interface.c',\
]
mask_dst = [os.path.join(dst, f) for f in mask]
mask_src = [os.path.join(src, f) for f in mask]
#----------------------------#


def main():
	def ignore(path):
		return any(i in path for i in ignore_containing)
	
	
	def get_files(path_root, my_mask = []):
		
		def masked(path):
			return any(path.startswith(m) for m in my_mask)
		
		all_dirs =  []
		all_files = []
		walk = os.walk(path_root)
		while walk:
			try:
				path, dirs, files = walk.next()
			except:
				break
			#print path
			if not ignore(path):
				
				# print path, dirs, files
				all_dirs.extend([os.path.join(path, e) for e in dirs if not ignore(e)])
				all_files.extend([os.path.join(path, e) for e in files  if not ignore(e)])
		
		# Mask out
		all_dirs = [e for e in all_dirs if not masked(e)]
		all_files = [e for e in all_files if not masked(e)]
		
		# smallest dir to biggest, so making new dirs works
		all_dirs.sort(key = len)
		
		
		return all_dirs, all_files

	def swap_root(_from, _to, path_from):
		return _to + path_from[len(_from):]

	src_dirs, src_files = get_files(src, mask_src)
	dst_dirs, dst_files = get_files(dst, mask_dst)


	#print swap_root(src, dst, src_dirs[0])
	# remove any files on the destination that arnt in the source.
	
	files_removed = 0
	print '# removing files...'
	for fdst in dst_files:
		fsrc = swap_root(dst, src, fdst)
		if not os.path.exists(fsrc):
			print "svn remove " + fdst
			print "rm " + fdst
			files_removed+=1
	print
	print '# removing dirs...'
	dirs_removed = 0
	for fdst in dst_dirs:
		fsrc = swap_root(dst, src, fdst)
		if not os.path.exists(fsrc):
			print "svn remove " + fdst
			print "rm " + fdst
			files_removed+=1
	print
	print '# creating dirs...'
	dirs_created = 0
	for fsrc in src_dirs:
		fdst = swap_root(src, dst, fsrc)
		if not os.path.exists(fdst):
			print "mkdirhier " + fdst
			print "svn add " + fdst
			dirs_created+=1
	
	print
	print '# copying files...'
	files_added = 0
	for fsrc in src_files:
		fdst = swap_root(src, dst, fsrc)
		
		# THIS IS ONLY NEEDED ON DIFF - BUT COPY ALL FILES FOR NOW
		print 'cat "%s" > "%s"' % (fsrc, fdst)
		
		if not os.path.exists(fdst):
			files_added += 1
			print "svn add", fdst

	print
	print '# files removed' , files_removed
	print '# dirs removed' , dirs_removed
	print '# files created' , files_added
	print '# dirs created' , dirs_created
	print
	
	# Add dirs in the dest that dont exist.
	print '# adding these dirs'
	

if __name__ == '__main__':
	main()

SVN Merge Helper

This script is intended to run in a branch, and merge all recent changes, it handles a few things I was doing manually.

  • Checks there are no local changes before doing the merge.
  • Merges all changes from the last merge to head.
  • Writes out the commit command to use.

Note that running this executes the merge but doesn't execute the commit.

To use add to a directory in your path and run from a branch directory without any arguments.

svn_merge.py 
#!/usr/bin/python3
 
import sys
import subprocess
 
def get_merge_info():
    cmd = "svn propget svn:mergeinfo .".split()
    output = subprocess.Popen(cmd, stdout=subprocess.PIPE).communicate()[0].decode("utf-8")
 
    root, sep, rev = output.partition(":")
    rev = int(rev.rpartition("-")[2])
 
    # example out
    #
    #  /trunk/blender:39992-41932
 
    return root, rev
 
def get_status():
    cmd = "svn status --ignore-externals .".split()
    output = subprocess.Popen(cmd, stdout=subprocess.PIPE).communicate()[0].decode("utf-8")
    return output.strip().split("\n")
 
def main():
 
    print("\nRunning Svn Merge Helper...\n")
 
    # cheap way to check on status, yeah yeah, should use XML...
    status = get_status()
    status = [l for l in status if l[:2] in {"M\t", "M "}]
 
    if status:
        print("Error, local changes found\n\n%s" % "\n".join(status))
        print("\nAborting!\n")
        return
 
    root, rev = get_merge_info()
    cmd = "svn merge ^%s -r%d:HEAD --accept postpone" % (root, rev)
    print("  %s" % cmd)
 
    print("\nExecuting...\n")
    subprocess.call(cmd.split())
    print("\n...Done\n")
 
    root, rev_new = get_merge_info()
    cmd = "svn ci \"svn merge ^%s -r%d:%d\"" % (root, rev, rev_new)
    print("  %s" % cmd)
 
 
if __name__ == "__main__":
    main()

SVN Native EOL 

import os
from os.path import join, exists
 
def source_list(path, filename_check=None):
    for dirpath, dirnames, filenames in os.walk(path):
 
        # skip '.svn'
        if dirpath.startswith("."):
            continue
 
        for filename in filenames:
            filepath = join(dirpath, filename)
            if filename_check is None or filename_check(filepath):
                yield filepath
 
def is_svn_file(filename):
    dn, fn = os.path.split(filename)
    filename_svn = join(dn, ".svn", "text-base", "%s.svn-base" % fn)
    return exists(filename_svn)
 
for f in source_list("."):
    if is_svn_file(f):
        try:
            open(f, 'rb').read().decode('utf-8')
            print('svn propset svn:eol-style "native" %s' % f)
        except:
            # print("skipping", f)
            continue

SVN TimeWarp Script

Sometimes bugs appear between revisions and its tedious to work out what revision they were made.

This script building blender binaries from 1 revision to another, creating named binaries, build and svn update log files for each. 

# Run this from the blender source tree you want to test with
# Run this from the blender source tree you want to test with
import os
# ------------------USER OPTIOINS
blend_bin = "/dsk/data/src/blender/cmake_bmesh/bin/blender"
source_dir = "/dsk/data/src/blender/bmesh"
build_dir = "/dsk/data/src/blender/cmake_bmesh"
rev_old = 42000
rev_new = 42346
rev_step = 10
rev_skip_externs = True
make_args = "-s -j8"
svn_args = ""
# ------------------END USER OPTIONS
 
os.system("svn cleanup %s" % source_dir)
os.chdir(build_dir)
build_count = 0
rev = rev_new
while rev > rev_old:
    rev -= rev_step
    build_count +=1
build_tot = build_count
 
print("Building:", build_count, 'revisions')
 
if rev_old > rev_new:
    rev_old, rev_new = rev_new, rev_old
 
if rev_skip_externs:
    svn_args += " --ignore-externals"
 
build_count = 0
rev = rev_new
while rev > rev_old:
    changed = False
    new_dir = os.path.join(build_dir, "versions", str(rev))
    new_name = os.path.join(new_dir, "blender")
    if os.path.exists(new_name):
        print(new_name, 'exists!. not building')
    else:
        # check that this revision effects this branch
        changed = os.popen("svn log %s -r %s" % (source_dir, rev)).read().strip().replace('-', '')
        if not changed:
            print("Nothing changed this revision!, next", rev-rev_step)
        else:
            print("Building rev: %d, number %i of %i" % (rev, build_count, build_tot))
            build_log = os.path.join(build_dir, "build_%d.log" % rev)
            update_log = os.path.join(build_dir, "update_%d.log" % rev)
            print("\tupdating from svn...")
            os.system("svn update %s --revision %s %s > %s" % (source_dir, rev, svn_args, update_log))
            os.system("rm %s" % blend_bin) # so we know if it didnt build
            print("\tbuilding...")
            os.system("cmake . > %s 2>&1" % build_log)
            os.system("make -s %s >> %s 2>&1" % (make_args, build_log))
            if os.path.exists(blend_bin):
                os.makedirs(new_dir)
                os.system("cp \"%s\" \"%s\"" % (blend_bin, new_name))
                rel_new = os.path.join(new_dir, "release")
                os.makedirs(rel_new)
                os.system("cp -R \"%s\" \"%s/\"" % (os.path.join(source_dir, "release", "scripts"), rel_new))
                os.system("cp -R \"%s\" \"%s/\"" % (os.path.join(source_dir, "release", "scripts"), rel_new))
                os.system('find "%s" -name ".svn" -type d -exec rm -rf {} \\;' % rel_new)
                print("\tdone!")
            else:
                print("build failed!")
    if changed:
        rev -= rev_step
        build_count +=1
    else:
        # dont jump a whole lot since the revision wasnt changed
        rev -= rev_step

SVN Branch Check (Lazy Mentor Script)

This script use useful to get a series of patches from a branch commited by a single user, ignore merges. 

my_student_activity.py

BRANCH = '/media/data/soc-2008-mystudent'
STUDENT = 'mystudent'
import os
f = os.popen('svn log --stop-on-copy --xml')
 
rev = auth = msg = None
totlines = 0 # rough indicator
for l in f:
	if '<author>' in l:
		auth = l.split('<author>')[1].split('<')[0]
	elif 'revision="' in l:
		rev =  l.split('"')[1]
		# print rev
	elif '<msg>' in l:
		# print l
		msg = l.lower()
		for c in '<>_-':
			msg = msg.replace(c, ' ')
 
	elif '</logentry>' in l:
		if not (rev and auth):
			print "We have a problem"
		elif auth != STUDENT:
			print auth, 'is helping out ;-)'
		elif msg and ' merge ' in msg or ' merged ' in msg or ' merging ' in msg:
			print 'Ignoring merge message was:', msg
		else:
			# Act on the info we have.	
			patch = os.popen('svn diff -c ' + rev).read()
			fname = '/tmp/' + STUDENT + rev + '.patch'
			print 'writing', fname
			file = open(fname, 'w')
			file.write(patch)
			totlines += sum(ln.startswith('+') for ln in patch.split('\n'))
 
		rev = auth = msg = None
 
print 'Total lines added', totlines

Diffing 2 SVN tree's with an external diffing tool

This is problematic because SVN will add an ID to the header that differs even when the files are from the same revision. You can do this in each folder to get rid of the ID lines. 

find . -type f -print0 | xargs -0r sed -i 's/\$Id:.*\$/\$Id\$/'

Peach System Setup

Moved to http://wiki.blender.org/index.php?title=Bf-institute


Development Tools

Scripts

mmeld

Multi-Meld is a python script to run meld diff viewer on all files in svn that have been changed, or can take file args. 

#!/usr/bin/env python
 
import sys
import os
import subprocess
 
# avoid encoding issues
sys.stdin = os.fdopen(sys.stdin.fileno(), "rb")
 
 
def modified_files(svn_path="."):
    # get svn status
    cmd = ("svn status --xml %s" % svn_path).split()
    output = subprocess.Popen(cmd,
                              stdout=subprocess.PIPE,
                              ).communicate()[0].decode("utf-8")
 
    # get xml of status
    from xml.dom.minidom import parseString
    tree = parseString(output)
 
    status = tree.getElementsByTagName("status")[0]
    targets = status.getElementsByTagName("target")
 
    files = []
 
    for target in targets:
        path = target.attributes['path'].value
        path = os.path.abspath(path)
 
        for entry in target.getElementsByTagName("entry"):
            subpath = entry.attributes['path'].value
            subpath = os.path.normpath(os.path.join(path, subpath))
            (wc_status, ) = entry.getElementsByTagName("wc-status")
            item = wc_status.attributes['item'].value
            if item in {"modified", "conflicted"}:
                files.append(subpath)
            elif item in {"normal", "external", "unversioned"}:
                pass
            else:
                print("WARNING - %s unknown" % item)
    return files
 
 
def make_original_copies(files):
    """ Special care, first arg return is always removed recursively!
    """
    import tempfile
 
    tmpdir = tempfile.mkdtemp(prefix="mmeld_")
 
    files_tmp_pairs = []
    for fn in files:
        fn_base = os.path.basename(fn)
        fn_tmp = tempfile.mktemp(prefix=fn_base, dir=tmpdir)
        files_tmp_pairs.append((fn, fn_tmp))
 
        # make the copy!
        cmd = ['svn',  'export', '%s@BASE' % fn, fn_tmp]
        print(" ".join(cmd))
 
        process = subprocess.Popen(cmd)
        process.wait()
 
    return tmpdir, files_tmp_pairs
 
def main():
 
    # assyme first arg is meld
    args = sys.argv[1:]
 
    files = [os.path.normpath(os.path.abspath(fn))
             for fn in args]
 
    if not files:
        files = modified_files()
 
    tempdir, files_tmp_pairs = make_original_copies(files)
 
    cmd_files = [arg
                 for fn, fn_tmp in files_tmp_pairs
                 for arg in ("--diff", fn_tmp, fn)]
 
    cmd = ["meld"]  + cmd_files
 
    print(" ".join(cmd))
 
    # print(" ".join(cmd))
    process = subprocess.Popen(cmd)
    process.wait()
 
    import shutil
    shutil.rmtree(tempdir, ignore_errors=True)
 
if __name__ == "__main__":
    main()

svn_clean.sh

#!/bin/bash
svn status --no-ignore \
	| grep '^[?I]' \
	| awk '{print $1, $2}' \
	| grep -v "blender.bin" \
	| grep -v "CMakeLists.txt.user" \
	| grep -v "user-config.py" \
	| xargs rm -rfv

svn_log_author.py

#!/usr/bin/env python
 
# your name here!
check_author = "campbellbarton"
 
import sys
import os
 
# avoid encoding issues
sys.stdin = os.fdopen(sys.stdin.fileno(), "rb") 
 
from xml.dom.minidom import parseString
tree = parseString(sys.stdin.read())
 
log = tree.getElementsByTagName("log")[0]
commits = log.getElementsByTagName("logentry")
 
for ci in commits:
    rev = ci.attributes['revision'].value
    author = ci.getElementsByTagName("author")[0].childNodes[0].nodeValue
    msg = ci.getElementsByTagName("msg")[0].childNodes[0].nodeValue
 
 
    paths = ci.getElementsByTagName("paths")[0]
    files = []
    for p in paths.getElementsByTagName("path"):
        fn = p.childNodes[0].nodeValue
        action = p.attributes['action'].nodeValue
        files.append((action, fn))
 
    if author == check_author:
        print("\n %s %s\n" % (rev, ("-" * (78 - len(rev)))))
        print(msg.encode("utf8").strip())
        print("")
        for action, fn in sorted(files):
            print("%s: %s" % (action, fn))

This is a shell script I use to get logs of work in the previous week. 

echo "Starting:"
 
trim() { echo $1; }
 
 
SVN_TRUNK="https://svn.blender.org/svnroot/bf-blender"
SVN_EXTEN="https://svn.blender.org/svnroot/bf-extensions"
 
TEMP_XML="/tmp/svn_tmp.xml"
 
REV_TRUNK_FROM="$1"
REV_EXTEN_FROM="$2"
REV_TRUNK="$(svn info $SVN_TRUNK | grep 'Rev:' | cut -d':' -f2)"
REV_EXTEN="$(svn info $SVN_EXTEN | grep 'Rev:' | cut -d':' -f2)"
echo "Trunk: $REV_TRUNK_FROM --> $REV_TRUNK"
echo "Ext:   $REV_EXTEN_FROM --> $REV_EXTEN"
 
REV_TRUNK="$(trim $REV_TRUNK)"
REV_EXTEN="$(trim $REV_EXTEN)"
 
echo "Getting Logs:"
 
SVN_TRUNK_BMESH="$SVN_TRUNK/branches/bmesh/blender"
SVN_TRUNK_PROPE="$SVN_TRUNK/trunk/blender"
 
 
svn log $SVN_TRUNK_PROPE --verbose --xml -r$REV_TRUNK_FROM:$REV_TRUNK > $TEMP_XML
cat $TEMP_XML | svn_log_author.py > log_trunk.txt
echo "Written: log_trunk.txt"
 
svn log $SVN_TRUNK_BMESH --verbose --xml -r$REV_TRUNK_FROM:$REV_TRUNK > $TEMP_XML
cat $TEMP_XML | svn_log_author.py > log_bmesh.txt
echo "Written: log_bmesh.txt"
 
svn log $SVN_EXTEN --verbose --xml  -r$REV_EXTEN_FROM:$REV_EXTEN   > $TEMP_XML
cat $TEMP_XML | svn_log_author.py > log_ext.txt
echo "Written: log_ext.txt"
 
rm $TEMP_XML
 
echo "New Revs: $REV_TRUNK $REV_EXTEN"

touchdiff

Sometimes you want to know if any of the files you edited caused errors. Rebuilds you may have missed some so touching all modified files, then rebuilding is useful for this.

Add this to your ~/.bashrc file. 

alias touchdiff='touch `svn st | grep "^M" | cut -f4-8 -d" "`'

crep

search *.c* and *.h* files for some text, prints out numbered results, so you can type in the number to open a text editor at that line. 

crep "Some words"

crep some_func

#!/bin/bash
 
OUT=$(mktemp)
find . \( \
	   -name "*.c"   \
	-o -name "*.h"   \
	-o -name "*.m"   \
	-o -name "*.cxx" \
	-o -name "*.cpp" \
	-o -name "*.cc"  \
	-o -name "*.hxx" \
	-o -name "*.hpp" \) \
	-type f \
	-printf '"%p"\n' | sort | xargs grep -n --color=never -H "$@" >  $OUT
 
# Print text with line numbers
sed = $OUT | sed 'N;s/\n/\t/'
 
echo -n "enter a number to edit > "
read -e LINE
 
if [ "$LINE" == "" ] ; then
        echo "nothing selected... exiting."
        exit 0
fi
 
LINE=$LINE"p" # 4 -> 4p - for sed
EDIT=$(cat $OUT | sed -n $LINE)         # file to edit
 
rm $OUT
 
FILE=$(echo $EDIT | cut -d":" -f1)
NUM=$(echo $EDIT | cut -d":" -f2)
 
# Now edit the file in scite or vim
# vim $FILE +$NUM
# nohup scite $FILE -goto:$NUM &
# qtcreator -client $PWD/$FILE+$NUM & disown
nohup gedit $FILE +$NUM &

Example output 

ideasman42@cambo:/b$ crep BPY_call_importloader
1	./source/blender/python/BPY_interface.c:2694:int BPY_call_importloader( char *name )
2	./source/blender/python/BPY_interface.c:2696:	printf( "In BPY_call_importloader(name=%s)\n", name );
3	./source/blender/blenkernel/intern/exotic.c:2430:					if (BPY_call_importloader(name)) {
4	./source/blender/blenkernel/bad_level_call_stubs/stubs.c:245:int BPY_call_importloader(char *name)
5	./source/blender/python/BPY_extern.h:121:	int BPY_call_importloader( char *name );
6	./source/blender/blenkernel/BKE_bad_level_calls.h:149:int BPY_call_importloader(char *name);
enter a number to edit >

prep

variation - prep, same as crep but searches python scripts - 
#!/bin/bash
 
OUT=$(mktemp)
find . -name "*.py" -type f -print | sort | xargs grep -n --color=never -H "$@" >  $OUT
 
# Print text with line numbers
sed = $OUT | sed 'N;s/\n/\t/'
 
echo -n "enter a number to edit > "
read -e LINE
 
if [ "$LINE" == "" ] ; then
        echo "nothing selected... exiting."
        exit 0
fi
 
LINE=$LINE"p" # 4 -> 4p - for sed
EDIT=$(cat $OUT | sed -n $LINE)         # file to edit
 
rm $OUT
 
FILE=$(echo $EDIT | cut -d":" -f1)
NUM=$(echo $EDIT | cut -d":" -f2)
 
# Now edit the file in scite or vim
# scite $FILE -goto:$NUM &
# vim $FILE +$NUM
gedit $FILE +$NUM

cedit

Search for a file from the current directory and open it at a line number, especially nice to copy/paste from compiling errors. 

cedit somefile.c:255

#!/bin/bash
FILE=$(echo $1 | cut -d":" -f1)
NUM=$(echo $1 | cut -d":" -f2)
if ! [ -f $FILE ]
then
        FILE=`find -name "$FILE" | head -n1`
fi

# scite $FILE -goto:$NUM
# vim $FILE +$NUM
gedit $FILE +$NUM

pyrename

Do you ever want to rename many files without having to use GUI apps? Pyrename is a simple script that takes your file names and puts them in a text file. You can then search/replace, regex, whatever you want to the list of names. Save and close the text editor and the files will be renamed.

just cd into the directory where you want to rename files and type 

pyrename .

http://code.google.com/p/ideasman42-dev-scripts/source/browse/trunk/bin/i42_batch_rename.py

include_lint.py

This script rebuilds blender, commenting one include file at a time, any change in the error output will revert the change. A simple way to get rid of unneeded includes from copy/paste's. Though it works best when used with 'sparse' which complains of undeclared functions, this way C files wont remove own includes. 

http://code.google.com/p/ideasman42-dev-scripts/source/browse/trunk/bin/i42_include_lint.py

bfbranch

an easy way to branch, Theeth wrote this (dont overwrite existing branches!). adding here for convenience. 

#! /bin/sh
if [ $# -ne 2 ]; then
	echo 1>&2 Usage: $0 branch log
	exit 127
fi

svn copy https://svn.blender.org/svnroot/bf-blender/trunk/blender https://svn.blender.org/svnroot/bf-blender/branches/$1 -m "$2"

bftag

Another one from Theeth 

#! /bin/sh
if [ $# -ne 2 ]; then
	echo 1>&2 Usage: $0 tag log
	exit 127
fi
 
if (svn ls https://svn.blender.org/svnroot/bf-blender/tags/$1 1> /dev/null 2> /dev/null); then
	svn del https://svn.blender.org/svnroot/bf-blender/tags/$1 -m "removing tag for retag"
fi
 
svn copy https://svn.blender.org/svnroot/bf-blender/trunk/blender https://svn.blender.org/svnroot/bf-blender/tags/$1 -m "$2"

Program Execution Tracing

Sometimes its useful to know the flow of an application without having to step through each line manually. In cases where similar input makes blender act differently, it can be useful to find out in what part of the code it branches into different areas.

This can be done by running gdb, executong commands from a file, redirecting the output to a file.

gdb_flow.txt 

set pagination off
break main
r  -b /home/ideasman42/test.blend -a
while 1
  step
end
  • insert your own breakpoint at the point before blender starts acting unpredictably.
  • run blender with whatever args you want. 
gdb ./blender --command gdb_flow.txt > ./log_1.txt

Now run the program twice with different input and run diff on the output, I use meld, a visual diffing tool. The memory locations will be different but you can still see where large parts of the flow changes.

Note after writing this I found a similar method that I could not get working http://www.brain-dump.org/blog/entry/101/comments

define lint

This script finds defines in headers and prints them out in order of how many lines use them. Warning - this script is not that nicely written. only runs on a unix system. 

import os
BLENDER = '/b'

# for a single header file
# headers = ['%s/source/blender/include/blendef.h' % BLENDER]

headers = os.popen('find %s/source -name "*.h" -print | grep -v "\.svn" | grep -v "~"' % BLENDER).readlines()
headers.sort()

for head in headers:
	head = head.strip()
	if not head or not os.path.exists(head):
		print '\tnot found', head
		continue
	
	header_text = open(head, 'rU').read()
	
	defs = []
	for i, l in enumerate(header_text.split('\n')):
		l = l.strip()
		if l.startswith('#define'):
			d = l.split()[1].split('(')[0]
			if not d.endswith('_H'):
				defs.append((d, i, l))

	defs = list(defs)
	defs_count = []

	for i, define in enumerate(defs):
		used_count = len(os.popen('find %s/ -name "*.[c|h]*" -print | grep -v "\.svn" | grep -v "~" | xargs grep -n --color=never -H "%s"' % (BLENDER, define[0])).readlines())
		
		used_count-=1 # exclude the define as a user
		
		# We may want to ignore case statements?
		defs_count.append((used_count, define))
	
	# defs_count.sort()
	output = []
	for users, define in defs_count:
		if users == 0: # may also want to check for 1 user
			d, line, fulldef = define
			# print head, users, d, line
			text = '%s:%s\t%s' % (head, line, fulldef)
			output.append(text)
	
	for t in output:
		print t

Filtering Intel C++ Compiler warnings

Intels compiler gives some useful warnings not found with GCC, heres a way to filter them for better review. 

l = open('/root/icc.tmp', 'r').read()
l = l.split('\n\n')

new = []

for a in l:
	first = a.split('\n')[0]
	if 'remark' in first:
		if 'may lose significant bits' in first:											continue
		if 'floating-point equality and inequality comparisons are unreliable' in first:	continue
		if 'argument is incompatible with corresponding format string conversion' in first:	continue
		if 'is not virtual' in first:										continue
		if 'access control not specified ("public" by default)' in first:	continue
		if 'zero used for undefined preprocessing identifier' in first:		continue
		if 'operands are evaluated in unspecified order' in first:			continue
		if 'was never referenced' in first and ('self' in first or 'unused' in first or 'closure' in first or 'dummy' in first or '"arg"' in first):			continue
		
		# may want this but not now, these can be made static
		if 'external declaration in primary source file' in first:			continue
		if 'external function definition with no prior declaration' in first:			continue
		if 'old-style parameter list (anachronism)' in first:			continue
		if 'extra ";" ignored' in first:			continue
		
		
		tag = first.split(':')[-1]
		# print tag
		new.append( (tag, a))



new.sort()
for tag, a in new:
	print a

Screencasting from linux

Start Blender at a lower resolution 

./blender -p 0 0 800 600

Run record my desktop, will create "out.ogg" using a very high bitrate so re-compressing dosn't look bad.

This argument is specific to my setup: 

-device hw:2,0

recordmydesktop -v_bitrate 2000000 --full-shots -fps 10 --overwrite -device hw:2,0 -windowid $(xwininfo | grep "Window id:" | sed -e "s/xwininfo\:\ Window id:\ // ;s/\ .*//")

For fullscreen just do... 

recordmydesktop -v_bitrate 2000000 --full-shots -fps 10 --overwrite -device hw:2,0

Press Ctrl+C in the terminal when your done.

Re-Encode for YouTube Upload Note, I had many issues with mencoder this time, frames dropping and out of sync audio, ffmpeg on its own worked though
You may want to try bitrates between 500k and 2000k 

ffmpeg -i out.ogv -b:v 1000k -vcodec libxvid -g 300 -acodec libmp3lame -ac 1 out.avi

Editing the Video in Blenders sequencer

If all you need is the video unedited you can ignore this part.

Export for editing in blender, even though blenders ffmpeg can read many formats, this one streams best. 

mencoder out.ogv -oac pcm -ovc lavc -lavcopts vcodec=mjpeg:vbitrate=10000 -af resample=44100:channels=1 -o out.avi

Render to high Quality 100 AVI-Jpeg from blender and MIXDOWN the audio from the sequencer panel.

You may want to edit the audio, Audacity is good for noise reduction, compressing and normalizing.

Mux these with... 

mencoder export.avi -ovc copy -oac copy -audiofile export.wav -o export_mux.avi

Compress any video to ogg for the web 

ffmpeg2theora --optimize --channels 1 --samplerate 22050 --audioquality 5 out.ogg -o out_web.ogg

Batch Converting OGG to WAV 

for f in *.ogg ; do mplayer -quiet -vo null -vc dummy -ao pcm:waveheader:file="$f.wav" "$f"  ; done

Development Tools Wishlist

  • Valgrind frontend that can filter output based on library names (could be written in python), since suppression files get out of date.
  • SVN Time Warp tool - Imagine SVN Blame (annotation) with a slider to step to different revisions.
  • Python module to control GCC - so one could write very comprehensive conditions for offline debugging. (Tried to write this but GCC's input is NOT easy to automate)
  • Utility to change an svn checkouts user and/or URL, for now this is ok...
    perl -pi -e 's/oldbob/newjim/g' `find | grep "\.svn/entries"`

BGE Logic Bricks Id like to write/modify

  • DONE! - Constrain Vecocity (local or global, could be used to fake anisotropic friction)
  • DONE! - Angular velocity for add object actuator (add get/setAngularVeclocity to py api)
  • DONE! - property actuator have a toggle option to toggle values, currently doing this requires a few logic bricks,
  • state actuator could have 3 state button so some states could be ignored and not set for copy actuator.
  • track actuator Time setting dosnt keep camera Z axis up.
  • set local loc/size/rot actuator (can be done with ipo's but thats a bit of a hack)
  • State sensor, so you could sens if some state was enabled.

Git/Blender

Recently I wanted to use my own git branch to avoid maintaining patches locally on my hard disk (where a crash would loose all my work!).

Firstly I logged into gitorious and made a clone of the main git branch we're using at the moment. 

https://git.gitorious.org/blenderprojects/blender.git

This was very straightforward so nothing much to say here.

Then I cloned the repo: 

git clone git@gitorious.org:~ideasman42/blenderprojects/ideasman42s-blender.git

Committing changes was easy with: 

git gui

... stage, commit, then push changes

So far so good.

BUT, I need to also get changes to sync with latest blender, this wasn't so simple and had to get help here.

Assume this only needs to be called once. 

git remote add jester git://gitorious.org/blenderprojects/blender.git


Now to sync with the git repo I cloned off: 

git pull jester master

Now to push the changed to my own repo: 

git push

To do a diff across branches (not sure why pull is needed here): 

git pull git://gitorious.org/blenderprojects/blender.git
git diff jester/master

I also made a ".gitignore" file containing lines...

.gitignore
build/

So it ignores my build dir.

Another error came up trying to commit.

The git
// protocol is read-only.

Turns out I need to change the push url in .git/config from... 

url = git://gitorious.org/blenderprojects/blender.git

to... 

url = git@gitorious.org:blenderprojects/blender.git


- Think this is all thats needed, logging here incase I forget or need to do it again.

Another note, if you make a github repo from existing svn source (disconnected from other blender repos) 

git init
git add *

git remote add origin git@github.com:ideasman42/blahblah.git
git push -u origin master

if you get tired of 'git push -u origin master' do... 

git branch -d origin/master
git branch --set-upstream-to origin/master

... now you can just do 'git push'.

BGE Nodal Logic

From discussions with Benoit

Some background information first. In the current implementation, sensors, controllers and actuators are executed separately: first the active sensors, which eventually generate pulse (positive or negative) to controllers. Then the triggered controllers are executed, which eventually generate pulses (positive or negative) to actuators. Then the active actuators are executed and the whole cycle restarts. Actuators are similar to controllers except that they can stay active and execute each frame even without receiving a trigger.

This architecture is easy to program but not easy to use. Normally it should be the opposite and that's why we want to introduce nodal logic. A logic graph allows to visualize a whole logic setup much better than logic bricks.

This proposal is an attempt to bring nodal logic with minimal change in the BGE code. In particular we want to keep the existing C++ sensor and actuator code because they execute complex operations at C++ speed.

A nodal logic is made of interconnected nodes. All nodes have inputs and outputs. There are 2 types of input: logic and data. Logic inputs are used to receive pulses from other nodes. There can be two types of logic inputs: start and stop. Receiving a pulse on start will cause the node to execute. Receive a stop will cause the node to stop executing. The stop input is only present on nodes that have persistent actions (actuators). The data input will be of type vector, integer, boolean, string, object reference, etc, depending on the type of node. The data input is only used when the node is activated. There are also 2 types of outputs: logic and data. Logic output are used to emit pulses. There are two cases where a node can emit a pulse. If the node is a sensor, it can create a pulse to one of its two outputs: a positive output when the internal state becomes true and a negative output when the internal state becomes false. Other types of node have only one logic output and they emit a pulse only when they finish their execution. Data output can be of type vetor, integer, string, reference, etc, depending on the node.

A graph of nodes can be grouped and become a single node for a higher level of logic.

We will define 3 types of nodes:

  • sensor nodes.

These are the only node that can produce pulses. They have 2 logic inputs to explicitely start and stop them. Starting a sensor node means that it will execute on each frame and produce pulse if the internal state changes. It's important to be able to stop sensors to save CPU when they are not needed anymore. Sensor nodes have 2 logic output as explained above. They are executed only once per frame before any other types of node.

  • atomic action node

These nodes are never active at the start of a frame. They can only be active if they receive a pulse from a sensor node or from another action node. Atomic action nodes have only one logic input and one logic output. If they receive a pulse on the input, they execute immediately and release the pulse on the output. Actions nodes can be chained: the chain of nodes is executed immediately in the same frame until the chain is broken or until it reaches a persistent action node or a sensor node. Beware that an infinite loop of atomic action node can be done, which will hang the game engine.

  • persistent action node

These nodes are similar to atomic action node except that they execute only once per frame. Execution of persistent action nodes is delayed after all atomic action nodes. A persistent action node has 2 logic inputs (for start/stop) and one logic output. The logic output produces a pulse when the persistent action is stopped. The pulse can be passed to another node and is handled on the next frame.

Implementation

These 3 types of node match closely the sensor, controller and actuator bricks that we have today so that little change will be necessary to implement them.

Starting/stopping sensors is currently done based on the number of links to controllers. An explicit method must be implemented.

C++ controllers will be removed and replaced by a variety of python node that implement the same functions: AND, OR, etc. All these nodes will be atomic action nodes. A lot of different python node can be predefined: for loop, condition, etc. A generic python node with configurable data inputs and outputs should be implemented to allow custom atomic nodes. This will replace the python controller. The advantage of having configurable inputs means that it's no longer necessary to retrieve reference in the script: the inputs will be set prior calling the node and readily available as variables in the script. It seems appropriate to implement the python node as a method of an class that has data members corresponding to the input and outputs of the node: this way the outputs are available to other nodes even after the script has executed.

C++ actuators remain the same. The actuators that are non persistent by nature will be defined as atomic action nodes, while the persistent actuators will be defined as persistent action nodes.

The execution of the graph can be done by converting it to a python script. Python is well suited to implement data connections between nodes: data inputs and outputs will simply be attributes of sensors/action nodes and easily accessible to python. Execution of sensors and persistent nodes will be left to C++, python will only be used to execute the atomic actions. When the pulse gets to a persistent action, python will simply register the action for execution at the end of the logic frame. Question: how a complex tree of actions can be converted in a python script.

Since the logic bricks are attached to objects, nodes will also be attached to objects. A graph mixing nodes of different objects should be possible. When a group of objects are linked in logic graph and this group is instantiated, the nodes must be instantiated too and the script corresponding to the graph must use references to the instance objects. Question: how the script will acquire the reference to the instance objects.

BGE Python Logic

Heres an idea I had about how developers who prefer to bypass logic bricks could do so...

Currently we have event managers (timer, joystick, keyboard, collision). Each event manager has a list of active sensors

My proposal to bypass logic brick sensors is to define a new type of sensor class that is only accessed from python with no need to have a logic brick in the UI or be connected to a controller.

Event Queue?

From the EventManager it will look like a normal sensor in the list however rather then trigger a controller, the sensor will collect a list of events in a typical event queue. This can be a python list containing any relevant info like the key pressed, location of a ray-hit. Collision-Points when 2 objects touch.

This sensor would need to be enabled (probably via python), otherwise it would be a waste to have this on when not used. 

GameLogic.EventLogEnable('keyboard')
...
evens = GameLogic.Events # a normal python list
while evens:
    event= events.pop()
    if event.type == 'keyboard' and event.key == 'a':
        print("blah blah")

Event Callbacks?

Rather then logging an event queue alternately we could also register python functions to be called on spesific event types. so you could have a function to be called on any collision, keypress etc. This has the advantage that you dont need to run python every logic tick to peek into the event queue. 

def MyFunc(event):
    if event.type == 'keyboard' and event.key == 'a':
        print("blah blah")

GameLogic.EventRegister('keyboard', myFunc)

You could ofcourse register multiple callback functions for each event type, or even use the same function for collision and timer callbacks for eg.

Its also a bit nicer that you only register the callbacks rather then telling python to start logging an event queue.


User documentation editing

  • Menu item for editing docs
  • Built in operator for description editing, (can be python)
    • XML/RPC submission from python (Mindrones)
  • Blog voting system, acceptance etc... (mindrones)
  • Automatic extraction of accepted descrptions (Mindrones)
  • Apply this as a patch to C/Python code + commit (me)


Graphing Memory usage 

 import os
 while 1:
   os.system("ps -eo pid,ppid,rss,vsize,pcpu,pmem,cmd -ww >> ~/monitor.txt")
   os.system("sleep 1")

Random Post Durian Discussion with Brecht

Speed

Cam suggests Re-Structure packages so not ALL modules are loaded at startup. "spaces", "properties" can be packages with all registered classes, each draw function would do its own import.

Brecht suggests we could have all classes autogenerated into a stub which does some tricky class swapping trick on access which swaps in the heavy classes.

We could do an async read on all py files on startup so they dont slow down load times when python waits on access for module load.

Python caches the string hash value, we could make use if this with RNA hashes so we dont need to do "unicode->ascii->ghash"

Plugins

  • Use python as a base, extend with C/C++ modules if needed.
  • Use C++ for plugins rather then C?, this means we dont need to expose BLI functions since C on its own is quite limited. Doesnt need to be forced but can be recommended.
  • Need to check on MSVC's visibility settings so internal C funcs can be exposed like in gcc.
  • Use general system for all plugins (fill in callback members), similar to modifiers.
  • For DNA - use string lookups which resolve to a plugin on load to avoid unique int ID problem.
  • Use derived-mesh arrays for mesh access.
  • Improve RNA api by using inlineed rna accessor functions so rna access can be as fast as direct access.


What Im working On

Page Moved Here: http://wiki.blender.org/index.php/User:Ideasman42/WhatImWorkingOn

TODO

  • ui_hsvcircle_vals_from_pos - avoid sqrt
  • use #pragme GCC warnings for DNA padding macros.
    _Pragma("#pragma GCC diagnostic push") _Pragma("GCC diagnostic error \"-Wpadded\"")
  • PADUP - make a static function.
  • use py identifier for re-used strings - _Py_IDENTIFIER(__import__);
  • auto-document ID property API.
  • add generic way to install presets.
  • add access to active scene layer via RNA
  • add note in docs that image pixel access is very slow. copy list first.
  • add support to export BVH with axis conversion - once done mail sybren _at_ stuvel.eu
  • Make render output path an operator property so testrenders can be written to arbitrary paths without setting scene data.
  • Write up patch so python can reference ID's across undo's & check with Ton if this is ok.
  • option for importers to re-use existing materials (OBJ specifically)
    see: http://blenderartists.org/forum/showthread.php?234521-Import-OBJ-but-using-existing-material
  • help(mesh.update) # and similar RNA functions should show keywords.
  • Option to show enum menu as a search box instead.
  • Run newly linked appended scripts when their 'Register' setting is enabled.
  • Enable operator presets for 3d view, currently they only work with the file selector.
  • Add window relative X/Y coordinates to RNA regions and areas.
  • Cleanup todo http://wiki.blender.org/index.php/Dev:2.5/Source/Development/Todo/Tools
  • Document windowing concepts in blender and how they relate to operators and context. (new section in quickstart)
  • obj.shape_key_add(name="whatever", from_mix=True) - look into a way to remove a shape key.
  • Test operator property comparison for better menu key shortcuts. [#27239]
  • Check mathutils/blf/bgl modules work with blenderplayer.
  • Report error on file load if the engine used is not found.
  • Preset Menu: Currently no nice way to remove a preset.
  • a way for rna functions to pass and return collections to python, eg: allow an rna function to receive a list of objects for eg.

Code Refactoring

  • refactor BLI api's
    • specify consistent naming convention, currently we mix get_object_foo, object_get_foo, object_foo_get
      Suggest to use RNA like naming convention:
      BLI_object_foo_get(...) / BLI_object_foo_set(...)
  • Add in generic header for portability - like BLO_sys_types.h, but not associate with blender modules, so any C file can include to ensure code compiles across different architectures. we may want to have more then one of these.
    • blender_compat_stdint.h - one for int types uintptr_t/size_t etc.
    • blender_compat_storage.h - another for storage/fs types
    • blender_compat_macros.h - another for common macros such as __line__, __func__
  • Other...
    • Look over all uses of dw->def_nr, many can be replaced with api functions.
    • Look over flip_side_name use, some of it can be improved.
    • replace inline def weight editing in ED_vgroup_nr_vert_remove

Python Integration

This section is to gather ideas about whats needed for better python integration:

  • callbacks for RNA properties
  • python listeners
  • python notifiers
  • depsgraph integration (knowing when object data changes, is re-calculated)

Use Cases

Fell free to add your requirements here, Id like to collect them to see whats most important. (just sign what you add)

Notifications for custom property changes: When implementing an additional rendering backend in Blender, there will usually be a slew of custom material properties. In LuxRender and Mitsuba, these are reachable via bpy.data.materials[index].(lux/mitsuba)_material.*.* (i.e. with some nesting). For instance, the 'v' roughness parameter of the Ward model in Mitsuba would be stored at someMaterial.mitsuba_material.mitsuba_mat_ward.vroughness.

An important use case would be to register a callback that is invoked whenever one of these properties is changed so that the user interface can be updated (for instance to cause a redraw of the material preview). In the most general case, the callback would have a signature that provides information about what particular property was changed, and it would also provide the old and new value.

Ideally, this callback would be active across all material instances, without having to register it manually for each one. -- Wenzel 09:21, 30 March 2011 (CEST)


The use case put forward by Wenzel is indeed a common one, which I've encountered myself a lot. Ranging from ID properties for rigging, to custom rna properties for user interfaces.
One thing to keep in mind is when the callback is invoked: while the property is being changed (example: dragging a slider), or after the property has been changed.

Depsgraph integration would be very nice for keeping track of changing mesh topology. It would also be nice for finding out when fcurves and drivers are changed.

For python listeners it would be good if we had access at a low level. For instance: I've written an add-on to display keypresses, which would greatly benefit from access to python listeners at a low level. So across editor-windows, and from within modal operators. Example: currently I can already catch the event that invokes the grab operator, but if a user then presses X-key to limit to x-axis, I can't catch that.

One thing I'd like to stress is the importance of stability of the system. Currently we already have bpy_struct.callback_add but when a new file is loaded and the callback is added again, Blender crashes. The same for disabling an add-on while the modal handler and callback are still attached. This is inconvenient and makes python integration unpredictable. --Crouch 17:22, 30 March 2011 (CEST)

The use cases for LuxBlend25 (LuxRender engine integration) mostly relate to material settings. Before RNA-write restrictions came in, we were able to update the base blender diffuse color setting with an appropriate value from the LuxRender material panels, so that objects in the 3DView were displayed with approximately the correct color. We have a temporary work-around of a "Copy material color to blender" operator, but it would be better if the underlying blender color could be updated automatically (and in addition, also trigger a preview update).

Another use case would be for real-time rendering control, for image-adjustment in the LuxRender core. Ideally, when the render is in progress, we would have several custom panels visible in the Render View which can feed values into the LuxRender post-processing pipeline to adjust levels, tonemapping, light layers etc. --Dougal2

Rigamarule is a system to add intelligence to rigs (that operates in edit mode / rig build time mode (not animation) ). It can be likened to constraints, except not animator visible ones, rather, edit mode constraints. things like aligning a bone with another, or aligning a bone to a lattice, or keeping rolls perpendicular to an IK chain-plane.. etc. Currently rigamarule implements it's own dependency graph, and calculates the rules when the users invokes the rigamarule operator. In the future, we'd like 'live' updating of rules, based on transformation of joints. To do this we need to hook into some kind of event system, that alerts python when certain events happen, and allows executing a callback - or something of this nature. Benefit will be extremely simple/vanishingly transparent UI for people wanting to customize rigs. --Bassamk

Retrieved from "http://wiki.blender.org/index.php/User:Ideasman42"