Dev:Ref/Proposals/MigrationToGit

From BlenderWiki

Note: This is an archived version of the Blender Developer Wiki. The current and active wiki is available on wiki.blender.org.

Overview

There are 2 main parts to git migration.

Converting the repository from Subversion to GIT.
Updating our infrastructure (bug tracker, mailing list, moving over existing logins, updating server software).

We have made a start on both of these, with the conversion mostly done (but there is a lot more turning and cleanup to do too), and testing new server software is also in progress.

Current target is to have this done by October 2013. (After GSOC)

--Ideasman42 15:02, 29 May 2013 (CEST)

Tasks

Could reorganize these..

Current things to be solved:

libraries (which seems to be an svn repo? or there were some tricks for non-versioned files in git repo)
extension+translation (which shall be separate repos, connected to main one as git submodule)
is it easy to synchronize git repo with svn, btw?
server infrastructure: ML + tracker + code review
mailing list: messages goes there by post-commit hook
documentation!
synchronization of trunk to git repo master
proposal: start git in the beginning of release cycle, shut down svn to the end of release cycle (ideal scenario)
how stuff like buildinfo will be handled?
consider which branches to ignore

Branches to Exclude

Over the years many branches have been created, some are valuable to keep but many are historic and never ended up being used.

With git we want to avoid everyone who downloads blenders source also having to download a lot of extra data which is not really necessary. (Git works differently to subversion in this respect).

When choosing which branches to include, there are a few factors taken into account.

if the branch was never merged into trunk.
branches that have very few commits, then merge into trunk.
branches that are syncing with an external repository (where the real development is done).
branches that will be re-written before merging into master (so the current history isn't used).
active branches that are developing more experimental features (not likely to be merged in next ~6 months).

Note - these branches will still be available from a read-only subversion, theres nothing stopping developers from continuing development on 3rd party hosting (github, gitorious, bitbucket... etc), or even adding the branch back into blenders git is possible.

For full list of branches, see: https://svn.blender.org/svnroot/bf-blender/branches

This is the proposed list of branches to exclude from git migration (extracted from the migration script)

# note: /USEGIT suffix means we should use git to delete the branches
branches = [
    "soc-2007-hcube",  #  never merged (sound API, we ended up going with Audaspace)
    "soc-2007-mosani",  # never merged (render api)
    "soc-2007-maike",  # never merged (glsl)
    "soc-2007-joeedh",  # never merged (deep shadow maps)
    "soc-2007-red_fox",  # bevel, since rewritten
    "soc-2008-unclezeiv",  # never merged (light-cuts)
    "soc-2008-djd",  # never merged (unit-testing only 17 commits)
    "soc-2009-yukishiro",  # image based lighting
    "soc-2010-aligorith-2",  # never merged (physics - ended up re-branching and working on later)
    "soc-2010-kwk",  # never merged (paint system, only ~20 commits)
    "soc-2010-leifandersen",  # never merged (testing project)
    "soc-2010-moguri",
    "soc-2010-moguri-2",
    "soc-2010-rohith291991",  # never merged (quad dominant mesh)
    "soc-2011-avocado",  # never merged (mesh tools, very few commits)
    "soc-2011-salad",  # salad is a mix, no need to merge.
    "soc-2011-oregano",  # almost no commits
    "soc-2011-onion",  # never merged (ptex)
    "soc-2011-onion-uv-tools",  # branch is deleted in svn
    "soc-2011-sven",  # never merged (from nodelogic branch, never committed to).
    "soc-2013-vse",  # wrote new sequencer engine - TODO, check if it will be used.
    "soc-2013-cycles_mblur",  # unfinished
    "soc-2013-meshdata_transfer",  # development can continue externally (rewrote started after gsoc finishes)
 
    "2-44-stable/USEGIT",  # use tags instead.
    "2.44-stable/USEGIT",  # use tags instead.
    "BLENDER_INTERN",  # branch made right after initial commit, never committed to.
    "BLENDER_PROJECTFILES",  # branch only contains msvc project files.
    "animsys-aligorith",  # we've merged animsys-aligorith2
    "asset-browser",  # looks like large parts will be re-written - perhaps rebranch once the git move is done.
    "bb_dev",  # never merged
    "bb_dev_25",  # never merged
    "bge_eigen2",  # never merged
    "blender-2.62-editmesh",  # was more of a tag incase bmesh merge failed.
    "blender-autokey",  # created accidentally
    "bmesh_tmp",  # temp branch
    "branch-farsthary",  # never merged (only 10 commits, unlimited clay)
    "carve_booleans",  # not very much useful history here, mainly just getting carve lib setup.
    "cloth-eltopo",  # never merged
    "game_engine/USEGIT",  # merged but only ever had 4 useful commits. remove via git so history isnt lost.
    "ge_candy",  # this branch has some great stuff but looks like no near term plans to merge into master, could continue in external repo.
    "ge_components",  # never merged
    "ge_dev",  # never merged
    "ge_dyn_load",  # we DID merge, but very few commits, not useful history
    "ge_harmony",  # never merged
    "ge_integration",  # never merged
    "ge_nodelogic",  # never merged
    "hairsim",  # never merged (eltopo)
    "hive",  # experimental?, never merged.
    "imgbrowser-elubie",  # 
    "itasc",  # we DID merge this but branch was removed from SVN and history is mostly in a single commit.
    "merwin-playground",  # never merged
    "merwin-tablet",  # never merged
    "merwin-tablet-2",  # never merged
    "multiview",  # only dumps of git branch, no useful history
    "nurbs",  # never merged
    "nurbs-merge",  # never merged
    "nurbs25",  # never merged
    "particles",  # also never merged
    "particles-2010",  # never merged
    "physics25",  # never merged
    "pyapi_devel",  # experimental, never merged
    "pyapi_devel_26",  # experimental, never merged
    "pynodes",  # includes LUA api, we dont use these now.
    "qdune",  # experimental, never merged
    "sound-branch",  # an update of soc-2007-hcube
    "web-plugin",  # never merged (only 4 commits)
    ]

Convert Repository

Using Reposurgeon

This is an alternative method of converting to svn to git we're looking into since svn2git is quite slow for a complete conversion including branches. --Ideasman42 03:45, 6 April 2013 (CEST)

Note, moving scripts to manage migration here:

https://www.gitorious.org/blender-git-migration/blender-git-migration

tested with reposurgeon-2.40

Initial conversion is very simple...

First dump svn (youll need the svn repo, not a checkout)...

svnadmin dump /path/to/my/repo > blender_svn_dump

Run reposurgeon, it will enter into a command prompt.

authors read <authors.txt
read blender_svn_dump
prefer git
rebuild blender

The text above can also run from a file as...

reposurgeon "script blender.txt"

Done!, now there is a git repo with trunk and branches. This on my system the svn dump took approx ~20min, reposurgeon ~30min. - campbell.

Inspecting Git Repo

List branches sorted by date.

git for-each-ref --sort=-committerdate --format='%(refname:short)' refs/heads/

Commit log on a branch

git log --no-merges SOME_BRANCH

Further Investigation

Twice we have had accidental deletion many dirs in svn, reposurgeon can correct the history here. Possibly other large accidental commits.

references

Example conversion configuration.
https://gitorious.org/reposurgeon/nut-conversion/blobs/master/nut.lift

Docs
http://www.catb.org/~esr/reposurgeon/reposurgeon.html
http://www.catb.org/esr/dvcs-migration-guide.html

Using svn2git

Note - It looks like we will use reposurgeon, keeping this for reference.

There're some plans to eventually switch to Git, but there're lots of things to be solved first. Here i'll collect some WIP notes.

Currently experimenting with svn2git scripts. Here're some usage notes so others could play with as well.

We'll need mapping from svn logins to real names+email (that's how Git is intended to store authors). This is created by running the oneliner in svn checkout folder:

svn log https://svn.blender.org/svnroot/bf-blender -q | awk -F '|' '/^r/ {sub("^ ", "", $2); sub(" $", "", $2); print $2" = "$2" <"$2">"}' | sort -u > authors.txt

Notices, that this will ignore some logins (like empty and root -- who're commited from root btw? :) Add this to end of authors.txt:

(no author) = nobody <nobody>
root = root <root>

Then run:

svn2git https://svn.blender.org/svnroot/bf-blender --verbose --authors authors.txt --exclude '.*trunk/lib.*' --exclude '.*tags/[A-Za-z0-9-.]+/lib.*

Basically, it starts running nice but at some revision fails because of some issue in git-svn. Currently checking if the same issue happens with newer git or not.

P.S. Generating authors.txt takes time, so here's current version of it:

alexk = alexk <alexk>
aligorith = aligorith <aligorith>
antont = antont <antont>
aphex = aphex <aphex>
apinzonf = apinzonf <apinzonf>
aramis_acg = aramis_acg <aramis_acg>
artificer = artificer <artificer>
ascotan = ascotan <ascotan>
bdiego = bdiego <bdiego>
bebraw = bebraw <bebraw>
ben2610 = ben2610 <ben2610>
benjycook = benjycook <benjycook>
billrey = billrey <billrey>
bjornmose = bjornmose <bjornmose>
blendix = blendix <blendix>
briggs = briggs <briggs>
broadstu = broadstu <broadstu>
broken = broken <broken>
campbellbarton = campbellbarton <campbellbarton>
cessen = cessen <cessen>
chingachgook = chingachgook <chingachgook>
cmccad = cmccad <cmccad>
cyborgmuppet = cyborgmuppet <cyborgmuppet>
dail = dail <dail>
damien78 = damien78 <damien78>
dan_w = dan_w <dan_w>
desoto = desoto <desoto>
dfelinto = dfelinto <dfelinto>
dingto = dingto <dingto>
djcapelis = djcapelis <djcapelis>
djd = djd <djd>
dna = dna <dna>
domino = domino <domino>
dougal2 = dougal2 <dougal2>
eeshlo = eeshlo <eeshlo>
elubie = elubie <elubie>
eman = eman <eman>
ender79 = ender79 <ender79>
eof = eof <eof>
erwin = erwin <erwin>
farsthary = farsthary <farsthary>
frank = frank <frank>
gaiaclary = gaiaclary <gaiaclary>
genscher = genscher <genscher>
goofster = goofster <goofster>
gsrb3d = gsrb3d <gsrb3d>
guignot = guignot <guignot>
guitargeek = guitargeek <guitargeek>
h_xnan = h_xnan <h_xnan>
halley = halley <halley>
hans = hans <hans>
harkyman = harkyman <harkyman>
hcube = hcube <hcube>
hos = hos <hos>
howardt = howardt <howardt>
ianwill = ianwill <ianwill>
imbusy = imbusy <imbusy>
intrr = intrr <intrr>
irie = irie <irie>
jaguarandi = jaguarandi <jaguarandi>
jandro = jandro <jandro>
jason_hays22 = jason_hays22 <jason_hays22>
jbakker = jbakker <jbakker>
jbinto = jbinto <jbinto>
jensverwiebe = jensverwiebe <jensverwiebe>
jesterking = jesterking <jesterking>
jhk = jhk <jhk>
jiri = jiri <jiri>
joeedh = joeedh <joeedh>
jwilkins = jwilkins <jwilkins>
kakbarnf = kakbarnf <kakbarnf>
kanttori = kanttori <kanttori>
kazanbas = kazanbas <kazanbas>
keir = keir <keir>
kester = kester <kester>
khughes = khughes <khughes>
kjym3 = kjym3 <kjym3>
kupoman = kupoman <kupoman>
kwk = kwk <kwk>
larstiq = larstiq <larstiq>
leifandersen = leifandersen <leifandersen>
letterrip = letterrip <letterrip>
lfrisken = lfrisken <lfrisken>
lmg = lmg <lmg>
lockal = lockal <lockal>
loczar = loczar <loczar>
lonetech = lonetech <lonetech>
lukastoenne = lukastoenne <lukastoenne>
lukep = lukep <lukep>
lusque = lusque <lusque>
maarten = maarten <maarten>
maike = maike <maike>
mal_cando = mal_cando <mal_cando>
mdewanchand = mdewanchand <mdewanchand>
mein = mein <mein>
merwin = merwin <merwin>
mfoxdogg = mfoxdogg <mfoxdogg>
mfreitas = mfreitas <mfreitas>
mfreixas = mfreixas <mfreixas>
michel = michel <michel>
migius = migius <migius>
miikah = miikah <miikah>
mikasaari = mikasaari <mikasaari>
mindrones = mindrones <mindrones>
mmikkelsen = mmikkelsen <mmikkelsen>
moguri = moguri <moguri>
mokazon = mokazon <mokazon>
mont29 = mont29 <mont29>
mosani = mosani <mosani>
mxcurioni = mxcurioni <mxcurioni>
n_t = n_t <n_t>
nazgul = nazgul <nazgul>
nexyon = nexyon <nexyon>
nicholas_rishel = nicholas_rishel <nicholas_rishel>
nicholasbishop = nicholasbishop <nicholasbishop>
nicks = nicks <nicks>
nlin = nlin <nlin>
phabtar = phabtar <phabtar>
phaethon = phaethon <phaethon>
phase = phase <phase>
phlo = phlo <phlo>
pidhash = pidhash <pidhash>
psy-fi = psy-fi <psy-fi>
quorn = quorn <quorn>
red_fox = red_fox <red_fox>
redtriangle = redtriangle <redtriangle>
rohith291991 = rohith291991 <rohith291991>
root = root <root>
rwenzlaff = rwenzlaff <rwenzlaff>
sateh = sateh <sateh>
schlaile = schlaile <schlaile>
scourage = scourage <scourage>
seldan = seldan <seldan>
sergof = sergof <sergof>
sgefant = sgefant <sgefant>
shul = shul <shul>
shuvro = shuvro <shuvro>
sirdude = sirdude <sirdude>
sjoerddevries = sjoerddevries <sjoerddevries>
smerch = smerch <smerch>
snailrose = snailrose <snailrose>
stiv = stiv <stiv>
svbrand = svbrand <svbrand>
theeth = theeth <theeth>
themyers = themyers <themyers>
ton = ton <ton>
trumanblending = trumanblending <trumanblending>
unclezeiv = unclezeiv <unclezeiv>
vekoon = vekoon <vekoon>
venomgfx = venomgfx <venomgfx>
vino = vino <vino>
xat = xat <xat>
xercesblue = xercesblue <xercesblue>
xglasyliax = xglasyliax <xglasyliax>
xiaoxiangquan = xiaoxiangquan <xiaoxiangquan>
yukishiro = yukishiro <yukishiro>
z0r = z0r <z0r>
zaghaghi = zaghaghi <zaghaghi>
zanqdo = zanqdo <zanqdo>
zer000 = zer000 <zer000>
zuster = zuster <zuster>
(no author) = nobody <nobody>
root = root <root>