User:Ideasman42/BlenderLibsForLinux

= Building Linux Libs =

This page contains notes on maintaining Linux libraries, it is more a working set of notes and not official documentation.

Things to do on Fresh CentOS7

 * run: `vi /etc/sysconfig/network-scripts/ifcfg-enp0s3` where `enp0s3` may change based on the system. Change line `ONBOOT=yes` (was no). Reboot.

https://wiki.blender.org/wiki/Building_Blender/Linux/CentOS7ReleaseEnvironment
 * Follow steps from:

Add public SSH key
Needed to log into the guest from the host.

``~/.bash_profile``:

SSH Into the Guest OS
While not essential, in general I find it much more convenient to SSH into the system, it allows copy/paste on-screen text, larger terminal display and doesn't lock the cursor to the window as VirtualBox does.

GUI-Less VirtualBox (host)
Running GUI-less is handy, allows to quit and restart the graphical session without interrupting the VM.

The VM can be started from a terminal as follows:

Run TMUX (guest)
Attach or join TMUX session, useful in case the SSH session disconnects (if the terminal is closed for e.g.).

Mount the guest VM's filesystem (from the host)
This is handy because it allows using development environment and tools from the host system, accessing files can be a bit slow, otherwise this is convenient.

Port forwarding needs to be set to forward 3022 on the host to 22 on the guest from virtualbox.

Notes from Sybren
What I do to release new builds is:


 * 1) SSH into my CentOS7 VM.
 * 2) Make sure the Blender sources are good to go (right branch, necessary patches applied).
 * 3) cd deps and make -j12 until done.
 * 4) on the real machine, run `rsync_from_buildbot.sh` the library (will post below) for every library that should be updated.
 * 5) do a test build of Blender and check nothing broke.
 * 6) submit the lot via SVN.

The ``rsync_from_buildbot.sh`` script:

Fish command to perform an SVN-ADD:

Modify SVN Config
`~/.subversion/config`

Otherwise it will also ignore `.a` and `.so` files, which is super annoying when you do the library maintenance work.

Notes from Sergey

 * Sometimes needs clean build.
 * We don't do reproducable builds, so manually delete libraries before copying new ones over.
 * SVN Add can ignore "`.a` so make sure those are added (by default these get ignored).
 * Deps and linking issue: Some deps pick up system libraries. This can cause `make deps` to fail when making deps on the development environment.
 * Building libraries for future releases.
 * Don't start on new libs until stable release has been released (and the current libs are tagged).
 * `build_files/build_environment/linux/linux-centos7-setup.sh` Script should be kept small, make everything part of "make deps".
 * build_files/build_environment/cmake/versions.cmake (is source of truth for deps), if comments need to be added for deps (what they are used for, any general notes).
 * document why deps are needed - nice to have.
 * Multiple Python's versions are kept in SVN (why?, easy bisecting).

Submitting new Libraries
Update the task for libraries https://developer.blender.org/T101403

Don't Remove/Add New Libraries
When updating libraries DO NOT use:

Doing so makes updating SVN fail if the directories contain any temporary files or cache (most common with Python's `__pycache__`), see:

See links where users ran into this problem:


 * https://lists.blender.org/pipermail/bf-committers/2022-November/051431.html
 * https://devtalk.blender.org/t/23102

Instead, delete the directory, copy over the new one, then use `svn st` to list files to be added/removed.

`TODO:` write a script that automates this step.

Patch Building DOES NOT use the latest libraries?
This can trip you up when testing if new functionality works with newly added/updated libs. The patch builder has logic to use the same version of libs when the patch was submitted.

The Solution is to use the branch-build functionality, instead of building patches.

Link Time Optimization (LTO)
Link time optimization causes static libraries not to link when developers build with different GCC versions.

For example:

So while the builds will work (and pass tests) on the VM, this should not be used.

Building One Library at a time with Multiple Jobs
Building a single library at a time makes it much easier to debug when something goes wrong as the output isn't mixed with unrelated libraries.

Building with a single thread has the down-side that each library is build with one job, leaving cores idle and taking a very long time. There is also the problem of some large libraries being a bottleneck causing long build times when other small libraries have finished building.

CMake doesn't provide an obvious way to handle this.

CD'ing into each build directory and running `make -j $(nproc)` is an option but requires manual intervention.

The solution is to run make and pass MAKEFLAGS, so the top-level make is single threaded, but sub-processes use multiple jobs.

The ninja build system (used by `meson`) will already use multiple jobs, so it doesn't need special handling.

Testing Libraries
Changes to libraries should rebuild Blender and pass all tests.

Slow rebuilds from GPU Kernels
A `make release` build is very slow as it builds all GPU kernels, these are not used or necessary for testing `make full` build is sufficient for running tests.