BIRCH/Meeting/Jan092012
From Bioinformatics.Org Wiki
Contents |
Overview
Key points:
- Phylip is now building using premake on all platforms, and is the working prototype of a birch auto-building submodule.
- See below for details.
- FASTA building is still in development
- FASTA is taking longer to do the initial conversion to premake because of the **huge** mess of makefiles (one for each platform!)
- FASTA's make framework uses makefiles which are pretty much copies of each other, and only differ in the macro definitions used.
- This is a perfect example of maintaining many files as a result of copying code, instead of a single file with the core semantics (make vs premake).
- GitKit is now successfully running with the only dependency being python 2.6+
- Basic sync action supported, a "safeSync" in development that is a also a permanent backup daemon is in development.
- CVS/SVN integration planned
- Outlines of paper discussed at last meeting prepared.
Phylip and premake
Since the makefiles for phylip have been reverse engineered into premake scripts, we can now generate makefiles for each platform automatically by following the following steps:
- git clone git@github.com:daleha/birch-native-sources.git # the sources used to generate the build artifacts
- git clone git@github.com:daleha/birch_build.git # premake binaries for every platform, as well as the premake source code
- cd birch-native-sources/phylip-3.69
- ../../birch_build/bins/[PLATFORM]/premake4 gmake
- ../../birch_build/bins/[PLATFORM]/premake4 build
That's it! Nothing more needs to be done, the steps to build phylip are now the same on all platforms!
Notes:
- The makefile that was bundled with phylip for osx didn't even work out of box! However, the autogenerated make file made by premake works flawlessly.
- On windows you must first install mingw and msysgit for a posix build environment -though we could use msbuild and visual studio if we chose to. This is however not necessary, as mingw builds native windows code with no posix dependencies, unlike cywin.
BIG QUESTION: When a new release of Phylip or FASTA or whatever comes out, and they change the makefiles, doesn't that break the entire premake process?
The FASTA situation
- The makefiles are pretty much copies of each other, that only differ in the macro definitions used.
- The developer of FASTA definitely pushed 'make' to it's limits... I am surprised it even compiles.
- Currently, I have a few FASTA programs compiling, and now more or less understand the quirks of the FASTA build process. I had to identify which object files needed to be in each static library, and preserve the semantics of the original makefiles (this involved hunting a lot of variables, drawing out dependency trees, etc). Now, I have a good procedure for manually converting gnu make to premake, and have developed a toolkit to assist in the automation of this process.
- I think that when I am done creating the premake script, the original developer of FASTA would be very interested in it, as he/she has clearly struggled a lot with platform-specific make procedures. Likewise, I spoke with the developer of Phylip, and he seems excited to have them deploy. He as also expressed interest in using Git, and is currently using CVS. I would simply need to integrate CVS repo syncing into the git-kit framework.
- A perfect example of human error in make files:
clean-up: rm -f *.o $(PROGS) $(APROGS); rm -rf $(BIN)/*
(there is no assurance that BIN is defined, and hence it deleted every file that I had permission to delete on my system). Luckily, I was using git-kit, and didn't lose much work related to BIRCH, but it clobbered my entire home directory. With automatic make file generation, cleanup actions are much safer
Gitkit
Gitkit is ready for deployment. (demonstrate gitkit now).
- New features in development:
- Workspaces: rather than hardcoding system paths, use workspace relative paths so that you can simply deploy a gitkit.cfg
- Daemon syncing
Winbirch
Winbirch now has phylip, and will have FASTA very soon. After that, we will only need to write premake scripts for any remaining sources, and create a repository for the binary-only distributed programs, then we should be ready for testing and release.
Premake
The steps to making a premake script are pretty simple. You can use an existing premake script as a template, and you basically just have to tell it which files need to be compiled, where they are, and what options they need.
Premake really simplifies cross-platform build quirks. For instance, i ran into some linker errors on OSX for some of the phylip programs, and i was simply able to specify "OSX " and "not OSX" configurations as a quirk work-around.
When a new version of phylip, FASTA, or another source program that we are using premake to build comes out, all that we will have to do is update any rules that have change for the build. If no new files have been added, then there is nothing to do, and the same script can continue to build multiple versions of the program.
In fact, I suspect that the FASTA developers will love our premake framework.
Possible Feature(s) Discovered
Static libraries
I don't think that we will be needing a "lib-xxx-xxx" folder anymore! Rather than ship system libraries with birch, we can simply keep a repository of static libraries (.a files) that each platform needs to link to. This way, the symbols that would normally be looked up at runtime using LD_LIBRARY_PATH will be pulled out of the static library at compile time, and placed into the executable. This will likely improve performance, definitely reduce the size of BIRCH, and reduce the file bloat in BIRCH.
Cross compilation
If we use mingw for all of our windows builds, then we should theoretically be able to build windows binaries on a linux server equipped with the mingw development toolset.
Furthermore, using premake, we can cross-compile linux32 binaries from linux64
This would allow us to build all of our binaries using only 3 machine configurations:
- Albacore
- A linux 64 machine
- The solaris system (for sparc and amd64)
Connecting the Dots
Premake + git-kit + integration APIs (github, cvs/svn) = an open-source, distributed build-automation system designed to maintain open-source submodules.
This applies perfectly to the requirements of the BIRCH framework. BIRCH is naturally made of sub-modules, as it ties 3rd party code together into a framework. By defining these submodules semantically, along with using a modern build framework, will be able to maintain our own branches of open-source 3rd party developer's software. For those programs that are no longer maintained, we would be able to take over as maintainers, and breath new life into dead programs. For programs that are in active development, we will be able to keep current rapidly and with elegant simplicity. If we can link directly in to 3rd party developer's VCSs, we will be able to establish a "birch nightly build", that serves up fresh copies of 3rd party software for end-users and developers alike to benefit from.
But that's not all, git-kit integration make's this a distributed build framework. Literally, with a single API call, a series of linked repositories will be able to produce code in the time it takes to compile it.
I would like to make this aspect the focus of the paper that you requested I outline at our last meeting. BIRCH/Developer_resources/git-kit
To Do
- Get access to a physical windows machine would help with development.
- Buying a windows machine
- Where is it going to go
- Run both XP and windows 7, 8.
- What is the boot configuration? Discussion
- If we can use just virtual machines, that is preferable
- VMware vs Virtualbox?
- Windows 365? cloud.
- Buying a windows machine
Things that I still need:
- Source code for XYLEM and FSAP, and anything else that is missing from BIRCHDEV/install.
- Is the EMBOSS-3.0 in install.old the correct one to use?
Things that would help me:
-
Albacore virtual windows7/windowsXP (these are currently down - I am working with my own windows installations for now) -
Git is no longer installed on albacore for some reason
- mrbayes
- FSAP
- XYLEM
- Weighbor
- TCOFFEE
- weblogo
- mrtrans
- DIALIGN
- cap3
- mapmaker3
- primer3
- Compile the binaries listed above
- Get copies of netblast, sequin
- Windows virtualization
- Reproduce getbirch virtualization error, determine solutions
- Find a way to get getbirch running faster (vmware?)