BIRCH/Forking /home/psgendb from development version

From Bioinformatics.Org Wiki

Rationale: We need to have two different standalone copies of BIRCH, a Production Version for CCU Unix (which will be indistinguishable from any installed copy of BIRCH) and a Development version, which will be kept synched with SVN.

• /home/psgendb version -
  • This is an install of the current stable BIRCH for CCU users.
  • Installed like any other copy of birch, using getbirch
• Reference copy - found in /home/psgendb/BIRCHDEV {we do NOT call this BIRCH, to distinguish it from all other BIRCH installations.}
  • All development copies on all machines are almost 'equal'. All of them are updating from the repo and commiting back to the repo. This eliminates the need, most of the time, for worrying about psgendb group access.
  • The one unique thing about /home/psgendb/BIRCHDEV is that this is the copy that we use to generate framework_xxx.tar.gz files. It has the hard paths that will be replaced, by birchconfig and by customdoc.py, when a copy of BIRCH is installed elsewhere.
  • Probably need to create a new group called 'birchdev'. Members of this group can ONLY touch the repo, and not anything else in /home/psgendb.

The big issue - hard coding of '/home/psgendb' in birch files and scripts

• birchhome.sh
• customdoc.py
• /home/psgendb/local/admin/*.param files - These files need to be checked to make sure that '/home/psgendb' gets substituted with '/home/psgendb'
• birchconfig is hardcoded to prevent installation in /home/psgendb/BIRCHDEV!

The end result is that /home/psgendb will become just another installed site. No development will occur on /home/psgendb. All testing and development will be done on /home/psgendb/BIRCHDEV or on other test sites. /home/psgendb/BIRCHDEV becomes the definitive reference copy.

Contents 1 Status 2 To Do 3 Consideratons 4 GIT 5 Roadmap

Status

Currently, there are two problems:

• The birchdev framework is to big
  • The solution to this is to make the binaries into a submodule.
• The birchdev framework currently lives on hake.cs.umanitoba.ca
  • We want to move it to git.cc.umanitoba.ca to consolidate development.

To Do

1. Recreate the repository to not include the binaries, and make these binaries into a submodule repository. This will make the framework small enough.
2. Make biolegato into a submodule repository.
3. Identify other submodule repositories.

Once we have the framework setup on git.cc.umanitoba.ca:

1. Add the metastore script to it
2. Make winbirch and getbirch changes
3. Create submodules for documentation, etc. Prevent commits to public_html making it a separate module.
4. Add a .nogetbirch lock file to the framework on birchdev, add this to git ignore.

Consideratons

1. Hard coded values of /home/psgendb
2. One other thing occurred to me. Please make a note on the SVN page that we most certainly do NOT want to call the development directory for BIRCH /home/psgendb/BIRCH, since $HOME/BIRCH is also the default location for a BIRCH install.
3. BIRCHDEV/local - we have to have a BIRCHDEV/local if BIRCHDEV is to be a working BIRCH installation.
   1. We probably don't want this to be in the Git repository
   2. we don't want it to be packaged into the framework.tar.gz file.
4. Where do we put the directories that are involved in creating a BIRCH release?
   1. build.deprecated
   2. getbirch scripts?
5. bin and lib directories - Reference copies of some bin-xxx-xxx and lib-xxx-xxx directories are maintained on other machines. eg. bin-osx-x86_64 is maintained on albacore. What do we do about that?
6. birchconfig and other programs that currently will run as userid 'psgendb'. That will have to be changed so that the test is to make sure that the target directory is not /home/psgendb/BIRCHDEV.

GIT

Git offers a solution to meta data storage that doesn't exist for SVN called metastore

I have writen by own version of metastore, which I have posted as an attachment (I have also added it to a git repository)

This, combined with Gits speed advantage over SVN, improved branch/merging, and the ability to work remotely and push changes asynchronously makes it an appealing alternative.

Roadmap

#Create a good stable BIRCH2.8.5 using the existing development protocols (eg. no subversion, makeframework.csh etc.) and get it to a releasable stage. That copy will be the basis for the new SVN repo, when we create it.

1. 1. First create a Development version and test ot make sure it installs and works, both with birchconfig and with GetBirch
   2. Run makeframework.csh again to create 2.8.5
2. Create Reference copy of BIRCH for /home/psgendb/BIRCHDEV
   1. Move bin and lib directories to BIRCHDEV.--DONE
      1. run makebin.csh to create up to date bin files
      2. use mv to move bin and lib files to BIRCHDEV<
      3. in /home/psgendb, reinstall bin and lib files from the FTP site
   2. --DONE Create a framework_D.tar.gz file using makeframework.csh, but without reading makeframework.excludefile. Also, we modify makeframework.includefile so that ALL of admin and pkg directories are included. That way, ALL files and directories in BIRCH that are normally in the development tree come along for the ride. A lot of non-essential stuff has been moved to /home/psgendb/local/java, so there aren't a lot of stuff left in the exclude file. We still need to sort out where axis2 and biojava should go, but for now, they'll come along.
      1. What are the consequences of moving axis2 to local? Would that primarily affect Netbeans?
      2. Do we actually use biojava for anything?
   3. Fresh install of BIRCH in /home/psgendb/BIRCHDEV. --DONE
      birchconfig will do a lot of the work for us, changing /home/psgendb to /home/psgendb/BIRCHDEV in most of the right places.
   4. --DONE manually re-create links to GenBank and GenPept
   5. --DONE manually copy files from /home/psgendb/local to BIRCHDEV/local.
3. Revise scripts in build.deprecated to use BIRCHDEV --DONE
   1. mv build.deprecated BIRCHDEV
   2. scripts may need little or no change
4. Revise install process
   1. -- DONE replace /home/psgendb with /home/psgendb/BIRCHDEV in scripts and birchconfig. This will require getting Netbeans to work with the copy of birchconfig in BIRCHDEV
   2. --DONE Modify birchconfig and UNINSTALL-birch.sh so that they can't clobber BIRCHDEV. (Currently, won't allow you to run as user psgendb.)
5. Clean up instances of /home/psgendb
   1. --DONE oldstr.param - This one requires careful consideration. Currently, line 3 of this file points to a symbolic link, but it looks like IST has eased up on the use of links, so it may be possible to have the same thing on lines 2 and 3. If we decide to change it, we have to run customdoc.py to change those lines in the web pages, and then we change oldstr.param to correspond to that change.
   2. --DONE scripts (ok)
   3. --DONE birchdb (ok)
   4. --DONE web documentation
6. import this copy to create a fresh repo.
   1. Wiki documentation should probably be reorganized to reflect how developers go through development cycles.
7. Test it thoroughly
   1. --DONE: Successful new install on linux-x86_64 using birchconfig.
   2. --DONE: Successful update using birchconfig
   3. --DONE: htmldoc.py correctly incorporates lbirchdb components into documentation
8. Do some cleanup of log files, old versions of programs etc.
   1. acedemo, birchdb, lbirchdb, l-gbirchdb: acedb seems to create a lot of scratch files that never get deleted. How do we leave those behind when creating a release?
9. Create a new framework file using buildrelease.py
   1. must preserve permissions and dates on files
   2. we probably don't want things like CVS and Git directories to appear in the framework
10. Create local production copy of BIRCH for CCU.
    1. rename official BIRCH directories (eg. admin, java etc.) with .bak extension (effectively doing what uninstall.sh would do)
    2. Run birchconfig to install the current stable release of BIRCH