Project Build Specification

The project build specification is designed around Phil Lord's build specification at BuildSpec

Abstract

The following defines a standard method of defining build scripts to avoid code replication

Requirements

• myGrid is being developed on different sites, by different people
• Although myGrid should exist as a single entity, it is also clear that different parts of the project will be useful to other groups.
• Other groups should be able to use parts of myGrid without getting the whole thing

From this it appears therefore that myGrid should consist of a number of modules, which are largely independent, or where dependencies are clearly enumerated.

• In order to have a unified build however, the different modules need to have a build system which can be unified, as well as operating "stand alone"
• Many of the modules will share external dependencies. These should not be duplicated between them, as this will lead to both download bloat, and also, however careful we are with version numbers, external dependency version conflict
• "Stand-alone" build should also allow developers to build just the subsystems within myGrid that they are actively developing, to reduce the time spent on the code-compile-test cycle. This will also reduce the amount of space they need to use, if they use multiple work file copies of their modules, as some people like to
• The build system should also be the common point of entry for the unit test system.
• It may also be appropriate to use it as a common point of entry for the unit test system.
• Likewise deployment and installation. And possibly even launch of the system.

Overall Design

The best way to address these issues is to define a single unified directory structure which all of the projects should follow. There will be a globally defined build file this will have preset attributes. Each project should also design a single build file with standard entry points (targets) that they wish to ammend or add to the global build file. These individual build files can then be directed "from above" by the top level build file to perform a unified build. Here I call the individual build files "module build files", and the top level build file "the main build file".

This addresses a number of issues, the subsystems each have a local build file for unit testing and for allowing distribution of each component. The build file, both in module and in the main file, become unified in design and structure and allows for more familiarity. It wis also remove the duplication that is currently occuring within the build files. This allows for an easier updating process and management

The method that I am going to employ is using a perl script to combine the main build file and each of the modules build files to generate a build.xml file for each module. The perl script will work the xml files being tagged to allow matching.

XML Tagging Structure

The XML tags will be of the follwing syntax

<!-- BEGIN tagName -->

<!-- END -->

Where tagName is the name defining the current tag. The names of the tags are based on the following format

• Project Name - projectName
• Property Files - propertyFiles
• Properties - propertyProp
• Task Definitions - taskDef
• Paths ID's - idPath
• Target Names - nameTarget

Directory Structure

There are any number of potential directory structures, but most of the projects that I have seen are broadly similar. The main consequences of the directory structure relate to the build files, and hopefully no one will care enough about it, to find it worth disagreeing.

The basic directory structure that I am proposing for myGrid is shown in the following table. I have somewhat mixed up end directory structure and directory structure in the CVS. Those directories marked with † should be present in the CVS, those without will be generated directories.

Directory or Filename	Usage
ext/	All external dependencies should be found here
Licenses/ †	Location of all the license documents should be here. There may be quite a few of these, as external dependencies will need them
doc/ †	Top Level documentation which applies to all modules. Would probably include stuff like this document, general help documentation. These should be in text format or pdf only. Also probably an "AUTHORS" document.
javadoc/	Unified Javadoc repository, created from all of the source.
build/	Used for temporary files during the build. What this includes will depend on the "build mode".
build/test-results/	JUnit test results should go in here.
build/classes/	Generated class files.
lib/	For jar files. What this includes will depend on the "build mode".
samples †	Examples which run and do cute things.
module-n †	The various modules that we need to do things.

Standard Targets

There will be a number of standard build targets that each module build file will need.

Target	Depends	Usage
compile	none	Compile everything that needs to be compiled for functional code. Not documentation. Compile means generate .class files, and not .jar files.
doc	none	Generate any documentation which needs generating.
javadoc	none	Generate javadoc.
jar	compile	Package everything up into .jars in the lib/ directory
dist	jar	Generate all of the distribution files, both zip and tar.gz
test	compile	Run complete test sets on files in build/ directory.
clean	none	Delete everything that is generated. This should return the system to what appears out of the CVS.
deploy	compile	Create the web apps structure within the build
install	deploy	Install webapps to tomcat, using the admin upload client. There are also reload, remove, and list targets using the ant tasks shipped with tomcat
gather	prepare	Gathers all of the required jars to ext/

A couple of these target could probably do with some further subdivision, in particular "clean", which probably needs a separate "clean_classes", which I've found very useful during development. Likewise dist, could depend on dist-zip, and dist-tgz.

Using the module build

I have included links to both the build-default.xml and an example build-local.xml at the bottom of this document. The build-default.xml will be used if no content is defined in build-local.xml. To create a build-local.xml see which items you want to ammend, replace or remove from the build-default.xml. If you wish to ammend a tag copy insert the relevant

<!-- BEGIN tagName -->

<!-- END -->

into the build-local.xml
Anything in the tags will overwrite the matching tag from the build-default.xml. So if you wish to remove a tag place an empty tag with the same name in the local file.

There are also two sections for the addition of extra properties and targets called additionalTarget and additionalProp.

The required files for the merge are located in the merge module in the myGrid CVS repository To generate a build.xml run the perl file merge.pl using the syntax:
merge.pl local-buid-file default-build-file output-file

I have not hard codeded any of the inputs or outputs to allow for testing and flexibility. When people are happy with the format and the method, I will change CVS to automatically create a new build.xml each time either build-local.xml or build-default.xml is checked in.

Once the build.xml has been generated the module can be built in the standard way as defined above.

Automated top level build

Anthill will be used - Link here

-- RichCawley - 27 Nov 2002