Added by Stian Soiland-Reyes, last edited by Stian Soiland-Reyes on 2009-04-07  (view change)

Labels:

Enter labels to add to this page:
Wait Image 
Looking for a label? Just start typing.

This is Stian's proposed structure of the Taverna Workbench 2.x (before t2 platform).

The code is to be hosted at Google Code's Subversion at http://taverna.googlecode.com/svn/ and follows the recommended multiple project layout, ie:

project1/
    trunk/
        src/
        pom.xml
    branches/
        stians-experimental-stuff/
            src/
            pom.xml
    tags/
        2.0/
            src/
            pom.xml
project2/
    trunk/
    branches/
    tags/

The main difference from CVS to Subversion is that there's just one big directory structure, and tags and brances are done directly in the tree as directories - purely by convention. trunk is the conventional name for what in CVS would be called HEAD, the front of the development, the newest code. In a Maven world, that's where you would find 2.1-SNAPSHOT for project1, while the released (and probably deployed) version 2.0 is under tags/2.0. See the Subversion book for more information about Subversion. 

To get write-access to the Subversion tree (ie. access over https), contact Stian. (Not really needed until the restructuring is sorted out.)

Philosophy and practices

I'm assuming/preaching:

  • The engine and the workbench are separate units
  • We don't need to check out/compile/redeploy modules we're not changing (and that are not affected by our change)
  • We should primarily do (many) online updates of activities, workbench components, etc, not big releases

How? First of all we have to think about how we work.

  • Don't check out everything
  • Run nightly tests

Hudson

If that's going to work, someone else needs to do that and deploy to our snapshot repository. That someone used to be Stian when he had a lot of spare time, but this has turned out to not be very reliable. The new replacement is called Hudson.

Now we've had Hudson for a while, but now he's finally doing some real work. Every night he will check out from Subversion the latest versions of various bits of the code, and start building it, testing it, and if everything works, deploy the snapshots.

So if you have checked out only some components of the workbench, Maven will fetch the rest from the snapshot repository. As they were built last night, and Maven checks again every day if there's a new snapshot, you should automatically be in the game, or at most one day lagging behind the code in Subversion. And if some radical changes makes an old snapshot not work anymore, it's always possible to go to Hudson and press the "Build" button manually to force a new snapshot.

In theory we can also get Hudson to send complaint emails to us pointing out who (probably) broke the build, as he watches the Subversion logs and detects changes - even if they were in an upstream project who's own tests works. (This should for instance cover the case where you modify t2core and add a method to an interface, but forgot to update one of the activities).

Eclipse plugins for Subversion and Maven

Everything in this document assumes you are using Eclipse 3.4.1 with the plugins Subversive 2.0.4, SVNKit 1.2.0 and Maven integration (m2eclipse) 0.9.7 installed. I've found that the alternative Subclipse plugin does not handle very well moves, trunk/tags/branches etc. For instance in Subversive you can just click and say "Create tag" and it will do the correct thing, while Subclipse just gives you a form where you have to fill in the URL - much more error-prone and not much benefit from using the command line.

Both of the SVN plugins uses the normal .svn structure locally, so if you have the command line tool svn installed, you can also do svn-commands on the command line. It's normally good to do a Refresh in Eclipse if you've done this to not confuse the plugin too much. I recommend to make sure the svn command is version 1.5 or later to support relative paths in svn:externals (used by the builds module), but note that this has made some changes to the file-format of .svn/ directories, forcing you to use SVNKit 1.2.0 in Eclipse instead of the older 1.1.7. I've not used the native bindings in Eclipse (JavaHL) - but I know at least in 32-bits windows these should also work fine. You can modify which Subversion implementation Subversive uses under Preferences.

Do not use the very outdated Eclipse plugin for Maven (mvn eclipse:eclipse) - this only builds a rough .project and .classpath file and does not help you much - instead use the m2eclipse Maven plugin for Eclipse.

Use m2eclipse efficiently

with "Separate modules for projects" and "Workspace resolution"

We've previously checked out large modules in Eclipse, and use the m2eclipse plugin for Maven support. Because the code was not very well structured before, we generally used the "Enable nested module" option and turned off "Workspace resolution". This had several disadvantages:

  • Needed to do mvn install on moduleA if we modified moduleA/blah/bluh and we want to test/use this in moduleB
  • Anything broken in moduleA meant we could do anymore work in moduleB
  • There was one big classpath container per module, and we used imports that were not included in the POM, it worked anyway because some other sister module had included it, and we only discovered the compilation problems when (if!) we did a mvn install
  • You got 20 different src folders and had to manually keep track of where your code was

Now if we use the m2eclipse plugin the way the author's have intended it to be used, we will use the right click and "Check out as Maven project" on the trunk folder (important - otherwise you'll check out all tags and branches as well!) from the SVN repository panel, and in the wizard that pops up, make sure both "Resolve workspace projects" and "Separate projects for modules" are ticked.

This will create one project per module. So for instance if we do this on http://taverna.googlecode.com/svn/taverna2/utils/net.sf.taverna.t2.lang/trunk/ we would not just get a lang project, but also the children io, observer}, {{partition and ui.

Now, this will be a lot of projects, you say, and that's correct - if you check out everyting. But still even for your areas of interest it could be a lot, so in the wizard selet "Add projects to working set" and create a new working set that you call "utils" or something like that. I've generally named the working set after the main module I've checked out - ie. the one that has the trunk/tags/brances folders.

Once you're back in the Java perspective (I've found that the Java EE perspective gets a bit confused by working sets), and the import is finished, you might notice that all your projects end up in "Other projects" - or that you have lots of projects. First of all click the little triangle in the Package explorer and select Top level elements -> Working sets. Then click again and go to Configure working sets. In here make sure at least the newly added working set (ie. utils) has been ticked. I generally have all of them ticked.

This should expose a tree in the package explorer, and you can open and collapse "Utils" to see io, lang, observer, partition, ui.

Each of the projects would have 1-4 source folders (src/main/java, etc) and a "Maven dependencies" container. Now, open this container on ui, and you should see that it lists log4j-1.2.12.jar and observer. Notice how the observer is referenced as a project rather than a JAR-file in your .m2/repository. This means that you can modify something in observer, and it would immediately be visible inside ui. For instance, modify Observable to have an additional method public void fish();. Click Save, and after a quick build you should see a red icon appearing down in ui as well to indicate tat you have not implemented the fish() method in ModelMap.java. This is the way most people develop code with Eclipse, and you should be glad to know that this works across the boundaries of our modules. So if you happened to have checked out workflowmodel-impl (but not workflowmodel-api), and you have this project open, then M2eclipse will automatically use this project instead of the JAR-file for workflowmodel-impl, and the JAR for workflowmodel-api.

This comes down to opening and closing of projects. I've found out that it's often good to close the projects I'm no longer working on, they won't clutter my display, and I know I will be using the JARs I have built with mvn install (or the ones downloaded from the nightly snapshots). You can try this, just close the observer project, and you should see in ui that the dependency changes to observer-0.5-SNAPSHOT.jar - and the red problem about the missing interface is gone again.

Now, just open observer again and let's first revert that change, and then save again - the red colours in ui should appear and then disappear again - and we've not done a single mvn install yet. You'll also notice that you can right-click on a whole working set and say "Open" or "Close" - this would affect all the projects in that set - you might find this useful if you are working on separate bugs at the same time. To force download of snapshots from the repository on a project, right click on it and select Maven -> Update snapshots.

On the other hand, if you have a project with modifications that you have not yet committed - you might be trapped on the downloadable nightly snapshots being newer than the patched ones you have done with Maven install. To avoid doing Maven install every day - simply keep that project open as well - the project dependency would be used instead of the snapshot JAR. Or a better solution - either commit your changes every day - or if it will break stuff - make a branch and change the version number to something like 0.5-myExperimentPatch-SNAPSHOT. This would allow other people to work with you. Note that you will have to register the branch separately in Hudson to get unit testing and snapshots of your experimental code. For such work you can even ask Hudson to pull from Subversion and build on demand as soon as something new has checked in.

Note that the Eclipse plugin can do the mvn install for you, even on a module by module basis. Right click on observer and see Run as --> Maven install. This should initiate the Maven install for you. After some Maven installs it might be useful to do a Refresh on the project, at least for projects where Maven will generate some code.

You will notice that our friend clean install is not in the Run menu. Don't despair, instead of doing Maven clean, waiting, and then Maven install, you can select Maven build - the first time it will pop up a window asking you which goals to run. Instead of trying the button that lists all possible plugins in the world, simply type in clean install here. The next time you do Maven build on the same project the plugin will remember this bit and just do a clean install.

Now, let's make sure observer is closed. Now go to ModelMap in ui, and right-click on "implements Observable" to do Open declaration. You would probably see something like "The JAR file .. has no source attachment". One way around this is to just open the project. But let's assume you had not checked out observer. Instead, right-click on ui and select Maven -> Download sources. You will notice that the sourcecode for Obserable appears on the screen. You can't edit the code though, but you can browse around it just as usual, for instance "Open declaration" on Observer, or using References->Workspace on Observer.

Now let's say you look at this and you do identify a possible bug. Right cilck on observer-0.5-SNAPSHOT.jar inside Maven dependencies (that's where you can browse the classes), and you'll see tat you could have done "Download sources" here as well on a JAR by JAR basis.

Additionally, there's "Import project". Now this is pretty cool. Click this,  and just go through the wizard. Note in "Select Maven projects" that /pom.xml is grayed out - that's because there's already a "observer" project.

This gives us the opportunity to go to Advanced and look at Name template. In here, select [groupId].[artifactId] - or if it's not there - [groupId].[artifactId]-[version]. This means the project will instead be called net.sf.taverna.t2.lang.observer - or net.sf.taverna.t2.lang.observer-0.5-SNAPSHOT.  I've found this longer form to be more useful when you have many projects checked out, as you will then be able to identify it properly in other project's Maven dependencies list - and in case you forgot to put it in the right workset. Now click Refresh and the box next to /pom.xml should be ticked. Click Finish - and you will notice that observer-0.5-SNAPSHOT.jar in the dependencies has been replaced with a project called net.sf.taverna.t2.lang.observer. Right click and select Go to project - and you will see that you have now checked out observer directly from Subversion. Now if you had ticked Use developer connection it would have used the https link so that you could also commit back in.

In all this means that you don't neccessarily need to browse around the Subversion repository hunting for what you need - if you have already got a dependency to it you can just click on it and get the sourcecode directly. Now for this to work there needs to be correct <scm> information in each of the main modules, so the plugin knows from where to download it. You will find that some open source projects, such as spring, also have their sources available on "Download source" which is very valueable in debugging.

I'm also thinking to build a virtual project that depends on everything else so you could use this as a starting point for such things as seeing who is implementing an interface, or how it's been used. (and possibly even run the workbench with the eclipse-true-hack) All our source code should have the source JARs deployed, thanks to instructions in the parent pom - so it means any developer would have the very same benefits - perhaps without checking out a single line of code from Subversion.

Subversion structure

See Subversion repository for description of the current structure.