response.getEntity().getStream().close();

or

response.getEntity().write(System.out);

Otherwise the client keeps trying to send to the server but can’t get an open socket to get there.  Another problem solved!

inputs.put(”fileUrl”, LocalworkerTranslator.class.getResource( “/AAC4_HUMAN.sp”).getFile());

you have to ensure that the absolute path is correct by doing

URI uri = LocalworkerTranslator.class.getResource(”/AAC4_HUMAN.sp”) .toURI();

File newFile = new File(uri);

inputs.put(”fileUrl”, newFile.getAbsolutePath());

Also, you can’t expect checks for “\n” style line breaks to work in Windows because they insert things like “\r\n” instead.  So , you should use System.getProperty(”line.separator”) instead.

BrowserLauncher launcher = new BrowserLauncher();

launcher.openURLinBrowser("http://www.google.com");

Job done.

The idea of Raven is quite simple, we use Maven to deploy all our modules, and all of our dependencies are also available either from our own or from official third-party Maven repositories. One thing we decided to support from Taverna 1.5 was plugins and dynamic updates over the web. How could we do this?

We decided to reuse the existing Maven infrastructure also at runtime. To install a plugin would then simply be a matter of specifying which Maven artifacts are needed, and from which Maven repositories to get the artifacts, such as in the plugin description for the t2 preview plugin. So we built Raven, which although inspired by Maven doesn’t use any Maven code.

Given a Maven artifact as described in the plugin file:

<artifact groupId="net.sf.taverna.t2"
artifactId="biomart-activity" version="0.2.0"/>

Raven fetches the POM and JAR files for biomart-activity from the repository, parses the Maven artifact description file biomart-activity-0.2.0.pom, and then does the same for each of the dependencies listed, and with their dependencies again, and so on until everything required has been downloaded.

What remains is to get all these JARs available on the classpath. The normal way to do this is to use a ClassLoader, such as the URLClassLoader. You can’t normally modify the classloader Java gives you at startup, because it has been determined from the -classpath parameter and you can’t officially add new URLs to it once it’s constructed. However you are free to construct a new classloader and load classes from there.

This is what the bootstrapper asks Raven to do. Since we can have several plugins, each plugin would need its own classloader. The plugins can come from different third-parties (you can add plugin sites to Taverna), and they might depend on different versions of common artifacts such as jdom. In some cases these versions are not compatible.

What we ended up with as a solution was that each Maven artifact gets its own LocalArtifactClassLoader. This classloader also have a list of the dependencies as declared in the POM, and when asked to resolve a classname, it will first search each of it’s dependencies. As the dependencies have the same kind of classloaders, they will use the same logic, and the search will be depth-first. If none of the dependencies have the class, then maybe the JAR file associated with this artifact does, so it does a normal search through its own superclass URLClassLoader which searches the single JAR file. If this fails, a ClassNotFoundException is thrown and whoever depended on this artefact would go on to check the next dependency.

When you download Taverna these days, what you get in the zip-file is a tiny bootstrapper class together with a configuration file raven.properties that says which are the global Maven repositories, and which profile should be used for the main program. The profile version 1.7.1.0 basically says what is Taverna 1.7.1.0 by listing all the required artifacts, very similar to the plugin definition. For instance you would find all the different processors listed here.

If we decide that we need to publish an update of a certain processor, we can just deploy the new version to the Maven repositories, and publish a new version of this profile and list it in the profile list that Taverna checks on startup.

Although this solution came with a few quirks we had to work out (for instance many of the official POMs didn’t correctly state what where the true dependencies of the library), in general it’s a nice solution with lots of possibilities, for instance it doesn’t prevent you to have two versions of the same artifact at once, so you could theoretically build a workflow that used both an old and a new version of the wsdl-activity - if for instance we had fixed a bug that you suspect might have affected some old workflow runs and you want to compare the outputs.

However, one of the difficulties with doing all of the ClassLoader work ourself is that as this is quite low-level and hard-core Java with many tiny little pitfalls to worry about and lots of concepts to understand. I’ll come back in the next post with the example related to our bug TAV-480.

If you are interested in using Raven in your own project, contact us on taverna-hackers!

You can download 1.7.1 from SourceForge.

This update to Taverna resolves a number of issues with Taverna and the preview of the new t2 enactor.

For instance, you should now be able to run a workflow in the Taverna 2 preview perspective and see the progress graphically with progress bars. If you run a sufficiently advanced workflow, such as one of Paul Fisher’s workflows- you should be able to see the streaming of t2 in effect, where a processor downstream will start iterating over the results even if the one above has not yet finished. This alone should be able to give a nice speed-up to workflows iterating over many items.

We’ve also reintroduced the interactive diagram editor to complement the normal, static diagram. Now, we know it’s still not 100% stable, so that’s why there is that little “experimental” there in the title, but many users find this much easier to work with. You can simply click on a processor in the diagram to select it, right click, select Connect output and draw a line to the processor you want to connect. You can even make connections the other way around, if you know you need an input, but don’t remember quite sure from where! You should also be able to drag services from Available services and onto this diagram to add them to the workflow.

Although this won’t be the editor you will see in Taverna 2.0 that we have scheduled for end of June 2008, but we’re aiming for a similar way to edit workflows, as one of many, as we’ll shine up the graphical user interface to go better with the new enactor. We’ll also now be working on security aspects, such as support for various grid systems, as you might have seen discussed on taverna-hackers.

If you’ve been using nested workflows and have had complains from your friends on myExperiment because their Taverna can’t find F:\workflows\nested\mynestedworkflow.xml - 1.7.1 has a solution for you. When you save a workflow, tick the Embed nested workflows checkbox, and all of your nested workflows will be embedded in the saved Scufl instead of referenced by filename.

Note that if you actually find the referenced nested workflows useful, say because you update the nested workflow on disk and want the new version to be used in all the workflows including that nested workflows, make sure you save the workflow without the checkbox ticked to keep the references. We’ll be looking at further improving the GUI so that it’s more obvious when a nested workflow is included by reference (ie. filename) or embedded - but expect this to look slightly different for Taverna 2.0 anyway.

The t2 plugin is now able to render results such as images and JMol data as in Taverna 1. Note that we’re still working on improving the user experience in the t2 result browser, as it’s still not optimal. For instance you currently need to expand the data document, select the first entry, right click and select Render as image.

Under the official plugin list we’ve included the GT4 processor and scavenger contributed by Wei Tan and Ravi Madduri of the caGrid project, note that this is still in it’s early stages as we’re sorting out issues to do with WSRF, security, etc. in tight cooperation with caGrid.

If you’re missing the alternative way to find services of Taverna 2 Activity Palette that was introduced in 1.7.0, you can also find this under available plugins, we chose to no longer include this by default as Taverna 2.0 will include a different activity palette and it would not be a realistic preview.

The source code download of 1.7.1 should also be more complete including the code for the plugins and the t2 preview - although we still recommend doing an anonymous CVS checkout.

We’ve resolved these issues according to Jira:

1.7.1 ( 2008-04-07 | Release Notes) Spring release 2008 - presumable last release before 2.0
		TAV-553	FIXED	Fetching results unpredictable
		TAV-709	FIXED	T2 Enactment Error with pauls workflow
		TAV-700	FIXED	Column ‘thing’ too short in LogBook database schema
		TAV-735	FIXED	Improved workflow editor adding improved drag and drop abilities
		TAV-716	FIXED	Remote execution service using outdated Taverna profile
		TAV-721	FIXED	T2 Valdation fails for processors containing merged ports.
		TAV-732	FIXED	documentation is out of date
		TAV-740	NOT A BUG	Workflow example directory does not replace previous installation
		TAV-694	BY DESIGN	t2: Conditionals fire even on errors
		TAV-737	FIXED	Allow multiple implementations of the Monitor interface.
		TAV-682	FIXED	Disconnected outputs fails workflow
		TAV-726	FIXED	Mac OS X plist file improvements
		TAV-664	FIXED	Serialise bean to stream instead of file
		TAV-724	FIXED	T2 FileDataManager and FileBlobStore uses File.mkdirs which isn’t thread-safe
		TAV-736	FIXED	T2 Renderers
		TAV-710	FIXED	T2 input dialogue doesn’t open after an error
		TAV-244	FIXED	The execution version of the workflow should be the same as the edit version.
		TAV-731	FIXED	goViz is not there
		TAV-722	WON’T FIX	Increase default memory size
		TAV-706	NOT A BUG	T2 XMLInputSplitterActivity problem
		TAV-741	NOT A BUG	T2 plugin initially disabled
		TAV-675	BY DESIGN	t2 should support implicit iteration in input port
		TAV-715	FIXED	Avoid Raven/maven dependency for remote execution service
		TAV-680	FIXED	Failing workflows goes to empty results
		TAV-742	FIXED	JTidyWorkerTest fails due to character set assumptions
		TAV-649	FIXED	SVG renderer does not work
		TAV-713	FIXED	Workflow scavenger dropping ports and script content for Rshell
		TAV-679	FIXED	Default “Metadata” tab for input/output ports confusing
		TAV-574	IMPLEMENTED	Nested workflow location
		TAV-719	FIXED	TestJavaProcess failed during Maven build
		TAV-728	FIXED	Add Unknown Host Exception in PluginManager

Well, that’s about what I remember for now, let’s hope you’ll enjoy the release, and as always, report any bugs, issues and questions to our mailing lists.

The problem is that these shims are designed to be use once, almost throw away, so they are not annotated with any sort of helpful information. This becomes a problem with the scientists greatest challenge - provenance and data lineage (OK, I might be being a bit melodramatic about it being the greatest challenge but it is up there somewhere - maybe nearer to the challenge of making the perfect cup of tea, no mean feat). Data goes into a shim and comes out the other side but what is happening inside? Shims are not the only processors guilty of this, there are plenty of black box services out there in the world. So, how can we address this problem? Well, we have started to collect the shims that people actually use in a my experiment group, we will then figure out all the similarities and come up with an annotated set which we can all use from Taverna 2 (T2). The current idea is for the T2 workbench to have an intelligent workflow designer which will recognise that you are trying to do some shim magic and suggest one to use. Maybe we will need a Taverna clippy style pop up (think Word etc) - ideas on a postcard……

So, if you are a shim it’s time to stand up and announce to the world - “I’m a shim and I’m proud of it”.

The work seems very promising, and in particular because it seems to have much of the same inspiration as t2 with regards to security and referenced data, although their plugin is for Taverna 1.6.2, before even the t2 plugin was included in Taverna.

They have two different solutions to how to run ARC grid jobs from Taverna. The first solution can be compared to SOAPlab, where a command line execution (in this case on the grid) is described in a simple XML description, for example for alignment of Protein Sequences with ClustalW:

<?xml version="1.0" encoding="utf-8" standalone="yes" ?>
<program name="clustalw_protein”
     description=”call clustalw”
     command=”clustalw -infile input -outfile output“>
  <output name=”alignment”>
    <fromfile path=”output” />
  </output>
  <input name=”multiple_sequences_FASTA“>
    <file path=”input” />
  </input>
  <RE name=”APPS/BIO/CLUSTALW-1.8.3″ />
</program>

Above I’ve highlighted the commands and the descriptions. These service descriptions, called use cases, are then put onto a repository that can be read by a special ARC scavanger and browsed like services in Taverna. There’s an ARC gateway service that based on these descriptions are able to dynamically generate WSDL-described web services, which Taverna and other clients can use normally, given a grid proxy certificate is used. The gateway reads the annotations from the use case repository, and when a request is submitted to a web service, it will submits the grid jobs and retrieve the results to the client.

This solution should enable any WSDL-capable client to use grid resources without worrying about the command line.

The authors realised that sending potentially large data back and forth over the WSDL interface to Taverna is a big resource waste and limitation, specially when using several ARC services on the same grid in succession. Therefore they developer a second solution, which is a Taverna plugin.

The plugin includes a Use case scavenger for browsing the annotations from Taverna’s Available services panel and adding them to the workflow. There’s a GUI to select the proxy certificate. The services are added as ARC grid processors instead of WSDL processors as in the first solution. This solution does not require the gateway service, as the ARC processor does the grid submission and result retrieval under the hood, similar to the SOAPlab processor already built-in to Taverna.

The crucial difference here is that the plugin allows passing data by reference, if you chain two such processors together, only the reference to the result is passed along (I assume basically a filename), and the big data can stay on the grid. The data is fetched on demand if passed to a non-ARC processors, so the workflow designer is able to combine grid and non-grid resources.

These are two main features we want to achieve in a more general sense with t2. A service should be able to return data by reference, and we fetch the data only when needed, otherwise just pass along the reference. There should be built-in support for secured services, with a GUI to select certificates (no requirement for “magic” files).

We met Möller at ISMB 2007 in Vienna, and we’re currently talking with him and his coders to collaborate on porting their code to t2 and to get more requirements for us, as this is both an interesting and real use case.

Note that it would also be possible to do the pass-by-reference solution with Taverna 1 even without a plugin. All you need to do is to have your service return an xsd:anyURI instead of the result value, and to accept that as input parameters. If you want to support both you can have mirrored methods, say a blast() (using values) and blastRef() (using references).

The URIs themselves can be in the style of:

• urn:uuid:815BBCE0-ED5A-4ED9-8768-22EED2793EC4
  (internal URI based on an UUID)
• http://myproject.com/2008/myservice/data/815BBCE0-ED5A-4ED9-8768-22EED2793EC4
  (http-namespaced, but only internally resolvable URI)
• https://myservice.mygrid.org:8081/data.msf?id=815BBCE0-ED5A-4ED9-8768-22EED2793EC4
  (http-resolvable URL, given valid security credentials)

The disadvantage of this in Taverna 1 is when you want to pass to a non-grid resource you’ll need to introduce a shim that downloads (resolves) the content, for the first two cases this would be a special getData(uri) method on the service (together with a putData(data) method for uploading large data and getting a URI that is valid with the *Ref services), in the last case it could just be the local worker Get web page from URL (for text) or Get image from URL. (OK, that’s another little secret hack, but these two workers should work with not just HTML and JPEGs, the only real difference is if you get text or binary data out.)

What t2 should be able to is to do this under the hood of Taverna and it’s activities (processors), so the only requirement is for the service to say that something it expects or returns is a reference. One of the problems we face is that there’s no way to do so in WSDL currently, but we could try initially with simply saying that xsd:anyURI’s could be used as references.

We’re happy to announce that OMII-UK (the mother organisation of myGrid) has been accepted as one of GSoC’s mentoring organisations. Some of the ideas for OMII-UK development suggests Taverna development, including:

• Taverna and Grimoires Integration
• Extending Taverna to integrate MatLab
• AJAX based Workflow Editor

Additionally, the Globus Alliance is also a part of this years GSOC, so if you fancy something that is even more grid-like, have a look at the Globus ideas.

We encourage all interested students to apply now, as the deadline is in a few days on Monday, March 31, 2008 April 07, 2008!

Note that you can even suggest other development in addition to the proposed ideas, given that it would be relevant for OMII-UK or myGrid/Taverna. For instance, if you fancied doing a 3D Taverna workflow visualisation based on Quake, feel free to suggest that. As far as I understand you can apply for several projects within a mentoring organisation.

We are promoting the programme in general, so if you would rather work for some of the other mentoring organisations, you can of course apply for those instead.

So if you are a student or otherwise interested, have a look now, remember the deadline 2008-03-31 2008-04-07.

Update: The deadline for student submissions has been extended by one week until 2008-04-07.

We recommend that you have a look at the advice for students, and that you submit a detailed plan or description of your proposed work.

There’s a few bug fixes, in addition to quite a bit of work on t2. There will also be a drag-and-drop-able, interactive workflow editor, basically you can edit the workflow by connecting lines between the processors. Those of you who remember Taverna 1.4 might notice that this is something we’ve had hidden in the shelves for a bit, we hope that as we bring it back this time it will be a bit more usable as an alternative way to build workflows. You will still be able to flip to the “Graphical” tab for the classic non-interactive (but usually prettier) diagram.

The t2 plugin will now be a bit more usable with support for BioMoby, we’ve also made the BioMart support use streaming so that as soon as the first row of the BioMart result set has been received, it will immediately be pushed down to the next step in the workflow. Further on, the results of those operations will also be streamed on immediately before the full list is processed. For large datasets this should significantly improve the execution time of the workflow.

The t2 plugin also now has a graphical representation of the workflow as it’s running, with progress bars. Note that when doing streaming the progress bar can look a bit weird as results are coming in - say initially processor 2 has finished 4 of 5 items, but 5 new ones comes in from above, then the progress bar will jump from 80% (4/5) to 40% (4/10). But the cool thing is that you will be able to almost see the data as it’s flowing through, almost like pipes with pumps between them. We’re planning to add more features to this view, say to let you click on a processor as it’s running and have a look, perhaps tweak some parameters or go deeper in detail.

After the 1.7.1 release, which we’re predicting will be out in about a week (2008-04-07), we’ll focus fully on the 2.0 release for June 2008. The 2.0 release will feature the t2 enactor as the core engine, and some graphical updates as well. We’ll try to post more on the progress for 2.0 here as we go along.

The Taverna 2 (T2) development continues with the initial release planned for late June. Although most of the team are collocated on one site (Manchester University, UK), Taverna is an open source project. However, we have not had much “external” development for a while and this is something we will address with T2. The plan is for both user and developer forums nearer the time (real and virtual (maybe via irc)) and to promote T2 and it’s open source ideals at all the conferences etc. that we attend. One of the problems we face is having a governance regime which can handle external contributions in a controlled manner. There are lots of successful open source projects out there and they must have solved this problem. You can look forward to a “Vote for this feature” on the new look Taverna/myGrid website soon(-ish)!

Also, OMII-UK - which Taverna is part of - has been accepted as a mentoring organisation in this year’s Google Summer of Code. Maybe you would like to develop some cool code for us and the open source community!!