Among a lot of other tools and technologies we use Lucene here at Delver. Lucene is a very mature, high-performance and full-featured information retrieval library, which in simple terms means: it allows you to add search functionality to your applications with ease. I could spend an entire day talking about Lucene, but instead I'd like to show you how to write scripts for Lucene with Python on Windows (and thus differs from Sujit Pal's excellent instructions on the same subject matter).

First, make sure you have your Python environment all set up -- at the time of writing this I use the Python 2.5.1 Windows installer. Next you'll need Python bindings for the library, so go and grab a copy of PyLucene; to make a long story short I suggest you use the JCC version for Windows which is downloadable here. Grab it and install it, no fussing necessary. Finally you'll need the Java VM DLL in your path, and this depends on which type of JRE you're using:

• If you're using the Java JDK, add the mental equivalent of C:\Program Files\Java\jdk1.6.0_03\jre\bin\client to your path;
• If you're using a JRE, add C:\Program Files\Java\jre1.6.0_03\bin\client to your path.

(As an aside, you should probably be using %JAVA_HOME% and %JRE_HOME% and indirecting through those.)

Now you can quickly whip down scripts in Python, like this one which took about two minutes to write:

#!/usr/bin/python
#
# extract.py -- Extracts term from an index
# Tomer Gabel, Delver, 2008
#
# Usage: extract.py <field_name> <index_url>
#

import sys
import string
import lucene
from lucene import IndexReader, StandardAnalyzer, FSDirectory

def usage():
	print "Usage:\n"
	print sys.argv[ 0 ] + " <field_name> <index_url>"
	sys.exit( -1 )

def main():
	if ( len( sys.argv ) < 3 ):
		usage()

	lucene.initVM( lucene.CLASSPATH )

	term = sys.argv[ 1 ]
	index_location = sys.argv[ 2 ]
	
	reader = IndexReader.open( FSDirectory.getDirectory( index_location, False ) )
	try:
		for i in range( reader.maxDoc() ):
			if ( not reader.isDeleted( i ) ):
				doc = reader.document( i )
				print doc.get( term )
	finally:
		reader.close()

main()

Note the VM initialization line at the beginning -- for the JCC version of PyLucene this is a must, but even like so using the library is extremely painless. Kudos to the developers responsible!

The Java implementation for generics is radically different from the C# equivalent; I won't reiterate issues that have been thoroughly discussed before, but suffice to say that Java generics are implemented as a backwards-compatible compiler extension that works on unmodified VMs.

The implications of this are considerable, and I'd like to present one of them. Lets fast forward a bit and consider a relatively new language feature in Java (introduced, I believe, with J2SE 5.0): autoboxing. A thoroughly overdue language feature, autoboxing allows the seamless transition from regular value types (e.g. the ubiquitous int) to object references (Integer); before autoboxing you couldn't simply add a value to an untyped ArrayList, you had to box (wrap) it in a reference type:

ArrayList list = new ArrayList();
list.add( 3 );			// Compile-time error
list.add( new Integer( 3 ) );	// OK

Eventually Java caught up with C# (which introduced autoboxing in 2002), and with a modern compiler the above code would be valid.

With the introductions out of the way, here's a pop-quiz: what does the following code print?

HashMap<Integer, Integer> map = new HashMap<Integer, Integer>();
map.put( 4, 2 );
short n = 4;
System.out.println( Integer.toString( map.get( n ) ) );

As a long-time C# programmer I was completely befuddled when the code resulted in a NullPointerException. Huh? Exception? What? Why?

It took me a while to figure it out: because Java generics are compile-time constructs and are not directly supported by the VM, what actually happens is that the underlying container class accepts regular Object instances (reference types); the compile-time check merely asserts that n can be promoted from short to int, whereas the actual object passed to the container class (via autoboxing) is a Short! Since the container doesn't doesn't actually have a runtime generic type per se, the collection merely looks up the reference object in the map, fails to find it (I guess the Object.hashCode implementation for value types simply returns the reference value as the hash code as in C#) and returns null. Doh! *slaps forehead*

A few ReSharper tips for your viewing pleasure (according to rumors it can extend your penis too!):

Alt+Shift+L locates the current file in the Solution Explorer (at least with the IDEA-style keyboard bindings we all know and love). This is particularly useful for extremely file-heavy solutions where you often want to Get Latest (or Undo Check Out) the current file.

Note: you can get this behavior automatically through Tools->Options->Projects and Solutions->Track Active Item in Solution Explorer, but in my opinion it's annoying as hell.

Easier unit testing: you can bind shortcuts to run or debug unit tests by context. Context means that the behavior is exactly as expected: using the shortcut with the cursor on a test will run just that test, using it on a class will run it as a whole fixture, and using it on a directory in the Solution Explorer will run all tests that reside in that directory. This is easily accomplished by going to the Tools->Options->Keyboard menu and binding:

• ReSharper.ReSharper_UnitTest_ContextDebug to whichever shortcut you wish for debugging purposes (my favourite is Ctrl+T, Ctrl+D)
• ReSharper.ReSharper_UnitTest_ContextRun to whichever shortcut you'd like to use in order to just run tests (my favourite is Ctrl+T, Ctrl+T)

Note: You can get the same behavior with the ubiquitous TestDriven.net, but it makes no sense to me to pay twice for the same functionality.

Quick link: download

In my work at Semingo I often encounter situations where it's impossible to unit- or integration-test a component without accessing the web. This happens in one of two cases: either the component itself is web-centric and makes no sense in any other context, or I simply require an actual web server to test the components against.

Since I firmly believe that tests should be self-contained and rely on external resources as little as possible, a belief which also extends to integration tests, I wrote a quick-and-dirty pluggable web server based on the .NET HttpListener class. The unit-tests for the class itself serve best to demonstrate how it's used; for instance, the built-in HttpNotFoundHandler returns 404 on all requests:

    [Test]
    [ExpectedException( typeof( WebException ) )]
    [Description( "Instantiates an HTTP server that returns 404 on all " +
                  "requests, and validates that behavior." )]
    public void VerifyThatHttpNotFoundHandlerBehavesAsExpected()
    {
        using ( LightweightWebServer webserver = 
            new LightweightWebServer( LightweightWebServer.HttpNotFoundHandler ) )
        {
            WebRequest.Create( webserver.Uri ).GetResponse().Close();
        }
    }

The web server randomizes a listener port (in the range of 40000-41000, although that is easily configurable) and exposes its own URI via the LightweightWebServer.Uri property. By implementing IDisposable the scope in which the server operates is easily defined. Exceptions thrown from within the handler are forwarded to the caller when the server is disposed:

    [Test]
    [ExpectedException( typeof( AssertionException ) )]
    public void VerifyThatExceptionsAreForwardedToTestMethod()
    {
        using ( LightweightWebServer webserver = new LightweightWebServer(
            delegate { Assert.Fail( "Works!" ); } ) )
        {
            WebRequest.Create( webserver.Uri ).GetResponse().Close();
        }
    }

The handlers themselves receive an HttpListenerContext, from which both request and response objects are accessible. This makes anything from asserting on query parameters to serving content trivial:

    [Test]
    public void VerifyThatContentHandlerReturnsValidContent()
    {
        string content = "The quick brown fox jumps over the lazy dog";

        using ( LightweightWebServer webserver = new LightweightWebServer(
            delegate( HttpListenerContext context )
                {
                    using ( StreamWriter sw = new StreamWriter( context.Response.OutputStream ) )
                        sw.Write( content );
                } ) )
        {
            string returned;
            using ( WebResponse resp = WebRequest.Create( webserver.Uri ).GetResponse() )
                returned = new StreamReader( resp.GetResponseStream() ).ReadToEnd();

            Assert.AreEqual( content, returned );
        }
    }

We use this class internally to mock anything from web services to proxy servers. You can grab the class sources here -- it's distributed under a Creative Commons Public Domain license, so you can basically do anything you want with it. If it's useful to anyone, I'd love to hear comments and suggestions!

So you want a web project, a build system and a reasonable IDE to take care of the annoying details for you, right? The good news are that it's actually quite possible and there're many ways to do this. The bad news are that it's nigh impossible to get them to play along if you don't already know how to do that. It took me days to find a solution that finally seems to work, and I'd like to share it with you. I'm probably missing a few important details or did something really really stupid along the way (I'd appreciate comments!), but this process does seem to work. I'm not going into essentials of Java-based web development here -- if you want a more basic explanation of the terminology, post a comment and I'll see what I can do...

1. Goal

I want to:

• Use a common, standard and powerful IDE to edit and debug my Java code, and preferably provide a usable GUI interface for dependency and project management;
• Use a common, standard servlet container to host my servlet and still be able to control and debug everything from the same IDE;
• Have a convenient way to handle internal-and-external dependencies without worrying too much about the details;
• Be able to quickly compile, test and package my servlet for deployment;
• Understand as little as possible about the dependency stack of the tools involved

I'm going to tell you how to achieve most of these goals, with two glaring omissions: I won't show you how to do testing (I haven't successfully managed servlet unit testing so far -- different post on that) and I can't help but delve into some of the more annoying details involved with these tools and their dependencies. Sorry about that. Additionally, some of the information here applies even if you use different tools, but you're bound to face issues not covered here; don't assume I know more than you do -- seek the answers, post them somewhere, and maybe the next person will actually find what they're looking for!

2. Tools

The tools used are:

• J2SE JDK is an obvious must-have. Version used: JDK 6 update 3;
• Eclipse (but please don't download it just yet) for code editing, debugging and project management;
• Web Tools Platform: this is the Eclipse plug-in that adds web development capabilities to the IDE, including J2EE dependency management, hosting and running servlets from within the Eclipse workspace etc. This would be a good time to run along to the WTP web-site and download the Web Tools Platform All-In-One package. I only used the release version (2.0.1 at the time of writing this), so if you use another version your mileage may vary;
• Apache Maven is the newfangled build system from Apache slated to replace ant. I've used it for the last few days and so far it appears to be quite robust and even fairly well-integrated into Eclipse (see next item). Version used: 2.0.8;
• M2eclipse is the Eclipse plug-in for Maven integration. I've only found one problem with it so far, which I'll detail later on;
• Apache Tomcat is a solid choice in servlet containers. It's robust, fast and open-source, and has terrific Eclipse integration. I haven't given any of the other containers a serious whirl yet though.

3. Preparations

Unlike Visual Studio, with the tools mentioned above there's no straightforward installation procedure. You'll have to designate at least a workspace directory (where your Eclipse projects, settings etc. go) and some location where the tools themselves go. For me, it's C:\Dev\Eclipse and C:\Tools respectively.

• Setting up Java:
  • Install the JDK and remember where it was installed (nominally in %PROGRAMFILES%\Java\jdk1.6.0_03)
  • Set up a system-wide JAVA_HOME environment variable pointing to the same directory
• Setting up Maven and Tomcat:
  • Extract both archives to your designated directory (e.g. for Maven it would be C:\Tools\apache-maven-2.0.8)
  • Add the Maven bin directory to your PATH environment variable (user- or system-wide, depending on your preference)
  • Add whichever J2EE libraries you desire from the Tomcat installation to your class-path. If you have no idea what I'm talking about, you'll probably just want to set the CLASSPATH environment variable to your equivalent of c:\tools\apache-tomcat-6.0.14\lib\servlet-api.jar;c:\tools\apache-tomcat-6.0.14\lib\jsp-api.jar
• Setting up Eclipse:
  • Extract the WTP all-in-one package (which contains Eclipse itself) to your designated directory (e.g. C:\Tools\Eclipse)
  • Load Eclipse and point it to your designated workspace location
  • Install M2Eclipse:
    • Go to Help->Software Updates->Find and Install..., select "Search for new features to install" and click Next
    • Click on New Remote Site..., use M2eclipse or whatever for the name and http://m2eclipse.codehaus.org/update/ for the URL
    • Click on Finish and let Eclipse install the M2Eclipse plug-in
  • Set up a web server runtime for Eclipse to host your servlets in:
    • Open Window->Preferences...
    • Under Server select Installed Runtimes and click on Add...
    • Choose (from Apache) the Apache Tomcat v6.0 runtime and click Next
    • Enter the Apache installation directory (e.g. C:\Tools\apache-tomcat-6.0.14) in the appropriate location and click Finish

4. Creating a new web project

First off, you must create the actual project, directory structure etc. To do this:

• Open a command prompt, go to your Eclipse workspace directory
• Decide on your Maven group and artifact IDs; it's worth noting that the artifact ID is also the directory name for the project
• Type in mvn archetype:create -DgroupId=your.group.id -DartifactId=your.artifact.id -DarchetypeArtifactId=maven-archetype-webapp
• You'll notice that a new directory was created under the workspace root
• Edit the Maven project descriptor POM.XML in the newly created directory:
  • Add (after the <url> tag, although I'm not sure the order matters) the following section:

    <profiles>
        <profile>
            <id>servlet</id>
            <activation>
                <activeByDefault>false</activeByDefault>
            </activation>
            <dependencies>
                <dependency>
                    <groupId>javax.servlet</groupId>
                    <artifactId>servlet-api</artifactId>
                    <version>2.5</version>
                    <scope>provided</scope>
                </dependency>
                <dependency>
                    <groupId>javax.servlet.jsp</groupId>
                    <artifactId>jsp-api</artifactId>
                    <version>2.1</version>
                    <scope>provided</scope>
                </dependency>
            </dependencies>
        </profile>
    </profiles>

  • Under <build>, add the following section:

    <plugins>
        <plugin>
            <groupId>org.mortbay.jetty</groupId>
            <artifactId>maven-jetty-plugin</artifactId>
        </plugin>
        <plugin>
            <artifactId>maven-compiler-plugin</artifactId>
            <configuration>
                <source>1.5</source>
                <target>1.5</target>
            </configuration>
        </plugin>
    </plugins>

• You can now thank my colleague Aviran Mordo for finding out this bit of Voodoo.
• In the command prompt, now enter the project directory
• Type in mvn eclipse:m2eclipse -Dwtpversion=1.5 to create an Eclipse project
• Run Eclipse if it's not already started, then from the package explorer right click anywhere and click on Import...
• Choose General->Existing Projects into Workspace. For root directory pick the workspace directory
• Choose the new project and click on Finish
• At this point you may encounter a "Java compiler level does not match the version of the installed Java project facet" error. If that's the case, just right-click on the error (in the Problems view) and select Quick Fix, which will allow you to change the Java project facet version to 6.0. If this isn't what you want, you probably know enough to resolve the issue on your own...
• You'll need a src/main/java directory as a root source folder (as per the Maven convention). Right-click on the project, select New->Source Folder and type in src/main/java.
• Finally, in order to execute or debug the project on an actual running server, right-click on the new project and select Properties. From there go to the Server tab and select the runtime you created in the previous chapter.

At this point you have a Maven web project with a corresponding Eclipse project ready for editing in your workspace. In practice you will have to do several things to have any meaningful results.

1. Add your own code into the mix, such as a servlet. When adding a new servlet (via right-clicking the project, New->Other and choosing Web->Servlet) your WEB.XML file is automatically updated with the new servlet.
2. Add your own dependencies. Maven handles dependencies quite well; for instance, in order to actually create a servlet you're going to need servlet-api.jar in your classpath; the easiest way to do this is to right-click the project, select Maven->Add Dependency and then simply type in servlet and choose javax.servlet servlet-api.
3. When you wish to run or debug your servlet, right-click on its Java file and select Run As->Run on Server (or Debug, as appropriate). Your applet should be happily up and running.

5. Converting an existing project to Maven

I'm not sure how to go about doing this for web projects, but converting regular projects to use Maven is actually pretty straightforward; move your sources to the appropriate directories according to the Maven conventions, right-click the project in Eclipse and choose Maven->Enable Dependency Management; this will implicitly create a Maven project descriptor for you (POM.XML) and that's pretty much it. From that point on your Eclipse and Maven projects should peacefully coexist, allowing you to leverage both tools for your purposes.

In case it wasn't obvious, I've been doing some Java development lately. One of the curious things about doing development in the Java world is that, whereas in the Microsoft world you get a fairly complete tool-chain direct from a commercial vendor, in the Java world you're pretty much dependant on the open-source ecosystem built around the essential Java technologies: Sun defines the APIs, the community provides the tools. In many ways this is really really cool: many Java tools like JUnit are so absolutely groundbreaking that they found their way into the common development idiom irrespective of language, and the availability of tools for just about any purpose is a tremendous advantage (being able to choose freely between Resin, Jetty, Tomcat or any other commercial container, for instance, is a huge boon).

This diversity and community-centric development ecosystem definitely comes with a price though. Java tools, even the high-profile ones such as Eclipse, are extremely difficult to work with for the uninitiated, with a learning curve somewhat like that of Linux: if you take the time to learn the tools you can do astounding things and remain in complete control of the system, but the sheer context required to do even the most trivial thing can be - and often is - daunting.

I've been battling these tools on and off for the last few weeks and often end up having to figure something out on my own. Unlike the .NET ecosystem, it's usually quite difficult to find a blog post detailing a solution to a particular problem. To that end I intend to document my successes - victories, if you will - over the tool-chain, and also the problems I encounter and haven't been able to solve, in the hope of helping others and maybe myself in the process. These posts will go under the Development->Java category, and I'd really appreciate any comments on the solutions (so that I can improve my own work) as well as the problems (so I can actually solve them). Here's to hoping

I've been developing software with .NET professionally for the last five years or so, and aside from the occasional foray into other languages I've more or less specialized in that environment. While merrily hacking away at our back-end here at Semingo, we've recently made the decision to develop an aspect of said back-end in Java. As it's always a good practice to keep an open mind and experiment with other technologies I've happily accepted the challenge.

After working with Java and its associated tools for the past three or so weeks I have several observations to make:

1. The prominent free Java IDE, Eclipse, is actually a very full featured and impressive platform but takes a lot of getting used to. Some of the idioms and concepts are radically different than Visual Studio; for instance, whereas in Visual Studio you'd create an "ASP.NET Application Project", in Eclipse you create (or convert to) a dynamic web project and then add something called facet to your project; for instance, a Dynamic Web Module facet allows you to easily create and debug servlets, and the "Axis2 Web Services Core" facet allows you to create Axis2-based web services and work on them from within your IDE. To actually make use of these features, however, one needs to develop a pretty hefty knowledge base on the various technologies involved (J2EE and servlets, servlet containers like Tomcat, WTP etc.)
   
2. Eclipse is next-to-useless without some tinkering; in particular, what I originally attributed to very immature web development plug-ins - the WTP umbrella project I already mentioned - turned out to be the default memory settings of the Eclipse launcher. The launcher hosts the Java VM and its baseline configuration is simply inadequate. In my case adding the following switches: -vmargs -Xmx512M -XX:MaxPermSize=128M to the command line resolved all of the problems I had with the various WTP plug-ins, as well as the myriad crashes I've experienced with the IDE. In fact it's rock-stable now.
   
3. The Java language has some unexpected caveats; for instance, whereas in C# the designers eschewed fall through in the switch statement (you can group labels to implementations, but you cannot fall through from the implementation of one case statement to the next), the Java designers elected to maintain C-style behavior. I'm of the belief that switch statement fall through is the cause of a huge number of subtle, hard-to-find bugs, and was surprised to learn of this discrepancy between the two languages.
   
4. Enumerations in Java, a relatively new feature added in 1.5, are an impressively diverse feature which is a great deal more powerful than its C# counterpart. I only wish the designers would also allow for a more simplified "SOME_CONSTANT = 3" type syntax, as it's somewhat cumbersome to have to actually use constructors for the purpose. Additionally Java does not (to my knowledge) support implicit conversion operators, which makes necessary constructs such as SomeEnum.CONSTANT_VALUE.getConvertedValue(). It's not a huge issue but it's one of the many areas where syntactic sugar in C# is useful.
   
5. Speaking of syntactic sugar, there're several aspects where Java simply falls short of C#: disposables, iterators and delegates. Yes, I know delegates are an essentially religious issue for the Java designers (mostly for historical reasons, I suspect), and I won't deny that anything you can do with delegates you can do with nested classes, but at ridiculous verbosity. As for disposables, I find that the using keyword in C# is one of the most useful language constructs I've ever encountered, the use of which goes way beyond the original intention of elegantly scoping unmanaged resource use; finally, iterators are tremendously useful and cut a lot of unnecessary boilerplate code out of the equation.
   
6. The Java ecosystem is riddled with code- and buzz-words, to the point of being annoying. If you thought .NET has too many sub-technologies and acronyms, you should try Java. Just to get the taste buds going, here are some of the keywords I've been messing with for the past couple of weeks: J2SE, J2EE, Servlet, Eclipse, WTP, (Apache) Tomcat, Axis2, SAX, JAXP, JAX-RPC, JAR, WAR, AAR, EAR, JDBC, JavaDoc, JSP and JavaBeans. And that's just off the top of my head! To someone with any sort of Java experience this list wouldn't seem intimidating or even exhaustive, but to a new-comer that's simply too much. There is also an import cultural distinction: Visual Studio and its associated technologies (.NET, ASP.NET, ADO.NET etc.) are designed to ease you in as you learn the ropes; I found it much easier to simply start working with them and learn as I go, whereas with the Java counterparts I usually found myself trying to rework code samples found on the 'net while scratching my head.
   Now don't misunderstand me: the Java technologies are generally impressive, mature and usable, but the learning curve is not nearly as comfortable as the competing technologies from Microsoft, and the tools and documentation just aren't as polished.

The Windows SDK command shell, setenv.cmd, is immensely useful, so much so that I wanted it as my default command prompt (i.e. when CMD is run, no matter by whom). A quick Google search didn't turn out anything, so I eventually figured it out myself. The trick is to add it to the command processor's AutoRun value in the registry (run cmd /? from the command prompt if you don't know what I'm talking about):

reg add "HKCU\Software\Microsoft\Command Processor" /v AutoRun /f /t REG_EXPAND_SZ /d "\"%programfiles%\Microsoft SDKs\Windows\v6.0\Bin\SetEnv.Cmd\" /debug /x86 /vista"

You'll notice that I explicitly set the arguments for setenv.cmd; I can't explain it (nor bothered to delve into the script), but without these arguments the script gets stuck along with the command prompt. You should obviously change the values to your own environment.

Our test code makes extensive use of a lightweight web server implementation based on the .NET 2.0 HttpListener class (I'll write a separate post on this, source code included, in the near future). The web server implementation randomizes an incoming port and exposes its URI via a property; this is done for two reasons: to avoid conflicts with other listeners on the machine, and to facilitate multi-threading test runners (like TestDriven.NET). Even with a serial test-runner (like ReSharper's unit test driver, which is what most of us at Semingo use) this means that for an average test suite a large number of web-server instances are initiated and disposed of in very rapid succession. Our implementation creates each web server on its own dedicated port (using Socket.Bind to make sure that the port is free), and closes the listener via HttpListener.Abort when the test ends.

Although the entire test suite seems to run fine locally (both via ReSharper's test runner and using NUnit-Console), at some point we started getting strange errors from our continuous integration server:

System.Net.HttpListenerException : Failed to listen on prefix 'http://+:40275/' because it conflicts with an existing registration on the machine.

I verified this by running the tests with NUnit-Console on the CI server and it was consistent (but only on that server). Since the only instances of HttpListener used throughout the test suite are those used by the test web servers, this means that HttpListener.Abort does not properly unregister its own prefixes. Since the documentation for HttpListener is rather sparse and I couldn't find any mention of this issue on the web, I eventually went the Reflector route. Check out the Reflector decompiler output for both HttpListener.Dispose (called via Close) and HttpListener.Abort methods:

HttpListener.Dispose	HttpListener.Abort
if (this.m_State != State.Closed) { this.Stop(); this.m_RequestHandleBound = false; this.m_State = State.Closed; }	if (this.m_RequestQueueHandle != null) { this.m_RequestQueueHandle.Abort(); } this.m_RequestHandleBound = false; this.m_State = State.Closed;

The primary difference is the call to HttpListener.Stop. Here's a code snippet from that method:

if (this.m_State != State.Stopped)
{
    this.RemoveAll(false);
    this.m_RequestQueueHandle.Close();
    this.m_RequestHandleBound = false;
    this.m_State = State.Stopped;
    this.ClearDigestCache();
}

I'll spare you the hunt and point the problem out. There are two tangible differences between the two calls:

1. HttpListener.Close closes the request queue handle (which is used in calls to the native HTTP API) whereas HttpListener.Abort aborts it. I didn't delve into this but the semantics seem to be the same as for the HttpListener itself.
2. HttpListener.Close calls RemoveAll before disposing of the queue handle, presumably in order to stop accepting incoming requests.

In order to solve the problem, you can either remove the prefixes manually, or call the internal method like so:

listener.GetType().InvokeMember(
    "RemoveAll",
    BindingFlags.NonPublic | BindingFlags.InvokeMethod | BindingFlags.Instance,
    null, listener, new object[] { false } );

It's actually pretty easy; get your web service up and running in axis, run svcutil.exe http://someserver/someurl/someservice?wsdl and you're good to go. Unless, of course, you're trying to return arrays from the Java side (primitive arrays at least, I didn't see much of a point testing with custom classes). Consider the following contract:

public interface ServiceInterface {
    int[] doSomething( String someParameter ); 
}

The generated WSDL looks something like this (edited for clarity... I hope):

<?xml version="1.0" encoding="UTF-8"?>
<wsdl:definitions 
    xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/"
    xmlns:xsd="http://www.w3.org/2001/XMLSchema"
    xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/">

 <wsdl:types>
  <schema xmlns="http://www.w3.org/2001/XMLSchema">
   <import namespace="http://schemas.xmlsoap.org/soap/encoding/"/>
   <complexType name="ArrayOf_xsd_int">
    <complexContent>
     <restriction base="soapenc:Array">
      <attribute ref="soapenc:arrayType" wsdl:arrayType="xsd:int[]"/>
     </restriction>
    </complexContent>
   </complexType>
 </wsdl:types>

 <wsdl:message name="doSomethingResponse">
   <wsdl:part name="doSomethingReturn" type="impl:ArrayOf_xsd_int"/>
 </wsdl:message>
 <wsdl:message name="doSomethingRequest">
   <wsdl:part name="someParameter" type="soapenc:string"/>
 </wsdl:message>

 <wsdl:portType name="ServiceInterfaceService">
      <wsdl:operation name="doSomething" parameterOrder="someParameter">
         <wsdl:input message="impl:doSomethingRequest" name="doSomethingRequest"/>
         <wsdl:output message="impl:doSomethingResponse" name="doSomethingResponse"/>
      </wsdl:operation>
 </wsdl:portType>

 <!-- More uninteresting stuff -->



</wsdl:definitions>

The WDSL is perfectly fine and is properly processed by Microsoft's svcutil.exe tool; the generated classes look and appear to function normally, but if you'll look at the deserialized array on the client (.NET/WCF) side you'll find that the array has been deserialized incorrectly, and all values in the array are 0. You'll have to manually look at the SOAP response returned by Axis to figure out what's wrong; here's a sample response (again, edited for clarity):

<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenv=http://schemas.xmlsoap.org/soap/envelope/>
  <soapenv:Body>
    <doSomethingResponse>
      <doSomethingReturn>
        <doSomethingReturn href="#id0"/>
        <doSomethingReturn href="#id1"/>
        <doSomethingReturn href="#id2"/>
        <doSomethingReturn href="#id3"/>
        <doSomethingReturn href="#id4"/>
      </doSomethingReturn>
    </doSomethingResponse>
    <multiRef id="id4">5</multiRef>
    <multiRef id="id3">4</multiRef>
    <multiRef id="id2">3</multiRef>
    <multiRef id="id1">2</multiRef>
    <multiRef id="id0">1</multiRef>
  </soapenv:Body>
</soapenv:Envelope>

You'll notice that Axis does not generate values directly in the returned element, but instead references external elements for values. This might make sense when there are many references to relatively few discrete values, but whatever the case this is not properly handled by the WCF basicHttpBinding provider (and reportedly by gSOAP and classic .NET web references as well).

It took me a while to find a solution (after which I stumbled onto this post, which wasn't trivial to find): edit your Axis deployment's server-config.wsdd file and find the following parameter:

<parameter name="sendMultiRefs" value="true"/>

Change it to false, then redeploy via the command line, which looks (under Windows) something like this:

java -cp %AXISCLASSPATH% org.apache.axis.client.AdminClient server-config.wsdl

The web service's response should now be deserializable by your .NET client.

If you have a WCF service that returns a stream to the caller, you should be extra-careful with the source of these streams. In my case I was redirecting a file stream over a MessageContract that looks something like this:

    [MessageContract]
    public class ItemInfo
    {
        ...

        [MessageBodyMember]
        public Stream ItemStream;
    }

It took one of the integration tests to fail... oddly to figure out that the stream wasn't getting disposed of after the operation was completed. The solution appears to be fairly simple, add the following lines to your service wrapper (if your concrete implementation is WCF-aware you can always just stick this straight in the implementation code):

    // Make sure stream gets disposed at the end of the operation
    OperationContext.Current.OperationCompleted += delegate { item.ItemStream.Dispose(); };

Installing Microsoft Team Foundation Server is a ridiculously arduous and difficult process. I'll spare you my own complaints and simply list the checklist for installing this beast. This assumes you're installing TFS in a domain-enabled environment and in single-server mode; this is the typical configuration for a small-to-medium-size organization:

1. Designate a machine to host your Team Foundation Server repository. This machine must not double as a domain controller as this configuration is not supported by TFS.
2. Set up at two regular user accounts (not administrators, and if you have any group policies you may - according to your configuration - want to keep these users out of the relevant OUs) in your Active Directory. I used the trivial TFSService and TFSReports accounts. Also you'll need a user with administrative privileges on the target server; I personally prefer to avoid the associated headaches, so I simply used a domain administrator user for installation purposes (but used the aforementioned two users to set the beast up).
3. If necessary, install Windows 2003 Server (whatever flavor) on the machine; don't forget the necessary service packs and updates. If your pipe is fat enough, just let Windows Update do its magic.
4. Add an Application Server role, make sure you enable ASP.NET 2.0 during the installation process
5. Install SQL Server 2005. Make sure you read the installation guide first though, as you'll need to set it up to "Use the built-in System account," enable all services except Notification and finally select Windows authentication as the preferred authentication mechanism. You'll also need to let the SQL Server installer install a bunch of prerequisites before actual installation begins.
6. Install SQL Server 2005 Service Pack 2.
7. From the TFS installation media, install hot-fix 913393 for .NET Framework.
8. Install Windows SharePoint Services 2.0 with Service Pack 2.0. Make sure you select server farm mode when installing, or you'll just have to redo the installation.
9. Install Team Foundation Server itself.
10. Back up the reporting services encryption key (you can find a description of the procedure here).
11. Install hot-fix 919156, a.k.a the Quiescence GDR (no, I have no idea what GDR stands for).
12. Install Team Foundation Server Service Pack 1.
13. Make sure TCP port 8090 is open in your firewall software if you want web access to your Team Foundation Server (to be honest, I haven't found any use for it yet.)
14. Install Team Explorer from the installation media (required for many add-ons, including eScrum).
15. Install Visual Studio 2005 Team Suite Service Pack 1. This can, and will, take forever.

If at this point you're not thoroughly exhausted, you might want to set yourself up a with a project. We're currently evaluating the Microsoft eScrum template for our purposes; my colleague Oren Ellenbogen, in his capacity as Scrum Master, will probably be posting his thoughts on eScrum as a platform. In the meantime here's a quick list of solutions to problems we've encountered while configuring the beast:

1. Make sure you install the various prerequisites; in this case, .NET Framework 2.0, IIS, TFS and Team Explorer, AJAX Extensions 1.0 and the Anti-Cross Site Scripting Library
2. At this point you're liable to get a strange SharePoint-related error if you try and create an eScrum-based project; if that's the case (or as a preemptive measure), just run iisreset on the TFS server.
3. If you can't seem to access the eScrum website (nominally at http://yourserver/eScrum) you may have to reconfigure the eScrumAppPool identity from the IIS manager (right click the application pool, chose Properties, go to the Identity tab and enter the right information under Configurable)
4. You may also get 404 errors from the eScrum website even though it's very obviously configured. We've found that the solution described here works as well:
   • From the command prompt, type cd "%ProgramFiles%\Common Files\Microsoft Shared\Web Server Extensions\60\BIN"
   • Run STSADM.EXE -o addpath -url http://localhost/eScrum -type exclusion
   • Run iisreset again
5. eScrum reports only update once every 1 hour. If this bothers you, follow the instructions here to reduce the lag.

Hope this saves someone out there a lot of time and headache (and if so, a comment or e-mail is always appreciated...)

Imagine that you work mostly with C#. Imagine that on occasion you sprinkle SQL, Java and JavaScript into your daily routine, and that you also use the command line a lot.

Can you see the problem?

Didn't think so. I'll clue you in: the last one in that list has one distinct difference with the first four. Look around you. Have you found what we're looking for?

That is correct. The answer is: the semicolon. A lightweight, generally meaningless way to end a statement in most common languages. Arguably useless to users, arguably beneficial to parsers, and the cause of almost half an hour of continuous head-scratching by both programmer and DBA. How so?

C:\Temp>mysql -C -usomeuser -phispassword thedatabase;
ERROR 1044 (42000): Access denied for user 'someuser'@'%' to database 'thedatabase;'

I'll let you draw your own conclusions.

I've been working with Rhino Mocks for a couple of days now and I'm duly impressed. Aside from being generally useful, the Rhino Mocks syntax is mostly natural and some very clever design hacks are used to make it work the way you'd expect it to. (In case you don't know what mocking frameworks are, the subject has been tackled ad nauseam. Just take your pick.)

There is one caveat that is not immediately visible in the Record-Playback syntax, though:

internal interface IFoo
{
    int Bar();
}

[Test]
[ExpectedException( typeof( InvalidOperationException ),
    "Type 'System.Object' doesn't match the return type " +
    "'System.Int32' for method 'IFoo.Bar();'" )]
public void TestRecordSyntax_ReturnTypeMismatchBehavior()
{
    MockRepository mocks = new MockRepository();
    IFoo mock = mocks.CreateMock<IFoo>();
    using ( mocks.Record() )
        Expect.Call( mock.Bar() ).Return( new object() );
}

This simple test merely returns an incorrectly typed return value for the IFoo.Bar method, so with the ExpectedException in place you'd expect it to succeed, but you're in for a surprise:

[failure] RecordPlaybackTests.TestSetup.ConsistentReturnTypeMismatchBehavior
TestCase 'RecordPlaybackTests.TestSetup.ConsistentReturnTypeMismatchBehavior'
failed: Exception of type 'MbUnit.Core.Exceptions.ExceptionExpectedMessageMismatchException' was thrown. Expected exception message: "Type 'System.Object' doesn't match the return type 'System.Int32' for method 'IFoo.Bar();'", got "Previous method 'IFoo.Bar();' require a return value or an exception to throw.".

To understand this, consider the following flow:

1. A method call expectation is set (by calling mock.Bar())
2. Attempting to set a wrong return type results in the expected exception being thrown from Rhino Mocks. This happens before the Return constraint is set
3. Because using calls IDisposable.Dispose in the finally block, the mocks.Record object calls mocks.ReplayAll()
4. ReplayAll asserts because a required expectation (return value) is not yet set, masking the expected exception!

This is in fact how I first encountered this behavior, however there are several scenarios where this can happen. One example: setting up a return value that throws an exception on construction, possibly due to invalid arguments, causes the same behavior thereby masking the true cause of the error.

An alternative syntax suggested by Ayende (author of Rhino Mocks) is:

    With.Mocks( delegate
                {
                    Expect.Call( mock.Bar() ).Return( new object() );
                } ); 

Aside from the esthetical aspect (i.e. butt-ugly), this syntax introduces a minor issue: you cannot Edit-and-Continue methods with anonymous delegates. This may not be a big deal for test code but it's worth noting.

All in all I don't consider this a big issue, but it can bite and should therefore be understood and documented. This post will be reflected in the Rhino Mocks wiki shortly and will hopefully save someone a little grief in the future...

Quick download link (v1.1)

I wanted to simple way to download an entire PicasaWeb album. Right-click and save simply will not do, installing the Picasa application is completely out of the question (I use FastStone, in case you were wondering) and I couldn't find any easy way of doing this, so I whipped up a quick and dirty application. It's pretty self-explanatory, really:

Download the application (source included) here.

I also used the chance to give #Develop a serious whirl, and I can honestly say that it's damn impressive; in several hours of use the only real qualms I've had with it are the non-configurable keyboard bindings (at least I couldn't find any configuration menu for this; it wouldn't be an issue if the default bindings weren't slightly different from what I'm used to) and the lack of immediate/watch debug windows. The dialog editor is full-featured and the environment seems to be quite responsive and robust. I really am impressed.

Update (20-Aug-2007): Version 1.1 is now downloadable. I've added support for AuthKeys (as per skolima's request) so you should now be able to download private albums as well (assuming you have the appropriate key). Also added a compiled binary to the archive, doh!

My brother was going on and on about a coding challenge he found as part of the "Lisp as an Alternative to Java" study (the challenge instructions are outlined here, and you can find the sample dictionary and input data by following the study link); eventually I agreed to give it a try.

Spoiler alert: Do not read on if you want to take on this challenge!

Although I had to stop in the middle because I had company, the initial (hacked-together, very inefficient and completely uncommented C# 2.0 code) version took me 1 hour and 20 minutes. While technically correct this version performed abysmally, but it served as a good basis for further optimizations; a second, optimized version took another 30 minutes and with another 20 or so minutes to clean it up, the final version took a grand total of 2 hours and 20 minutes. Total length is 193 lines of code (including comments and white-space), which you can find here.

The crux of the solution is a very simple tree structure to hold the word dictionary. Each node has a 10-entry branch table, each entry representing a phone digit; a node can have an arbitrary number of leaves, each leaf representing a dictionary word. This allows for a very simple (and hopefully efficient) inner search loop:

• Use successive digits as indices for traversal into the tree;
• When a node with leaves is encountered, the leaves (dictionary words) are added to the potential result set. The rest of the input is then recursively encoded;
• If no leaves are encountered for the entire input set, a digit is added to the encoded string and the rest of the input is again encoded. A boolean flag is passed along with the recursion to make sure two consecutive digits are never encoded.

This is basically a prefix tree (or trie). With the provided dictionary and input sets, runtime on my 1.7GHz Pentium M-equipped laptop is approximately three seconds. I haven't properly profiled the application for CPU and memory utilization because, frankly, it's good enough. There's also a lot of leeway for optimizations: the current tree implementation is woefully inefficient, as tree traversal can result in additional memory pressure.

With my project finally nearing completion, it's nigh time for Microsoft to release yet another update to the .NET Compact Framework 2.0. Service Pack 2 ought to bring it to about "beta 2" level.

Check out these gems:

• NETCF deadlocks on exit if native callback delegate has been called on native thread (this is one of the few bugs in the CF I haven't encountered. Ironic, considering we make heavy use of native code in our project.)
• Access violation marshaling a class with a string field (there is a dent in the nearby wall on account of this one.)
• TypeLoadException using generics with NETCF 2.0 (TypeLoadExceptions in general are a lot of fun in the CF.)
• Installing multiple locales of same MSI results in multiple instances of NetCF showing up in Add Remove Programs (we've had some complaints regarding this from our client. They'll be mighty pleased to hear this, I'm sure.)

Now don't get me wrong - I think the CF is an impressive platform, or at the very least could've been. I would venture to say that the people on the CF implementation team are probably skilled professionals just doing the best job they can. But I can't forgive Microsoft - as a company - for shipping a half-baked, half-assed product that even at version 2.0 and after two service packs is still riddled with bugs! It boggles the mind that for any but the most hard-core developers, a third-party extension to the .NET CF is practically a necessity because the class library itself is simply inadequate.

As an aside, I seriously doubt we'll chance regression bugs this close to the launch date, so we'll probably stick with SP1 (we've worked around the issues we've encountered anyway.)

By far one of the most annoying aspects of the .NET Compact Framework is how heavily it relies on P/Invoke to fill in the gaps. The framework itself is missing huge pieces of functionality, and with the lack of C++/CLI for the Compact Framework a developer is often time left with no choice but to hack around the missing functionality with P/Invoke (or COM interop, if you have the patience to muck about with ATL).

The problem is that, on occasion, a P/Invoke call would result in a MissingMethodException (in fact, if you're really unlucky, the type loader will throw the same exception on loading the method actually making the P/Invoke call). Although a lot of the scenarios have been thoroughly ironed out by now and can be resolved through a Google search or some hacking on the developer's part, there is one scario that is esoteric enough that I couldn't find any references to it on the internet: you can get a MissingMethodException when you are out of memory.

We're working on an extremely large (in mobile proportions) and complex project, involving massive amounts of .NET logic combined with a very large, performance-concious and memory-hungry native codebase. We've had to hack around a lot of missing capabilities in the .NET Compact Framework (as well as some bugs and/or shortcomings in other parts of the OS), and one of our native calls would inconsistently throw a MisingMethodException; having reearched the problem for a day or two I was convinced that the problem was an incorrect function prototype for the exported function and added explicit calling convention declarations. This seemed to have resolved and I was content for a couple of days, until the problem resurfaced.

The exception content itself is next-to-useless, and since the same P/Invoke call would work intermittently I was hoping that enabling the loader log might supply some further information. Alas, all the log would provide was the following line:

Failed to find/load [SomeDll.dll] (even in [\Program Files\Somewhere\])

Not good. I then dug up another article on the subject and proceeded to enable the interop log, which provided some additional information:

JIT ERROR FOR PINVOKE METHOD (Managed -> Native):
[pinvokeimpl][preservesig]
int Workarounds::GetSinkWrapper(out IImageSink , IImageEncoder );
int (I4_VAL) GetSinkWrapper(IImageSink& ** (INTF_REF) , IImageEncoder *(INTF_VAL) );

I was originally interested in the bizarre native signature generation (specifically the IImageSink& ** parameter - where'd the reference come from?), but upon reading some valid log files for working methods I was convinced that it's a dead duck. I then set my attention to the JIT message: what has the JIT to do with a native function? I theorized that the JIT is responsible for the native signature generation for native functions and kept working under that assumption. That narrowed the question down to, "what went wrong with the JIT compiler?"

Eventually it occured to me that this might be yet another manifestation of the memory limitation issue (processes under Windows CE 5.0 are limited to a 32MB address space). The P/Invoke call is made fairly late into the application; I added a dummy function to the library which I called on application initialization. This forced the JIT to take care of the native library before anything was sucking up memory, and the issue was resolved.

The moral of the story? If you get MissingMethodExceptions on your P/Invoke calls, fisrt make sure your DLL is actually deployed; then check the DllImport signature (you can find a lot of useful resources on this FAQ). Finally, make sure you're not out of memory.

I was screwing around with XmlSerializer and manual recompilations (you can enable raw code output from the XmlSerializer) and somehow this happened:

.NET Framework reinstall time, I guess...

Footnote: This is fast becoming a series of posts on the woes of using the .NET Compact Framework. In fact, I've added a subcategory for this exact purpose; you can access the category and its RSS feed from the Categories list on the right-hand side of the screen.

Here's another one of those "bang your head repeatedly against the wall and look for a sharp object" issues: closing a stream derived from HttpWebResponse.GetResponseStream takes an abnormally long time. In fact, the (blocking) call to Stream.Close takes approximately the same amount of time as it would to read to the end of the stream. I've only encountered one other reference to this in a newsgroup post from 2004, which (according to Todd, the original author of the post) was never actually answered.

Seeing as I was trying to use HttpWebRequest to create a minimalistic download manager for one of our applications, I had to come up with a solution: one option was to close the stream asynchronously and just let the download run its course, wasting valuable memory, CPU time and bandwidth; another was to never close the stream and make do with a resource leak. Not content with these solutions I decided to dig into the framework code with the ever-useful Reflector. The first hurdle was locating the assemblies; they're not trivial to find, but if you look hard enough you can find actually useful versions in your Visual Studio 2005 installation folder, under SmartDevices\SDK\CompactFramework\2.0\v2.0\Debugger\BCL. These are regular .NET Assembly PEs so it's trivial to load them up in Reflector.

Some digging into the BCL sources proved that HttpWebRequest does, in fact, read until the end of the stream when closing; this is the relevant code excerpt, from HttpWebRequest.doClose:

if (!this.m_doneReading)
{
      byte[] buffer1 = new byte[0x100];
      while (this.ReadInternal(buffer1, 0, buffer1.Length, true) != 0)
      {
      }
}

It only started to make sense when I did some reading on HTTP (a protocol I'm not deeply familiar with). Apparently, HTTP 1.1 connections are persistent by default, which means the connection is maintained even after the request is completed, and further requests are served from the same connection. Technically, this means that the KeepAlive property of HttpWebRequest is true by default, and a Connection="Keep-alive" header is added to the HTTP request. I can only surmise that, with persistent connections, the response must be read in full in order to allow future requests (if the response was cut off, some concurrency issues may apply). Unfortunately, setting the KeepAlive property to false did not resolve the issue and the connection was maintained until closed.

Since I could find no way to resolve the problem, I decided to hack around it:

stream.GetType().BaseType
    .GetField( "m_doneReading", BindingFlags.Instance | BindingFlags.NonPublic )
    .SetValue( stream, true );

Screwing around with BCL internals is obviously NOT a recommended practise and it's something I tried very hard to avoid, however this problem had to be resolved in one way or another. I traced the code a bit deeper to see if this would actually have the expected results; apparently HttpWebRequest keeps an internal "reference counter" (actually a bitfield, but same principle) on the connection. According to the implementation for HttpWebRequest.connectionRemoveRef, if the reference counter reaches 0 the private m_connection field of the request becomes null, and the connection is released. I started out by testing this:

Before and after - you can see that the m_connection field becomes null

Unfortunately I have no idea how to test whether ot not the connection is actually disposed (conceptually, if KeepAlive is set to false then the connection should be immediately closed); when I find the time I'll try and track the HTTP session using Wireshark, but currently this horrible hack will just have to do.

I've been hacking away quite a bit at an application that contains large managed and unmanaged portions, and a necessarily complex interop layer in between. It appears that interop marshalling behaves somewhat differently between the full and Compact framework. Here's a somewhat laconic list of the issues I've encountered and how to resolve them:

1. You may encounter NotSupportedException on calls to Marshall.SizeOf. Although the documentation do