This is the first of several posts based on my experience using NCover4 on a large, new development project. Click here to see all the updates in order.
My team has been using NCover4 the last several months and I must say I have mixed emotions. I think it could be a good product — someday. The problem is right now it has all the rough edges of an early beta with the high price of an enterprise-ready development tool.
So how about what’s good? Well, let’s start with Code Central, their centralized web server. The idea of it is pure genius. All your test results can go there whether they are run at developer stations, the build server or manually by your QA testers. It tracks trends. It tracks line coverage, branch coverage and provides a number of other interesting metrics like CRAP scores. It lets you drill down into the details of any run. It’s even reasonably easy to configure.
Unfortunately, like most things in NCover 4, the implementation has some pretty significant flaws. The web user interface is pretty but ridiculously slow. We’re talking minutes to finish drawing a page-worth of coverage stats. Drill downs are not much faster. Installation with defaults is easy, but step outside of the defaults and your are in for a painful experience. Want to install to the D drive? Well, you’d better crack open the command line. Version upgrades, which they claim are automatic, have been hit and miss. Sometimes they work. Sometimes they don’t. You quickly learn to allocate a few hours to each version upgrade.
So how about the desktop? Well, it’s a web application too, which is kind of strange. It works about the same as the server. Low-overhead coverage, reasonably easy to configure but slow as heck to draw each web page when you have recorded more than a handful of results. It can manage and use configurations from the server, which is very nice feature. It can send test results to the Code Central server as well.
The Visual Studio plugin, included with the desktop, has promise. When it is working, it highlights lines in your code with coverage information. But it too seems unfinished. It behaves strangely at times. It causes error dialogs in Visual Studio periodically. It just doesn’t seem quite ready for prime time.
As far as build server integration goes, it is again a story with lots of promise and spotty results. Because of the way it works, you really don’t integrate it with the build server; Rather, you run the collector at the build server just like you run the collector anywhere else. We use TeamCity and once we switched to running NUnit instead of TC’s built in test runner, NCover4 was able to capture results. The latest couple of NCover4 versions periodically hang our build when tests start running. When this occurs, the NCover4 service cannot be stopped. Instead, we have to set it to manual start and then reboot the server. As long as we let one build run without coverage after the boot, we can then startup the NCover4 service and it works again for awhile.
That brings up the subject of support. It’s friendly and professional but hamstrung by the product. The bottom line is our typical down time when we have a problem is measured in days rather than hours. Early on, licensing for the desktops stopped working. We had to wait more than a week for a release to fix the problem. At one point, coverage stopped working all together. We spent hours over several days running utilities to capture information for support to examine only to arrive at the solution of doing a clean install, which, of course, lost all our history in the process. On average, NCover has been partially or completely down two days out of each of our ten day sprints.
In summary, NCover4 is currently early beta quality — full of bugs and lacking the polish of a finished product. Given time, it can be very good and worth every penny of its price. If you have the time to deal with the flaws, it might be worth a try in your project. However, be prepared to spend significant time and effort to keep it running.
Tom Cabanski 