Why My Team Gave Up on NCover4

This is the third of several posts based on my experience using NCover4 on a large, new development project.  Click here to see all the updates in order.

After struggling with problems caused by NCover4 for nearly a year, my team finally gave up on it on New Year’s Eve.  The writing was on the wall for a couple of weeks.  NCover4 Code Central had gotten so slow that it was completely unusable.  Chrome would pop up a dialog saying the server was unresponsive with buttons to kill or wait on the extremely long running request.  Trying to delete history was impossible because it would take 30 minutes plus from the time you pressed the button to delete one 25 record page of history to the time when the screen was ready to delete another.  Support recommended an index fix utility that ended up running in excess of 12 hours that did improve things a little.  Unfortunately, it still took up to a minute or two to draw one page of results and deletes still took 30 minutes per page.

After looking at the resources NCover4 was using, I moved its data onto a dedicated raid-0 array on our Amazon EC2 server, which roughly tripled the available I/O performance.  This dropped screen redraws to a still unacceptable 30 seconds to 1 minute.  It did not have any measurable impact on delete performance.

The final straw was intermittent test failures at the build server caused by NCover4 hogging resources for 15 – 20 minutes after the completion of the previous build.  If a build started too soon after the previous one finished, some tests involving an in-memory RavenDB would timeout waiting for stale indexes to update.  The second build would also take nearly three times as long as the first thanks to the load put on the server by NCover4.

After discussing the issue with the team, we pulled the plug on New Year’s Eve.  I spent an hour switching us over to JetBrains DotCover.  Although it does not offer the breadth of statistics that NCover4 does and has its flaws, it provides access to the basic code coverage metrics needed to identify and fix poorly covered code.  It is less expensive than NCover4 on the desktop, and, if you use TeamCity, it is free on the build server.  It puts less load on the server as evidenced by a 20% drop in build times.  It does not cause any of our tests to fail even when running builds back-to-back.  Because it is built into TeamCity, it is quite easy to integrate with the build process.  TeamCity also has a nice page that shows trends over time:

Image

The folks at NCover are working to improve performance.  They plan to add automatic history archiving to cut down on the amount of data that needs to be processed to draw their overview graphs.  They also plan to cache the coverage statistics they currently calculate for each page to cut down on the CPU load.  A release including these improvements is expected soon.  However, my team is pushing towards our release and we no longer have time to risk on something we are not sure is production ready.  Therefore, we’re going to stick with DotCover at least until Q4 of this year.  Even after that, NCover4 would have to be substantially better to justify the investment in time and money it would take to switch back.  I cannot recommend NCover4 for any team on any project at this time.

NCover4 Gets a Little Faster (and Hopefully Will Stop Hanging)

This is the second of several posts based on my experience using NCover4 on a large, new development project.  Click here to see all the updates in order.

Yesterday, I installed the latest upgrade to NCover4 (version 4.1.2078.723).  NCover support tells me that their developers found and fixed a thread deadlock that was causing the hanging build issue we observed on several occasions.  They also significantly improved the performance of the screen used to view the list of coverage results for a project.  Now it draws the main part of the screen in about 30 seconds.  Still a little slow, but considerably better than the many minutes it used to take.  I will post a more complete update after using this release for another week or two.

NCover4 — Could Be a Good Product Someday

This is the first of several posts based on my experience using NCover4 on a large, new development project.  Click here to see all the updates in order.

My team has been using NCover4 the last several months and I must say I have mixed emotions.  I think it could be a good product — someday.  The problem is right now it has all the rough edges of an early beta with the high price of an enterprise-ready development tool.

So how about what’s good?  Well, let’s start with Code Central, their centralized web server.  The idea of it is pure genius.  All your test results can go there whether they are run at developer stations, the build server or manually by your QA testers.  It tracks trends.  It tracks line coverage, branch coverage and provides a number of other interesting metrics like CRAP scores.  It lets you drill down into the details of any run.  It’s even reasonably easy to configure.

Unfortunately, like most things in NCover 4, the implementation has some pretty significant flaws.  The web user interface is pretty but ridiculously slow.  We’re talking minutes to finish drawing a page-worth of coverage stats.  Drill downs are not much faster.  Installation with defaults is easy, but step outside of the defaults and your are in for a painful experience.  Want to install to the D drive?  Well, you’d better crack open the command line.  Version upgrades, which they claim are automatic, have been hit and miss.  Sometimes they work.  Sometimes they don’t.  You quickly learn to allocate a few hours to each version upgrade.

So how about the desktop?  Well, it’s a web application too, which is kind of strange.  It works about the same as the server.  Low-overhead coverage, reasonably easy to configure but slow as heck to draw each web page when you have recorded more than a handful of results.  It can manage and use configurations from the server, which is very nice feature.  It can send test results to the Code Central server as well.

The Visual Studio plugin, included with the desktop, has promise.  When it is working, it highlights lines in your code with coverage information.  But it too seems unfinished. It behaves strangely at times.  It causes error dialogs in Visual Studio periodically. It just doesn’t seem quite ready for prime time.

As far as build server integration goes, it is again a story with lots of promise and spotty results.  Because of the way it works, you really don’t integrate it with the build server; Rather, you run the collector at the build server just like you run the collector anywhere else.  We use TeamCity and once we switched to running NUnit instead of TC’s built in test runner, NCover4 was able to capture results.  The latest couple of NCover4 versions periodically hang our build when tests start running.  When this occurs, the NCover4 service cannot be stopped.  Instead, we have to set it to manual start and then reboot the server.  As long as we let one build run without coverage after the boot, we can then startup the NCover4 service and it works again for awhile.

That brings up the subject of support.  It’s friendly and professional but hamstrung by the product.  The bottom line is our typical down time when we have a problem is measured in days rather than hours.  Early on, licensing for the desktops stopped working.  We had to wait more than a week for a release to fix the problem.  At one point, coverage stopped working all together.  We spent hours over several days running utilities to capture information for support to examine only to arrive at the solution of doing a clean install, which, of course, lost all our history in the process.  On average, NCover has been partially or completely down two days out of each of our ten day sprints.

In summary, NCover4 is currently early beta quality — full of bugs and lacking the polish of a finished product.  Given time, it can be very good and worth every penny of its price.  If you have the time to deal with the flaws, it might be worth a try in your project.  However, be prepared to spend significant time and effort to keep it running.