This is the third of several posts based on my experience using NCover4 on a large, new development project. Click here to see all the updates in order.
After struggling with problems caused by NCover4 for nearly a year, my team finally gave up on it on New Year’s Eve. The writing was on the wall for a couple of weeks. NCover4 Code Central had gotten so slow that it was completely unusable. Chrome would pop up a dialog saying the server was unresponsive with buttons to kill or wait on the extremely long running request. Trying to delete history was impossible because it would take 30 minutes plus from the time you pressed the button to delete one 25 record page of history to the time when the screen was ready to delete another. Support recommended an index fix utility that ended up running in excess of 12 hours that did improve things a little. Unfortunately, it still took up to a minute or two to draw one page of results and deletes still took 30 minutes per page.
After looking at the resources NCover4 was using, I moved its data onto a dedicated raid-0 array on our Amazon EC2 server, which roughly tripled the available I/O performance. This dropped screen redraws to a still unacceptable 30 seconds to 1 minute. It did not have any measurable impact on delete performance.
The final straw was intermittent test failures at the build server caused by NCover4 hogging resources for 15 – 20 minutes after the completion of the previous build. If a build started too soon after the previous one finished, some tests involving an in-memory RavenDB would timeout waiting for stale indexes to update. The second build would also take nearly three times as long as the first thanks to the load put on the server by NCover4.
After discussing the issue with the team, we pulled the plug on New Year’s Eve. I spent an hour switching us over to JetBrains DotCover. Although it does not offer the breadth of statistics that NCover4 does and has its flaws, it provides access to the basic code coverage metrics needed to identify and fix poorly covered code. It is less expensive than NCover4 on the desktop, and, if you use TeamCity, it is free on the build server. It puts less load on the server as evidenced by a 20% drop in build times. It does not cause any of our tests to fail even when running builds back-to-back. Because it is built into TeamCity, it is quite easy to integrate with the build process. TeamCity also has a nice page that shows trends over time:
The folks at NCover are working to improve performance. They plan to add automatic history archiving to cut down on the amount of data that needs to be processed to draw their overview graphs. They also plan to cache the coverage statistics they currently calculate for each page to cut down on the CPU load. A release including these improvements is expected soon. However, my team is pushing towards our release and we no longer have time to risk on something we are not sure is production ready. Therefore, we’re going to stick with DotCover at least until Q4 of this year. Even after that, NCover4 would have to be substantially better to justify the investment in time and money it would take to switch back. I cannot recommend NCover4 for any team on any project at this time.
Tom Cabanski 