Test driving and test coverage

Yesterday I enabled rcov on my current project and was surprised that it had 100 per cent unit test coverage.

But perhaps I should not have been surprised. After all, if you are truly test driving, how could you possibly leave anything uncovered?

Advertisements

Passing cucumber specs on windows

I had some trouble getting cucumber test features to pass on Windows due to directory separator differences, something like:

 Failing Scenarios:
-cucumber features/background/failing_background.feature:8 # Scenario: failing background
+cucumber features\background\failing_background.feature:8 # Scenario: failing background

There is an easy solution — run features with windows_mri profile, which results in the CUCUMBER_FORWARD_SLASH_PATHS environment variable being set to true.

Example

cucumber features/background.feature:93 -p windows_mri

Raking .NET projects in TeamCity

Faced with the unpleasant prospect of assembling yet another stack of xml files for an automated build, I thought I’d try rake instead. A couple of people here at 7digital have used Albacore before, so I started there.

1. Build

Use Albacore‘s msbuild task:

require 'albacore'

desc "Clean and build"
msbuild 'clean_and_build' do |msb|
    msb.properties :configuration => :Release
    msb.targets :Clean, :Build
    msb.verbosity = "quiet"
    msb.solution  = "path/to/ProjectName.sln"
end

2. Run tests

This is also very straight forward with Albacore, but slightly more useful is applying the usual TeamCity test result formatting and reporting.

2.1 Tell your build where the NUnit test launcher is

TeamCity already has an NUnit runner, and the recommended way to reference it is with an environment variable.

Note: The runners are in the <TEAM CITY INSTALLATION DIR>/buildAgent/plugins/dotnetPlugin/bin directory.

2.2 Write the task

Once you have the path to the executable, you’re free to apply any of the available runner options.

Assuming you have added the TEAMCITY_NUNIT_LAUNCHER environment variable then the actual execution is then something like:

asm = 'ProjectName.Unit.Tests.dll'
nunit_launcher = ENV["TEAMCITY_NUNIT_LAUNCHER"]
sh("#{nunit_launcher} v2.0 x86 NUnit-2.5.0 #{asm}")

Beats hundreds of lines of xml I reckon.

References

• Albacore — rake tasks for.NET systems
• TeamCity NUnit launcher
• Setting environment variables for your TeamCity build

I’ve created my gem, but can’t require it

Creating gems is easy, all you need is a gemspec:

Gem::Specification.new do |spec|
    spec.name = 'chubby-bat'
    spec.version = '0.0.1337'

    spec.files = [
	'./lib/chubby_rain.rb',
	'./lib/crack_murphy.rb',
    ]

    spec.summary = 'An example gem'
    spec.author = 'Ben Biddington'
end

And then to build it:

$ gem build chubby-bat.gemspec

And then to install it:

$ gem install chubby-bat-0.0.1337.gem

No such file to load — xxx

Trouble is, when requiring the new gem, you may encounter an error like:

irb(main):001:0> require 'chubby-bat'
LoadError: no such file to load -- chubby-bat
        from C:/Ruby/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in
            `gem_original_require'
        from C:/Ruby/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
        from (irb):1
irb(main):002:0>

I found this pretty confusing, until I realised the interpreter is looking for a file called chubby-bat.

In short, a gem needs to contain a file with the same name as the gem itself in its lib folder:

And gemspec becomes:

Gem::Specification.new do |spec|
    spec.name = 'chubby-bat'
    spec.version = '0.0.1337'

    spec.files = [
	'./lib/chubby-bat.rb',
	'./lib/chubby_rain.rb',
	'./lib/crack_murphy.rb',
    ]

    spec.summary = 'An example gem'
    spec.author = 'Ben Biddington'
end

Rubygems require

The comment in rubygems custom_require explains:

# See: path/to/Ruby/lib/ruby/site_ruby/1.8/rubygems, custom_require

##
  # When RubyGems is required, Kernel#require is replaced with our own which
  # is capable of loading gems on demand.
  #
  # When you call require 'x', this is what happens:
  # * If the file can be loaded from the existing Ruby loadpath, it
  #   is.
  # * Otherwise, installed gems are searched for a file that matches.
  #   If it's found in gem 'y', that gem is activated (added to the
  #   loadpath).
  #
  # The normal require functionality of returning false if
  # that file has already been loaded is preserved.

Ruby loadpath, gem install and Kernel#require

Installing a gem simply copies files to somewhere in $LOAD_PATH, and invoking require triggers a file search within these locations.

Kernel require does the following:

• Ruby tries to load the library named string, returning true if successful.
• If the filename does not resolve to an absolute path, it will be searched for in the directories listed in $:.
• If the file has the extension “.rb’’, it is loaded as a source file; if the extension is “.so’’, “.o’’, or “.dll’’,or whatever the default shared library extension is on the current platform, Ruby loads the shared library as a Ruby extension. Otherwise, Ruby tries adding “.rb’’, “.so’’, and so on to the name.
• The name of the loaded feature is added to the array in $”.
• A feature will not be loaded if it‘s name already appears in $”.

This means if you print out the contents of $”, you’ll see chubby-bat.rb in there.

Rake — shell task failure

For my current project, we’re running cucumber as one of our tasks. Our report generation runs afterwards — which we need to happen regardless of cucumber’s exit code (cucumber exits with code 1 when any tests fail).

Handling shell task failure

We noticed that if cucumber fails, rake exits immediately — without running our subsequent report tasks.

We’re running cucumber with rake’s ruby function — which runs the ruby interpreter in its own process by invoking a shell command.

Pseudo-stacktrace (irrelevant source lines omitted):

Shell::CommandProcessor.system(command, *opts)
Rake::Win32.rake_system(*cmd)
FileUtils.sh(*args, &block)
FileUtils.ruby(*args, &block)

The key to this is the behaviour of rake’s FileUtils.sh. Examining the source shows:

If it is not supplied a block, and the shell command fails, and rake exits with error status.

Aside: For some reason, Kernel.system echoes errors.

Example

This task will cause rake to exit immediately with status 1:

task 'this-one-fails' do
    ruby("xxx")
end

while the following task does not cause rake to terminate  normally, any subsequent task will still run:

path = File.expand_path(File.dirname(__FILE__) + ‘/output/cucumber-manual.txt’)

cuke_path = ‘C:/ruby/lib/ruby/gems/1.8/gems/cucumber-0.3.3/bin/cucumber’

ruby(“#{cuke_path} -h > ‘#{path}'”) do |success, exit_code|

puts “Exited with code: #{exit_code}” if !success

end

task 'this-one-succeeds' do
    ruby("xxx") do |success, exit_code|
        puts "Exited with code: #{exit_code.exitstatus}" unless success
    end
end

Getting the last exit status

The $? variable contains the status of the last child process to terminate. This may be useful if steps subsequent to the failed one require this information.

Cucumber — ANSI formatting

When cucumber output is piped to file on disk, it may contain non-printable characters and markup, for example:

[32m6 passed steps

Actually looks like this when output to console:

6 passed steps

This is due to cucumber emitting Ansi formatting.

[ANSI escape sequences] are used to control text formatting and other output options on text terminals.

Most of these escape sequences start with the characters ESC (ASCII decimal 27/hex 0x1B/octal 033) and [ (left bracket). This sequence is called CSI for Control Sequence Introducer(or Control Sequence Initiator). There is a single-character CSI (155/0x9B/0233) as well.

The ESC+[ two-character sequence is more often used than the single-character alternative, for details see C0 and C1 control codes. Devices supporting only ASCII (7-bits), or which implement 8-bit code pages which use the 0x80–0x9F control character range for other purposes will recognize only the two-character sequence. Though some encodings use multiple bytes per character, in this topic all characters are single-byte…

This formatting can be suppressed using the:

--[no-]color

switch.

Ruby require

While diagnosing problems with a ruby application, we discovered we had to move certain folders around so the interpreter could find them. Clearly we were missing something fundamental.

Require: definition

Inspecting the source (v1.9) for require shows:

Ruby tries to load the library named _string_, returning
 *  +true+ if successful. If the filename does not resolve to
 *  an absolute path, it will be searched for in the directories listed
 *  in <code>$:</code>. If the file has the extension ``.rb'', it is
 *  loaded as a source file; if the extension is ``.so'', ``.o'', or
 *  ``.dll'', or whatever the default shared library extension is on
 *  the current platform, Ruby loads the shared library as a Ruby
 *  extension. Otherwise, Ruby tries adding ``.rb'', ``.so'', and so on
 *  to the name. The name of the loaded feature is added to the array in
 *  <code>$"</code>. A feature will not be loaded if it's name already
 *  appears in <code>$"</code>. However, the file name is not converted
 *  to an absolute path, so that ``<code>require 'a';require
 *  './a'</code>'' will load <code>a.rb</code> twice.

Require does not canonicalize paths

Here’s the relevant part of the comment:

However, the file name is not converted
to an absolute path, so that
require 'a';
require './a'
will load a.rb twice.

So, as described, ruby treats require statements literally: a different path is a different file, even if it’s a different path to the same file. And this may result in a file being loaded twice.

This can have confusing side effects. We encountered this in one of our cucumber projects — after scenario handlers were firing twice and we didn’t know why.

Require does not expand paths

Another implication of this behaviour is that an application may not run as expected unless the correct working directory is used.

Adding to the $” list does not work

I thought another option may be adding items directly to the $” list. I guess this shows that require is doing more than just inserting an item in a list.

Adding to $” is in fact not enough, demonstrated by the fact that this works:

require file_to_include

$”.unshift(file_to_include)

require file_to_include
$".unshift(file_to_include)

While the reverse order does not:

$".unshift(file_to_include)
require file_to_include

This proves that require has exited early because the file already exists in $” – and hence the include fails.

Solution

Fortunately there is a way around both of these problems.

Rather than require relative paths:

require '../lib/application'

absolute paths may be specified:

File.expand_path(File.dirname(__FILE__) + '/../lib/application')

This means that:

1. Working directory is no longer relevant. The second example ensures that the required path is relative the the file that requires it.
2. Duplicate requires are not possible

EDIT, 2009-05-12: Also consider require_relative, added in v1.9