TeamCity — rake runner aborts early

How to run TeamCity rakerunner locally

It is important to get the working directory right, otherwise you’ll get a bunch of require errors, for example:

C:\ruby\bin/ruby.exe C:\BuildAgent\plugins\rake-runner\lib\rb\runner\rakerunner.rb

Fails with error:

C:/ruby/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require’: no such file to load — teamcity/rakerunner_consts (LoadError)

from C:/ruby/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require’

from C:/BuildAgent/plugins/rake-runner/lib/rb/runner/rake_ext.rb:20

no such file to load -- teamcity/rakerunner_consts (LoadError)

from C:/ruby/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'

from C:/BuildAgent/plugins/rake-runner/lib/rb/runner/rake_ext.rb:20

rake_ext is including a file with path:

require 'teamcity/rakerunner_consts'

And we know require works relative to the working directory which means our working directory must be set to:

C:\BuildAgent\plugins\rake-runner\lib\rb\patch

Rake abort

TeamCity rakerunner exits with message “Rake aborted!” as soon as it encounters a non-zero exit code from any task. Any subsequent tasks are therefore skipped.

This is not desirable behaviour for us because we have reporting tasks that must run always.

Stack trace showing the overridden members:

C:/BuildAgent/plugins/rake-runner/lib/rb/runner/rake_ext.rb:158:in 'process_exception'
C:/BuildAgent/plugins/rake-runner/lib/rb/runner/rake_ext.rb:95:in 'target_exception_handling'
C:/BuildAgent/plugins/rake-runner/lib/rb/runner/rake_ext.rb:266:in 'execute'
C:/ruby/lib/ruby/gems/1.8/gems/rake-0.8.4/lib/rake.rb:578:in 'invoke_with_call_chain'
C:/ruby/lib/ruby/1.8/monitor.rb:242:in 'synchronize'
C:/ruby/lib/ruby/gems/1.8/gems/rake-0.8.4/lib/rake.rb:571:in 'invoke_with_call_chain'
C:/ruby/lib/ruby/gems/1.8/gems/rake-0.8.4/lib/rake.rb:564:in 'standard_invoke_with_call_chain'
C:/BuildAgent/plugins/rake-runner/lib/rb/runner/rake_ext.rb:235:in 'invoke'
C:/BuildAgent/plugins/rake-runner/lib/rb/runner/rake_ext.rb:90:in 'target_exception_handling'
C:/BuildAgent/plugins/rake-runner/lib/rb/runner/rake_ext.rb:234:in 'invoke'
C:/ruby/lib/ruby/gems/1.8/gems/rake-0.8.4/lib/rake.rb:2027:in 'invoke_task'
C:/ruby/lib/ruby/gems/1.8/gems/rake-0.8.4/lib/rake.rb:2005:in 'top_level'
C:/ruby/lib/ruby/gems/1.8/gems/rake-0.8.4/lib/rake.rb:2005:in 'each'
C:/ruby/lib/ruby/gems/1.8/gems/rake-0.8.4/lib/rake.rb:2005:in 'top_level'
C:/BuildAgent/plugins/rake-runner/lib/rb/runner/rake_ext.rb:311:in 'standard_exception_handling'
C:/ruby/lib/ruby/gems/1.8/gems/rake-0.8.4/lib/rake.rb:1999:in 'top_level'
C:/ruby/lib/ruby/gems/1.8/gems/rake-0.8.4/lib/rake.rb:1977:in 'run'
C:/BuildAgent/plugins/rake-runner/lib/rb/runner/rake_ext.rb:311:in 'standard_exception_handling'
C:/ruby/lib/ruby/gems/1.8/gems/rake-0.8.4/lib/rake.rb:1974:in 'run'
C:/BuildAgent/plugins/rake-runner/lib/rb/runner/rake_ext.rb:179:in 'run'
C:/BuildAgent/plugins/rake-runner/lib/rb/runner/rakerunner.rb:40

Here’s what rake Application.top_level looks like:

# Rake lib
def top_level
    standard_exception_handling do
        if options.show_tasks
            display_tasks_and_comments
        elsif options.show_prereqs
            display_prerequisites
        else
            top_level_tasks.each { |task_name| invoke_task(task_name) }
        end
    end
end

The TeamCity version has overridden standard_exception_handling:

# TeamCity Rakerunner
def standard_exception_handling
    begin
        yield
    rescue Rake::ApplicationAbortedException => app_e
        raise
    rescue Exception => exc
        Rake::TeamCityApplication.process_exception(exc)
    end
end

So the yield there is yielding to the result of executing each top level task. The rake task implementation has an execute method like:

# TeamCity Rakerunner
def execute(*args, &block)
    standard_execute_block = Proc.new do
        standard_execute(*args, &block)
    end

    if application.options.dryrun
        Rake::TeamCityApplication.target_exception_handling(
            name, true, "(dry run)", &standard_execute_block)
    else
        Rake::TeamCityApplication.target_exception_handling(
            name, true, &standard_execute_block)
    end
end

Where Rake::TeamCityApplication.target_exception_handling raises Rake::ApplicationAbortedException. We can disable this behaviour by commenting it out.

Solution

The Quick solution is to comment out the line that raises the exception:

# rake_ext line 158, method TeamCityApplication.process_exception

raise Rake::ApplicationAbortedException, exc

The problem now though, is that a failing task will not be reported automatically.

In this project that’s fine because we’re doing this ourselves in a subsequent step, but for other projects using the rakerunner this may prove confusing.

Rake — shell task failure

For my current project, we’re running cucumber as one of our tasks. Our report generation runs afterwards — which we need to happen regardless of cucumber’s exit code (cucumber exits with code 1 when any tests fail).

Handling shell task failure

We noticed that if cucumber fails, rake exits immediately — without running our subsequent report tasks.

We’re running cucumber with rake’s ruby function — which runs the ruby interpreter in its own process by invoking a shell command.

Pseudo-stacktrace (irrelevant source lines omitted):

Shell::CommandProcessor.system(command, *opts)
Rake::Win32.rake_system(*cmd)
FileUtils.sh(*args, &block)
FileUtils.ruby(*args, &block)

The key to this is the behaviour of rake’s FileUtils.sh. Examining the source shows:

If it is not supplied a block, and the shell command fails, and rake exits with error status.

Aside: For some reason, Kernel.system echoes errors.

Example

This task will cause rake to exit immediately with status 1:

task 'this-one-fails' do
    ruby("xxx")
end

while the following task does not cause rake to terminate  normally, any subsequent task will still run:

path = File.expand_path(File.dirname(__FILE__) + ‘/output/cucumber-manual.txt’)

cuke_path = ‘C:/ruby/lib/ruby/gems/1.8/gems/cucumber-0.3.3/bin/cucumber’

ruby(“#{cuke_path} -h > ‘#{path}’”) do |success, exit_code|

puts “Exited with code: #{exit_code}” if !success

end

task 'this-one-succeeds' do
    ruby("xxx") do |success, exit_code|
        puts "Exited with code: #{exit_code.exitstatus}" unless success
    end
end

Getting the last exit status

The $? variable contains the status of the last child process to terminate. This may be useful if steps subsequent to the failed one require this information.

Cucumber — ANSI formatting

When cucumber output is piped to file on disk, it may contain non-printable characters and markup, for example:

[32m6 passed steps

Actually looks like this when output to console:

6 passed steps

This is due to cucumber emitting Ansi formatting.

[ANSI escape sequences] are used to control text formatting and other output options on text terminals.

Most of these escape sequences start with the characters ESC (ASCII decimal 27/hex 0x1B/octal 033) and [ (left bracket). This sequence is called CSI for Control Sequence Introducer(or Control Sequence Initiator). There is a single-character CSI (155/0x9B/0233) as well.

The ESC+[ two-character sequence is more often used than the single-character alternative, for details see C0 and C1 control codes. Devices supporting only ASCII (7-bits), or which implement 8-bit code pages which use the 0x80–0x9F control character range for other purposes will recognize only the two-character sequence. Though some encodings use multiple bytes per character, in this topic all characters are single-byte...

This formatting can be suppressed using the:

--[no-]color

switch.