I’ve almost certainly missed some frameworks here. Let me know what’s missing and we’ll get it added asap.
Despite the fact that there are defacto testing frameworks for both Android and iOS a number of alternatives have cropped up. These 3rd party frameworks address some of the shortcomings of the vendor provided options by adding additional features, addressing ease of use and adding language options. The biggest differentiators between these frameworks are typically 1) Preferred testing language (i.e. JavaScript, Obj-C, Cucumber, Ruby, etc.) 2) Dependencies on things like a web server 3) Required project changes 4) The ability to run tests inside an IDE (XCode/Eclipse).
I think the cause of this fragmentation has been the explosive success of mobile over the last few years. The testing and tools space just hasn’t been able to keep pace. That said, as one of the sponsors of the recent Mobile Testing Summit we saw first hand that there is a growing community dedicated to making testing a first class citizen for mobile. The fragmentation problem is definitely a big one but it’s likely that within the next year or two a clear winner will emerge and that will be good for everybody.
Here we’re attempting to layout a number of these popular testing frameworks useful specifically for native mobile development. Our motivation is to try and bring some clarity to a space that is still highly fragmented. This list is broken up into two categories. The first category is Unit Testing Frameworks. These are frameworks that are meant to test small pieces of code in isolation. The second is Functional Testing Frameworks, which are meant to test at the UI level. Functional tests test the system as a whole, essentially reproducing user interaction.
Unit Testing Frameworks
OCUnit
OCUnit is Apple’s “Out Of The Box” Unit Testing solution. It’s been included with XCode since version 2.1. Probably the biggest advantage is it’s tight integration with XCode. These tests are often quite easy and fast to run as well because there’s no dependency on having a running Simulator.
- (void) testCentigradeFreezingPoint {
NSString *kelvinFreezingPoint = @"273";
NSNumber *centigradeFreezingPoint = [transformer transformedValue:kelvinFreezingPoint];
STAssertEquals(0, [centigradeFreezingPoint intValue],
@"Centigrade freezing point should be 0, it was %d instead!",
[centigradeFreezingPoint intValue]);
}
JUnit
JUnit is the corresponding “Out Of The Box” solution for Android developers. Similiarly it is tightly integrated with Eclipse but can also be run from the command line using Ant. Test runs are quite slow because of the dependency on a running emulator.
public class SomeTest extends AndroidTestCase {
public void testSomething() throws Throwable {
Assert.assertTrue(1 + 1 == 2);
}
public void testSomethingElse() throws Throwable {
Assert.assertTrue(1 + 1 == 3);
}
}
Kiwi
Kiwi is a BDD framework for iOS with a focus on simplicity and readability. It reminds me a lot of RSpec for you rubyists, which is definitely a plus. Tests are written in Objective-C and run very nicely inside Xcode.
describe(@"Team", ^{
context(@"when newly created", ^{
it(@"should have a name", ^{
id team = [Team team];
[[team.name should] equal:@"Black Hawks"];
});
it(@"should have 11 players", ^{
id team = [Team team];
[[[team should] have:11] players];
});
});
});
Cedar
Cedar is the brainchild of the folks over at Pivotal Labs. Like Kiwi it’s also a BDD style framework but has a bit more functionality. Some interesting features include Matchers (a cleaner approach to assertions) and built in type mocking. The installation process even installs Xcode code snippets to make writing tests even easier.
spy_on(someInstance);
id fake = fake_for(someClass);
id anotherFake = fake_for(someProtocol);
id niceFake = nice_fake_for(someClass);
id anotherNiceFake = nice_fake_for(someProtocol);
NSUnit
NSUnit is a unit testing framework for Objective-C projects. NSUnit builds on top of SenTestingKit and provides a more Objective-C friendly API, particularly for testing against Foundation classes. NSUnit adds some really useful API’s including an Assert class, primitives handling and a fails method on Assert that makes performing failing tests a one-liner.
[Assert that:name isNot:[AnEmpty string]];
[Assert that:name isNot:[Equal to:@"Omar"]];
[Assert that:name is:[Equal to:"McNulty"]];
[Assert that:someArrayOfInts areAll:[Less than:@20]];
[Assert fails:^{
[WebService connectToUrl:nil];
}];
Functional Testing Frameworks
UIAutomation
Like OCUnit, UIAutomation is deeply integrated with XCode and is the default Functional testing framework for iOS. Tests are written in JavaScript and can be run on either the Simulator or a Device.
var testName = "Test 1";
var target = UIATarget.localTarget();
var app = target.frontMostApp();
var window = app.mainWindow();
var textValue = window.staticTexts()["RecipeName"].value();
if (textValue === recipeName){
UIALogger.logPass( testName );
}
else{
UIALogger.logFail( testName );
}
Appium
Appium is a newly introduced framework from the folks at Sauce Labs. Appium is built on top of UIAutomation which is nice because no modifications are needed to your project. Also, tests can be written using any language. It does have a dependency on running a web server which could be a win or loss depending on your setup. Perhaps the most notable feature of Appium is that it is designed to be compatible with all off the shelf Selenium client libraries.
Robotium
The name pretty much says it all, “It’s like Selenium, but for Android”. There really aren’t a lot of competing frameworks on the Android platform so Robotium has become quite popular. It’s even supported by a number of hardware testing services like Testdroid.
public void testPreferenceIsSaved() throws Exception {
solo.sendKey(Solo.MENU);
solo.clickOnText("More");
solo.clickOnText("Preferences");
solo.clickOnText("Edit File Extensions");
Assert.assertTrue(solo.searchText("rtf"));
}
GHUnit
GHUnit was written by Gabriel Handford and has some really interesting features like the ability to test UI elements and parallel and asynchronous tests. It also has a feature that allows you to record a diff of view images so you can see how they’ve changed over time.
- (void)testFoo {
NSString *a = @"foo";
GHTestLog(@"I can log to the GHUnit test console: %@", a);
// Assert a is not NULL, with no custom error description
GHAssertNotNULL(a, nil);
// Assert equal objects, add custom error description
NSString *b = @"bar";
GHAssertEqualObjects(a, b, @"A custom error message. a should be equal to: %@.", b);
}
Calabash
Calabash is actively developed and supported by the guys at LessPainful. It’s meant to be a cross platform tool for enabling tests to interact with both native and hybrid applications. In theory actual tests can be written with any Ruby based test framework but Cucumber seems to be the most popular.
Feature: Rating a stand
Scenario: Find and rate a stand from the list
Given I am on the foodstand list
Then I should see a "rating" button
And I should not see "Dixie Burger & Gumbo Soup"
When I touch the "rating" button
Then I should see "Dixie Burger & Gumbo Soup"
KIF
KIF is short for Keep It Functional and was developed by the engineering team over at Square which right off the bat gives it a lot of validation. It’s 3 main design tenants are Minimize Indirection, Easy Configuration and Test Like A User. Also, the fact that tests are written using Objective-C is a big win for most iOS developers.
+ (id)scenarioToLogIn;
{
KIFTestScenario *scenario = [KIFTestScenario scenarioWithDescription:@"Test that a user can successfully log in."];
[scenario addStep:[KIFTestStep stepToReset]];
[scenario addStepsFromArray:[KIFTestStep stepsToGoToLoginPage]];
[scenario addStep:[KIFTestStep stepToEnterText:@"user@example.com" intoViewWithAccessibilityLabel:@"Login User Name"]];
[scenario addStep:[KIFTestStep stepToEnterText:@"thisismypassword" intoViewWithAccessibilityLabel:@"Login Password"]];
[scenario addStep:[KIFTestStep stepToTapViewWithAccessibilityLabel:@"Log In"]];
// Verify that the login succeeded
[scenario addStep:[KIFTestStep stepToWaitForTappableViewWithAccessibilityLabel:@"Welcome"]];
return scenario;
}
Frank
Frank is another Cucumber based testing framework for iOS and claims that setup takes less than 10 minutes. It also has the ability to record video of your UI tests and can run on both the Simulator and Devices. A really useful feature is View Selectors which make it quite easy to interact with UI elements.
• view
• view marked:'Foo'
• view marked:'Foo Bar Baz'
• label marked:"I'm a label"
• view:'CustomViewClass' tableViewCell view marked:'blah'
cisimple 