Frank is a tool that lets you write automated end-to-end acceptance tests for your iOS mobile application using Cucumber. Frank ships with a set of steps (the parts after the Givens, Whens, and Thens) that make it quick to start writing scenarios, but the project documentation recommends avoiding relying on the built-in steps for the long-term health of your scenarios:
If you start to build a large suite of tests using these steps in your feature files you will quickly find that they are trouble to maintain and can be brittle depending on how much your UI is changing. You’ll be much better off if you write your own steps which communicate the language of your specific domain.
So why does the Frank team recommend avoiding using these steps? Let's take a look at an example.
Here's a scenario that describes how a user might sign up for an account:
Feature: As a new user I want to sign up for the app using my email address So that I can access the system with my own account Scenario: Successful sign up Given I launch the app And I touch the button marked "Sign up with Email" And I wait for 1 second When I type "email@example.com" into the "Email address" text field And I type "Dan" into the "First name" text field And I type "Wellman" into the "Last name" text field And I type "New York, NY" into the "Your location" text field And I type "password" into the "Password" text field And I touch "Sign up" Then I wait to see "Welcome!"
This scenario works, so why do the Frank and Cucumber folks recommend avoiding this?
- It's 10 lines long. That's long enough that I can't scan it quickly to determine the intent of the test.
- It's not immediately clear which parts of the test are more important than others. Are they all equally important, or some more than others? There are specific values listed for each form field - are those values important? I can't tell.
- It's brittle in the face of future UI changes. Specifically, here are some changes that might require the test to be changed:
- The names of the text fields change. The test lists all the names (actually the accessibility labels) of the fields. Changing any of these will require changing the scenario.
- The validation rules change. I infer from this test that all the entered values in the form fields are valid data, since this test is for successful sign up. But if the password validation rules change such that 'password' is not valid (and I hope this is very likely change, for security's sake!), this test will start failing and I'll need to change it.
- The text on the welcome page changes. Right now it checks that the text "Welcome!" appears on the last step, but if this text is removed or changed, the test will fail.
There's a name for this style of test -- it's called an imperative style. It is a series of step-by-step instructions for how a user interacts with the application. Like the example shows, imperative style tests can often be long, have lots of details, and can be brittle in the face of change.
When writing tests using Frank and Cucumber, if you see several of the built-in steps, the test may be imperative.
The Declarative Style
The opposite of an imperative style is a declarative style; it describes what the user does but not necessarily how. This requires writing your own steps, using a language in your mobile application's domain. For example, if you are practicing Domain-Driven Design, you might choose to describe your scenarios using the ubiquitous language of your project.
Here's the same test above rewritten in a declarative style:
Scenario: Successful sign up Given I have chosen to sign up with an email address When I sign up with a valid email, full name, password, and location Then I see my welcome screen
We see some things have changed in the test:
- It hides the names of the form fields and which button the user taps to submit those credentials.
- It hides how and what they enter, focusing only on the details that are relevant to the test. If I want to describe what makes an input field valid or invalid in a Cucumber test, I'd split that out into a set of separate tests for each interesting field.
- The first step that talked about starting the application is now hidden inside the implementation of the first step. That's an implementation detail that I don't think is key to understanding this scenario.
- It tells us what view/page the user ends up on after logging in, not how to identify it.
This means that we've pushed down the details of how the user enters this data into the step definitions. I won't show all the step definitions in this article, but here's a sample implementation for one step:
When(/^I sign up with a valid email, full name, password, and location$/) do fill_in 'Email address', :with => 'firstname.lastname@example.org' fill_in 'First name', :with => 'Dan' fill_in 'Last name', :with => 'Wellman' fill_in 'Your location', :with => 'New York, NY' fill_in 'Password', :with => 'password' touch_sign_up_button end
Notice how this has changed to use the Ruby Frank helper methods directly and some custom helper methods. For information about how to implement these custom steps in Frank Cucumber, see the Writing your own steps documentation, and Pete Hodgson's Writing Your First Frank Test blog post.
I wasn't sure about my "When" step - is it important to list the required fields, or could I have simply said "When I sign up with valid registration details"? I decided which fields are collected was important to communicate with our business stakeholder, so I listed them.
There's a little bit of a lie in here: At some point we might add another required field to the registration form, let's say phone number. I could cheat and add some code in my step implementation that fills in phone number, without changing the English step. That is, I could still leave the step as "When I sign up with a valid email, full name, password, and location", even though the underlying implementation actually fills in a phone number. I see this as a form of duplication - both the scenario English and the step implementation list the required fields. However, I'm willing to live with it in this case because I think listing the details is an important part of what the Cucumber scenario is communicating. Another alternative might be to put the required information in the Scenario description instead of keeping it in the executable steps.
The Declarative Style is Not Always Better
An imperative style may be appropriate for describing how a user interacts with your mobile application. If a given feature has some particularly interesting UI, or the number of gestures is important, describing 'how' may be a focus point of your scenario. "The Cucumber Book" (p. 75 in "Nesting Steps") says:
It’s entirely appropriate for us to go into this much detail about the authentication process in this scenario, because that’s where our focus is.
On the other hand, it also describes the user's interaction experience. This information could be useful feedback for a stakeholder who is interested in the user experience of signing up for the site. When we focus on the steps required to sign up, I can easily see that there are four pieces of data required for registration via e-mail.
As a longer term practice for acceptance tests in general (whether they are written using Cucumber or another tool), I try to lean towards declarative tests and use the imperative style when I know it's an exceptional case like communicating how a user swipes, types, and taps in the app. I think the concise nature of declarative tests hides details which in turn requires fewer changes to the scenario steps - I find editing code faster to change than changing the English/Gherkin in the scenario text files and possibly updating the step matchers.
So Why Does the Frank Team Ship Steps They Don't Recommend Using?
Frank includes a set of built-in steps that make it easy to write scenarios for iOS applications - taps, swipes, form field entry, etc. It can be very fast to write new tests using these steps. In the "Included Steps" page of the Frank documentation, it qualifies that using these built-in steps can be a good way to automate short-lived tedious tasks, or documenting steps to recreate a defect. The challenge is that these imperative tests can get long quickly and may obscure the intent of the scenario. Refactoring to use a declarative style with some steps using an imperative style can make these tests shorter and more expressive.
For More Information
For another take on why writing with imperative steps might lead to hard-to-maintain Cucumber tests, this story from Aslak Hellesoy, the founder of the Cucumber project, describes why the Cucumber-Rails team removed its built-in imperative web browser steps.
If you'd like to read more about the difference between imperative and declarative styles, see Ben Mabey's blog post "Imperative vs. Declarative Scenarios in User Stories". I found Matt Wynne's talk "Why your step definitions should be one-liners and other pro tips" to have a lot of great tips for writing clean, communicative Cucumber tests as well.