Recently I finished a new feature of my Customfield Editor Plugin for Atlassian JIRA®. Normally the plugin provides mostly REST API Features which are already heavily tested by an REST-API-Testuite. The new feature on the other hand involves more UX and I wanted to have it tested automatically.
Then further questions arouse - Why should we test? Who should test? What to test? When to test? And how to test?.
Being on other Projects and asking for end-to-end tests mostly there were none, they were poorly maintained or didn’t even run successfully. This brings the Testing Pyramid to mind and the reality of most projects test coverage versus the desired state.
That being said we want to dive into each question and answer them in relation to agile methodologies like Scrum. That means setting the need to test software in relation to the Manifesto for Agile Software Development and the Twelve Principles of Agile Software
Firstly we should look at the agile aspect. When we think of ‘Working software over comprehensive documentation’ then I come to the result that well written tests that actually work, that are well maintained and bring constant stability into the project should count to the ‘Working Software’ part. To think of automated tests as part of the working software makes automated testing as important as delivered features in an agile context. I don’t say that you can write only tests for three sprints in a row and not deliver business value but that automated tests should not be seen as some optional thing you can do but as crucial part of the Definition-Of-Done.
Secondly let’s have a look at the monetary aspect. The above paragraph might be enough to convince agile thinking people but as you might know there are still some people around that think in numbers. So let’s try to convince them too. From my experience I can say that I have been to many projects and the ones with a well maintained amount of automated tests had the following benefits. Faster time-to-market since releases can be rolled our faster because your tests assure you that the software works. Faster reaction to change since you can implement new features faster when the processes are already there to develop and test new features. And in my opinion the biggest factor is that software developers have a less painful work days since they do not need to bugfix bugs in the middle of the night after a not-so-successful release rollout which leads to more motivation and better software which should lead to more business value. Also bringing new members into a project is simpler. New developers can run the tests and understand the project faster which means that the learning curve is steeper and they can contribute to the business value faster.
Lastly just some commentary. For me the real question is: ‘Would you buy a car which just in theory should be able to drive but no one actually tested that?’ Sure the factory tells you that of the 1000 cars they produced they tested one and that could drive. But would that really convince you? Bring that common sense into your project and just write tests. Your life will be much easier on the long run.
Now that we have heard about the ‘why’ let’s talk about the ‘who’. In agile projects we have a very clear separation of responsibilities. In Scrum we have the ‘Team’ which is fully responsible for the ‘how’ part regarding used technology. The ‘Scrum Master’ is fully responsible for the ‘process’. And the ‘Product Owner’ is responsible for the ‘what’. That being overly simplified we can say it even more simpler: ‘The Product-Owner says WHAT he wants. The Team decides HOW to do it. And the Scrum Master makes sure everything runs smooth.’. So in theory the responsibility for the testing sits in the ‘Team’ and somewhere around the ‘Scrum Master’.
But in reality that is not enough. If you just do it that way you might just end up with poorly maintained tests no one really cares about. So rather than explaining what could go wrong I will give you an outline of how I see the ideal agile responsibilities regarding automated tests.
What should the Scrum-Master do?
What should the Product-Owner do?
What should the Team do?
Alright. I have talked waaaaay too much about theoretical stuff so let’s just have a look at the new feature I implemented and inspect what to do from there on out. The feature is called ‘#11 Hide administrator Sidebar-Navigation Links from Non-Administrator Users’ and it basically should do the following:
You can see a preview of the feature below. On the left side I am in a Browser-Session logged into JIRA as administrator. On the right I am logged in as a non-administrator user. The GIF shows the complete feature in use.
So now we know what feature we want to test but what parts on which layers of the App do we test? That depends on how you implement your features and how your application is structured. In my case:
So that is how the feature is implemented. And I decided to test it this way:
What rules to follow? What about agile?
As already written above you should run your tests on every code change in every branch. Therefore it might be a very good idea to setup CI-Server like Jenkins or Bamboo. And configure them to run your tests on code changes.
Now it finally comes down to the ‘how’ part. I will only write about the End-to-End test here, since Unit-Tests and REST-API Tests are not the scope of this blog-post (I might write about those some time).
You already saw the Workflow-GIF above and what steps our End-To-End Test will test is:
We use a new testing tool called CodeceptJS which is really nice and leads to understandable test code. Let’s get started to set up CodeceptJS for our End-to-End Test.
You need Node.js installed and have a Project initialized with ‘npm init’. First we install a bunch of tools we need to write some runner to start the tests
After that we install PhantomJS which will server as a replacement for the Selenium Server and test the installation.
Should output 2.1.1.
Now we want to configure CodeceptJS. You can do this with the init command, so check out the Quickstart Guide.
This will create this codecept.json config (if you select options accordingly).
Add the script to package.json so that we can run npm run test:e2e later
Create a ‘run-e2e-test.js’ file and put the code there which will start the PhantomJS Selenium-Server and run the CodeceptJS tests when we execute ‘npm run test:e2e’.
Since our JS-Code is ES6 already we need a .babelrc file so also create that one
Now we change the ‘tests/end-to-end/codecept-js-steps-file.js’ file an define some Actors for login and logout. Actors are basically things you need more than once and you can use them anywhere in your tests. E.g. the ‘loginToJIRA’ actor can be called from the test with I.loginToJIRA(‘admin’, ‘admin’); which logs in the user admin with password admin.
Now we create the actual End-To-End Test and create the ‘tests/end-to-end/admin-settings_test.js’ file and paste the following.
You should be able to understand the test just by reading it. I used some pretty long ids in my HTML code that are self explanatory so that I know what ‘I.click(‘#cep-admin-settings—navigation-links-form—hide-from-non-admins-radio-true’);’ means. My used ids explain this way:
Ok now we run the npm run test:e2e and have of course JIRA already running via ‘atlas-run’.
If you need help with JIRA, Confluence or any other Atlassian product we offer Atlassian Experts Services. We are experienced in the Setup of Atlassian Products in your own datacenter and on AWS. We also offer consulting via webmeeting worldwide. Feel free to contact us via the contact form below.