A great presentation from the recent 2018 Selenium Conference with a few shout-outs to this blog. I really like what Wes and Kurt were able to achieve using AWS Lambda – amazingly fast e2e tests running in parallel.
John X asks…
I have dodged the AngularJS /Protractor bullet for many years now. May last foray, some 5 years ago, was a cluster to put it mildly! Cucumberjs/Angularjs/Protractor/chai/mocha/ the stack was in its infancy and failed miserably!! Many cycles were spent pulling fixes from our own repos instead of waiting for PRs to get done. This was in stark contrast to the ease and stability of the automation I wrote using Watir-Webdriver and eventually Watir.
I am now faced with automating the regression test cases for an Angularjs App.
Question: Do I finally jump back into the using stack that almost caused me to lose my mind, or is it possible to use Watir/Selenium to build out meaningful e2e UI automation for an angularjs app as we dawn on 2019?
It’s still my opinion in 2018 that writing e2e tests in Node using either Protractor or WebDriverJs is still more difficult than using Watir in Ruby.
Sure using async functions with await commands makes things easier than before (see examples in our project), when you would have first come across Protractor, but I still feel like there’s a lot of catching up to do to get to the stability and ease-of-use of Watir.
My decision would come down to whether others on your project are going to be comfortable maintaining tests in ruby – if they are I’d use Ruby and Watir, otherwise I’d revisit Protractor if they really want to use Node.
This is an approximate transcript of the talk I delivered at TestBash in Sydney on Friday 19th October 2018.
Today I’d like to share my story about how we started with automated end to end testing at WordPress.com since I started at Automattic over 3 years ago.
“I did enjoy reading the article about e2e test on wordpress. I noted that e2e test are in a separate repo.
My question will be what is the workflow to make sure new changes does not break the e2e test on pull request ?
For example, if a developer work on some changes, then they need to change the e2e test first and make sure everything pass, however the environment on the pull request might not be stable, developer can overwrite each other changes”
Thanks for your question Liam.
We have reasons for and benefits in having the WordPress.com e2e tests in a separate repository:
- The e2e tests test the entire WordPress.com experience so these test things that happen in different repositories (for example our Calypso user interface or services/API) and having them in the user interface repository isn’t really representative of what the breadth of their scope;
- Making changes to the e2e tests are easier in a separate repository since we don’t have to deploy e2e PRs that don’t contain functional changes (we deploy every merge to our master branch immediately dozens of times per day)
The obvious downsides are:
- How do we make sure e2e tests know about incoming AB tests?
- How do we couple new changes to updates in the e2e test repository?
For incoming AB tests we make sure that our e2e tests know about the change by ensuring we create a matching PR in our e2e tests that override our AB tests during our test runs.
If someone updates the AB tests in Calypso they’re politely reminded to update the e2e tests:
For making sure e2e tests are up to date we automatically run two (of about 40 total) of the most critical e2e tests (in three browsers) when a PR is ready to be reviewed. These can fail and indicate a change is necessary to the e2e tests (or something is broken!)
There’s also a label we can add to any PR that runs the entire set of e2e tests against a PR running live and reports the result back to that PR:
If changes are required to the e2e tests someone can create an e2e PR with the exact same branch name which will be used to run against the feature changes before they are merged. This means PRs can be coupled and tested together but merged separately.
To answer the second part of your question I understand it to be about conflicting changes? One of our key philosophies for work is “merge early and merge often” so we make sure that PRs are short-lived and merged quickly to minimize the chance of conflicts. These still do happen occasionally, we just deal with them as they come up.
Whilst there’s been some downsides to the separate repositories all-in-all the benefits continue to outweigh the downsides but we constantly assess and at any point in time we can easily merge them if need be.
This month marks my three year anniversary (Matticversary) at Automattic, which means I have been solely working from home for three years now.
Working from home every day can at times feel like either, or both, the best thing in the world, or the worse thing ever invented. I personally don’t believe that full-time working from home is suitable for everyone as I’ve found it orders of magnitude more difficult than working in an office environment.
It’s taken me almost three years to really work out what works well for me as a full time “telecommuter” – and I thought I’d use my work anniversary to share some of these.
Like anything, you’re mileage my vary, but these tips have definitely helped me over the years.
- Create a morning routine: since I don’t commute I’ve found it hard to mentally separate finishing non-work time and starting work time in the morning which is something a commute provides (without you realising). Since I have a flexible schedule I make sure that in the mornings I have a routine. I am lucky enough to live right next to a beautiful forest so I go for a 4-5km walk each morning which not only provides exercise and nature exposure benefits but symbolises to my brain that work begins at the end of my morning walk. This can sometimes be hard to justify to myself as some of my colleagues are finishing up around the same time as I am walking – and I may miss checking in with them – but I definitely need this morning routine.
- Dedicate a home office space: as a family of five we live in a small cottage so it’s hard to justify a dedicated home office space in our home, but I’ve found it vital for separating my home and work lives. When I’m in the office I am working – when I am not in the office I am not working. My three children know they can’t play in my office – if I take a break to spend time with them I come out of my work space. I read in The Organized Mind book that “students who studied for an exam in the room they later took it in did better than students who studied somewhere else”. I believe this is due to the mental association.
- Dedicate a computer just to work: again this seems luxurious but having a dedicated work computer, or at least a dedicated computer profile/account, creates a good association between that computer and work. From The Organized Mind book again: “Create different desktop patterns on them so that the visual cues help to remind you, and put you in the proper place-memory context, of each computer’s domain“.
- Disable app notifications on your phone: unless I’m an in person work meetup (which happens a few times a year) I disable app notifications on my phone for work apps (like Slack, WordPress and GMail) which means I rarely check or open those apps outside of my work environment (see #2 and #3).
- Avoid checking work in bed: it’s very tempting to check on work things on my mobile phone in bed, particularly when you work for a global 24×7 365 company like I do, but this disrespects the importance of sleep in our lives. As Matthew Walker, one of the world’s leading sleep scientists says in Why We Sleep: “The silent sleep loss epidemic is the greatest public health challenge we face in the twenty-first century in developed nations. If we wish to avoid the suffocating noose of sleep neglect, the premature death it inflicts, and the sickening health it invites, a radical shift in our personal, cultural, professional and societal appreciation of sleep must occur.” You can use a morning ritual (see #1) to signify when you’ll start checking work things, instead of opening your eyes in bed and checking work immediately.
- Create an end of work day ritual: I am my most productive in the hours leading to dinner time so I use dinner time as my end of the work day ritual when I transition from working to not-working. I also have a couple of apps on my phone I like using and I don’t let myself use them during the day so I can signify at the end of the work day when I’m not working. Like a play reward that a working dog gets when they’re finishing working.
- Find social activities outside of work: working from home all the time can get very, very lonely and you can feel particularly isolated – especially working for a global company with flexible work hours, and having no team members close to your timezone. It’s important to find social activities outside of work as working in an office a lot of this is a given – even something like chatting to someone in the coffee room you won’t get working from home. I’ve tried attending Toastmasters, hiking groups and making sure I get along to meetups and user groups. Also I recommend meeting up with friends and ex-colleagues for lunches – travelling to where they work. This gives you a good break from working from home and can reduce the feeling of isolation.
- Spend some time away from home on holiday: this is one of the most expensive parts of working from home in my opinion, and it is easy to blow all the cost savings of not-commuting on holidays away from home that you need so you don’t feel like you’re at work. If you take time off and just stay at home (sometimes known as a “staycation”) I have found I don’t feel like I’ve really disconnected from work as I’m still thinking about work being home (even though I have a dedicated workplace – see #2). I’d love to work out a way to solve this one without spending more and more money on holidays away from home.
- Have something to listen to: I listen to electronic dance music mixes on Mixcloud all day long – but your tastes may vary. This provides a soundtrack for working.
- Set yourself a daily goal: I work in a Results Only Work Environment where my employer doesn’t care so much about my hours as long as I achieve results. This is both good and bad (I could write an entire post on it) but you need to be disciplined and set your set a daily goal and stick to it. Sometimes this means working longer than you planned, other times it’s important if you achieve your daily goal earlier than expected you finish earlier than expected to make up for all those other times. Otherwise you can get caught in a never-ending work loop.
- Close loops: working in a distributed work environment I find I have lots of different things on the go at any one time – this is because I might be expecting a reply or some work from someone else on something and everyone works different hours and lives in various time-zones. I find it’s important to re-visit all these “open loops” and close them, just so your mind doesn’t think about them anymore. This is sometimes a case of leaving a comment like “I didn’t hear anything back on it so I’ll close it” to close the loop – and remove it from your focus and bandwidth.
- Find tasks that you can finish to completion outside of work: most of the things I work on have a medium-to-long event horizon – or an indefinite horizon as I await asynchronous feedback – so I’ve found hobbies that I can finish completely in one attempt – gardening, cooking a meal, washing the car etc. are very important for my mental well-being as I get a good sense of accomplishment with these tasks and hobbies. See #11
- Getting things done creates motivation to do more things: when I lack motivation to start something I find the smallest thing that I can get done and complete that. The sense of accomplishment of getting something done motivates me to do more.
- Over-communicate: I can’t emphasise this enough – no one will see you in the corner of an office getting stuck on a problem – you need to constantly communicate everything you’re doing – good and bad. Communicating context creates empathy in a distributed team.
- Set expectations for family and friends: some people assume you’re not really “working” if you work from home. Setting aside a work space (#2) helps – but you’ll need to continually explain to people that even though you are at home you are actually at work.
- Don’t be afraid to ask ‘stupid’ questions: in a work from home environment there’s bound to be other people thinking the exact same thing as you – so ask away!
- Don’t be afraid to be a bit weird: “Today you are you, that is truer than true. There is no one alive who is youer than you” (Dr Seuss)
That about covers it for now. As I said, working from home can be very rewarding, but it’s one of the hardest challenges I’ve ever faced, and after three years I’m still working at it!
I have a question to ask your post on iterative vs. Incremental Software Development:
In the incremental approach, the few features implemented in all of their requirements can be changed after user feedback? Or, does this only happen with the iterative approach?
Thanks for your question Mario. This can, and should, happen with both approaches, but I’d say the incremental approach is actually more likely to get customer/user feedback as it’s a more polished, albeit smaller, user experience, and therefore more likely to land in front of users. The painting analogy isn’t the best as the requirements and level of ‘done’ are pretty clear, but the general rule is to seek feedback as soon as possible, and both approaches are designed to do just that.
Making a code change when using a distributed version control system (DVCS) like Git is usually done by packaging a change on a branch as a “pull request” (PR) which indicates the author would like the project to “pull” the change into it.
This was, and is, a key part of open source projects as it allows outside contributors to contribute to a project in a controlled way, however many internal software development teams also work in this fashion as there are many benefits of this approach over committing directly to a shared branch or trunk.
I’ve seen the pull request approach have a positive impact on software quality since pull requests facilitate discussion through peer reviews and allow running of automated tests against every commit and change that is proposed to be merged into the default branch.
What is a LGTM PR culture?
I’ve also seen some negative behaviours emerge when moving to pull request based development which I’ll call a LGTM PR culture.
LGTM is a common acronym found in peer reviews of pull requests which means “Looks Good To Me”, and I’ve seen teams let unsuitable changes through with LGTM comments without doing solid peer reviews and testing.
How do you know if you have a LGTM PR culture?
One way to “test” your peer review process is by creating PRs and leaving a subtle bug or something not quite right that you know about in the PR. When it gets reviewed do you get a LGTM? I did this recently and whilst the PR didn’t even do what it was meant to do I received a LGTM 😕
How can you move away from a LGTM PR culture?
It’s tempting to just tell everyone to do better peer reviews but it’s not that easy!
I’ve found there’s some steps that an author of a pull request can do to facilitate better pull request reviews and lead towards a better culture.
1. Make pull requests as small as possible:
The smaller the pull request the more likely you’ll get specific and broad feedback on it – and you can then iterate on that feedback. A 500 line change is daunting to review and will lead to more LGTMs. For larger refactorings where there’ll be lots of lines changed, you can start with a small focussed change and get lots of review and discussion, and once the refactoring is established with a smaller example you can easily apply that feedback to a broader impact PR that won’t need as much feedback as you’ve established the new pattern previously.
2. Review your own pull request
Review your own work. This works best if you do something else then come back to it with a fresh mind. Anything you’re unsure about or doesn’t look right leave it as a comment on your own PR to encourage other reviewers to look closely at those areas also.
3. Include clear instructions for reviewing and testing your pull request
A list of test steps is good as well as asking for what type of feedback you’d like – you can explicitly ask reviewers something like “please leave a comment after your review listing what you tested and what areas of the code you reviewed.” This discourages shortcuts and LGTMs.
4. Test your peer review process – see above.
Developing software using pull requests can mean much higher quality code and less technical debt due to the feedback on peer reviews that accompany pull requests. As an author you can take steps to ensuring pull requests are easy to review and encourage a culture of effective peer reviews.