import test blog

What’s strange about this process is we don’t give customers instructions on how to use a product or an application. When we really look “under the hood” of this process, we also quickly realise most customers take different journeys when using an application and it can be time consuming and expensive to cover every path, which leaves room for error.

To address these challenges, Bugwolf assembles elite teams of testers which act like customers. They have little to no experience with your applications and products, and are not provided step by step instructions on how to use your products or applications. They make purchases, complete profiles, create and interact with content, like any normal customer.

And this means, we cover a significantly more customer flows than a traditional testing method, and subsequently reduce the chance of defects slipping through to production, and negative social amplification.

Earlier this month, Microsoft halted the rollout of the latest Windows 10 update after users reported major personal data losses after installing the update, advising anyone who had already downloaded the update to delete it without installing—a significant and unusual move for the company.

Users on Reddit and Microsoft forums posted warnings that the latest update—Windows 10 version 1809—deleted files from their Documents and Pictures folders, the default locations for saving files in the Windows operating system. There did not appear to be a consistent or explicable trigger for this bug; some users described updating multiple systems at once and only experiencing data loss on a few of them.

While the nature of the bug wasn't immediately clear, it shouldn't have come as a surprise to Microsoft, and some are wondering why the update was released at all. Feedback Hub, the bug-reporting tool Microsoft uses for Windows 10, contained multiple complaints about data loss after installing preview releases of the update, some going back as far as three months before the public rollout.

It's unclear why Microsoft did not take any apparent action to fix the bug when the early reports starting coming in through the Feedback Hub. The company did lay off most of its dedicated software testers in 2014, prior to the release of Windows 10, and has relied more on developer testing and user reporting from the Feedback Hub for finding bugs.

On Twitter, former Microsoft engineer Hal Berenson speculated that Microsoft "put in a fix, but it didn't fix all cases," adding: "I've seen that happen many times in my career."

Earlier this year, Windows 10 had issues with bugs causing system freezes and the infamous Blue Screen of Death. These, too, appear to have been reported on the Feedback Hub well before Microsoft took action to fix them.

At this point, update 1809 is still on hold. Users who lost files to the update may have some luck restoring them with file recovery/undelete programs, but the only guaranteed way to bring them back is to restore them from a backup. In a blog post about the update and its issues, Microsoft stated that they have made improvements to the Feedback Hub to help it better detect serious problems in the future.

As organisations become large they often become more difficult to move. These businesses go through a transition from being rebellious,innovative and breaking the rules, to focusing more on productivity gains as they take on more investors and as the cost of risk increases they become more protective of these changes.

Xero, Salesforce, and Atlassian are not immune. They were established in early 2000. Their focus has most likely shifted to  more on productivity gains and sales growth than product. These companies on the rise to the top have been acquiring companies. As they integrate these companies they become big complex applications, which take too many clicks to do anything.

Meanwhile as these big complex applications grow, the rest of world has decided we want simple narrow focused products. Products that do one thing and solve one problem but solve it really well. People are also becoming more aware of user experience and the value of design. These applications were built in a different era and design is front and centre.

Bugwolf aspires to build a company like Atlassian but believe their product is complex, and our clients says the same. The testing market has changed and no longer do static apps cut it. Users want living breathing applications, which are smart and intuitive, don’t require training or lengthy installation processes to be able to use them to their fullest.  

At Bugwolf we address these challenges but taking a design first approach when building our application. Our product and company has been built with a  design first approach.. We have designed a product anyone can use and we don't believe you need so many different products and services in the testing space to solve today’s digital quality and user experience challenges.

User volatility and a brave new world of unknowns

If you’re in the business of mobile application, the traditional methods of requirements gathering generally fail to be effective. According to Gartner, “..mobile apps are a new category for most users and secondly, mobile apps are constrained by the nature of the platform and the size of the screen, so porting the workflow of a mature desktop app is not viable.” In other words, due to the complexity of multiple devices, screen size, and the challenge of reproducing a useful application workflow on a mobile platform means the flexible and dynamic Agile approach is necessary.

The volatility of mobile applications will push businesses to put more focus on the importance of internal testing to gauge user experience prior to a release. User acceptance and beta testing with customers also increases in priority in order for mobile development to get customer feedback.

Agile for development speed and increased release rate

Agile methodologies use short iterations and release more frequently. Getting a mobile app to market can’t wait for a traditional release cycle. Gartner states that the “rapid pace of change in the mobile market is putting pressure on development and operations teams to adopt rapid development and deployment practices that constantly iterate…”

Agile methodology practices provide frequent, smaller code releases to both increase speed of release to market both for the release and any bug fixes necessary to improve user experience. Mobile app users expect rapid updates and a constant flow of innovation.

Another mobile force is the frequent release of new versions of mobile OSs, devices and service providers. The sheer number of variants that affect mobile app functionality make it crucial to be able to roll with the punches and get quality code to users as quickly as possible.

Testing and the rise of user experience and usability

Gartner states that, “Most complaints about mobile apps have to do with a poor user experience.” In other words current mobile application development fails to meet users workflow needs To counter this, Gartner suggests altering testing methods to a multi-tier approach that includes testing using simulators and a subset of devices, OSs, and screen sizes.

Gartner suggests supplementing testing with usability and user experience testing “in-the-wild.” Usability, acceptance and beta testing by users with a variety of parameters become more essential for providing a quality mobile application release as they provide a method of testing how users actually work. According to Gartner, “this alone will result in a higher chance for a successful development effort.”

Testers and UI professionals rejoice! Finally the voice of the end user comes to the forefront. After all, in order to have a successful release of a mobile app on multiple devices with differing providers, it’s the user’s experience with the app functionality that determines app success or failure or as Gartner states: “This alone will result in a higher chance for a successful development effort.”

Access the Gartner report here: http://www.gartner.com/newsroom/id/2823619

In June 2009 an Air France passenger plane from Rio de Janeiro to Paris crashed without any survivors in the middle of the Atlantic Ocean.

Investigators found that the plane's speed indicator malfunctioned and the autopilot disengaged when it encountered turbulence. When the pilots took over the controls they erroneously tried to gain altitude instead of doing the opposite, causing the engines to stall mid-flight.

This tragedy once again shows that no automated systems are perfect. When it comes to complex tasks like flying a passenger jet or running tests on an enterprise software system automation without human involvement will almost always result in something bad happening.

This is the reason why an overwhelming majority of testing professionals say that test automation by itself doesn't save money or time or guarantee quality software.

But there’s just no stopping automated testing, mainly because of a sea change in business environment.

The attractions of automated testing

Across all industries, enterprises are wrestling with the challenges of digital transformation.

While business used to drive IT, the trend has reversed. Nowadays IT plays a major role in how products and services are delivered.

Enterprises no longer can get away with clunky software. Consumerisation of enterprise software and increasing customer (both internal and external) expectations means that the mandate of testing and QA departments isn't just about reporting bugs.

It has expanded to business outcomes like increases in revenue growth, customer satisfaction, and uninterrupted business operations.

Couple that with the rise of Agile and DevOps and it's easy to see why automated testing makes business sense. Manual testing alone can no longer keep up with rapid development and release cycles.

And then there’s the ROI on automated testing.

While you shouldn’t run automated tests if your only goal is to save money ROI, considerations will come up and it’s good to understand the automated testing math. While your mileage may vary, this post lays out a roadmap to calculating your ROI.

“If a tester on average costs $50 an hour and if a senior tester who creates automated tests costs $75 an hour, that would cost about $400 and $600 respectively per day per tester.

Now, consider a team of 10 testers, five senior-level and five entry-level, with a monthly loaded cost of $105,000 (for 168 hours per month). We’d get a total of 1,350 hours costing $78.00/ hour (this is assuming each tester realistically works 135 hours per month due to breaks, training days, vacations, etc.). If we automate testing, the cost of labor would remain the same, but for the effort of 3 test automation engineers, we’d achieve  16 hours a day of testing and will run 5x more tests per hour.

This results in  the equivalent of 5,040 hours per month of manual testing created by the three test automation engineers. Then, consider the rest of the team doing manual testing (7 people x 135 hours/month). That amounts to 945 hours more, ending with a combined total of 5,985 hours of testing at $17.54/hour ($105,000 divided by 5,985 hours).”

Another benefit of automated testing is the speedy discovery of bugs. The sooner the bugs are discovered, the cheaper it is to fix them (bugs found post release cost 5x more to fix than bugs found before release).

But people are still intimidated by automated testing.

The challenges behind automated testing

Running centralised tests in centers of testing excellence doesn’t cut the mustard in a decentralised and fast moving agile development environment.

While the need of the hour is agile, and in many environments, automated testing, there are still a number of hurdles to overcome.

These are the top challenges of applying testing to agile development as reported by Capgemini in their 2017 World Quality Report:

• Early involvement of testing team in inception phase or sprint planning (44%)

• Difficulty in identifying the right areas on which test should focus (44%)

• Lack of appropriate test environment and data (43%)

• Lack of a good testing approach that fits with the agile development method (43%)

• Lack of professional test expertise in agile teams (43%)

• Inability to apply test automation at appropriate levels (41%)

• Difficulty to re-use and repeat tests across sprints/iterations (40%)

• No real difficulties with testing in agile (1%)

Apart from these, here are some of the reasons why automated testing fails:

1. Not considering a testing project like a software development project

Automated testing projects are like software projects and testers are increasingly expected to be able to code or at very least be familiar with software development methodologies.

Because of complexity in setting up automated testing environments and the specialised knowledge needed for writing scripts and planning tests a small team of testers with coding experience will beat a large but poorly skilled team any day.

2. Not running enough manual testing first

It may sound oxymoronic but if you want to automate software testing you will have to know what to test manually. Knowing a system inside out will let you optimise testing for higher impact, and you can gain that insight only after you have gotten your hands dirty with the system.

You also have to keep in mind that automation is a means to an end, and not the end result. Ideally when you are running automated tests you free up the time of your team to do more high value tasks, like exploratory testing or destructive testing where testers try to break the product apart or do things that it was not designed to do.

3. Not focusing on the philosophy behind automated testing

Ad hoc, unstructured automation is a recipe for failure. Automation should always be approached in an on-going, systematic way. 

Rather than expecting your automation processes to be perfect from day one, organisations should start small and adopt a culture of continuous improvement where teams are rewarded for increasing test coverage over time.

Similarly, teams should be aware of the limitations and blind spots that may exist, even with robust automation processes, and ensure they complement automation with appropriate manual testing.

4. Not considering the complete costs of automation

Automating tests isn’t simply about the costs associated with the tool. There is the costs associated with implementing the test environment into your IT stack, the costs associated with creating the tests, and the cost of maintenance.

In fact, test environment management is one of the most important challenges respondents faced according to the World Quality Report 2016-17:

• 48% had to maintain multiple versions of the test environment.

• 46% didn’t have the facilities to book and manage their environments.

• 46% faced issues with the right mix of tools.

Another point to consider is corporate culture. If there isn’t a serious mandate from the top in favour of automated testing the resistance to change will kick in and that perfectly set up automated testing rig will sit unattended, resulting in wastage of time and money.

5. Not identifying a proper testing strategy

Most organizations run end-to-end because they focus on real user scenarios.

Developers love it because it offloads most of the testing to others.

Decision makers favour it because these tests are based on what the user will experience, and testers have the satisfaction of reporting bugs which might come up in an actual user environment.

But automating end-to-end tests is inefficient because the user is ultimately concerned not with the number of bugs, but with bug fixes. End-to-end tests can also become drawn out and unreliable if there’s no way to isolate the code which causes failures.

Automating this inverted pyramid anti-pattern will frustrate your testing efforts, and will essentially result in a Garbage In Garbage Out scenario.

Instead, do the opposite. Invert your testing strategy so that you run multiple unit tests for every end to end test. This will uncover bugs faster and generate more reliable results.

Once you have the process down pat, follow rule #2 and automate it.

Google follows the pyramid approach to testing and recommends a split of 70/20/10 in favour of unit/integration/end-to-end testing to start with.

Can automated testing and UAT co-exist?

Test automation isn’t a replacement for UAT. Automation helps you to find out problems that you already know exist much more efficiently.

But UAT is about figuring out bugs which you had no idea even existed. Bugwolf UAT teams routinely uncover hundreds of bugs while working with enterprises with extremely sophisticated automated testing systems.

Once you have run user acceptance tests you can use the insights gathered from that sweep to tweak automated test cases, and improve the overall code quality.

This process will eventually let you ship code quickly without compromising on quality and reliability.

So, what’s preventing your automated testing campaigns from showing the desired results?

This has placed an unprecedented amount of pressure on your digital products. They must work consistently and when they no longer do, customers call a single number for support, they jump into social media to share not only what they had for lunch but also their digital experiences. It’s also easy for customers to move their business these days.

At Bugwolf we address these challenges by helping organisations de-risk their digital products at significantly less cost and time than traditional quality assurance testing models. We do this by making software testing more engaging for the participants, producing a much more accelerated output and efficient results for our customers.

The bottom line is that while the pressure of quality mounts, the cost of maintaining quality does not, enabling better quality customer experiences, without breaking the bank.

Slack is one of those unicorns. It's never just one thing that makes them change the game, it's typically a combination or recipe or things, such as brand, team, design, and customer experience. In their case security also plays a big part. They have essentially taken the concept of ICQ, made it consumable for everyone, and giving it a layer of security.

When reflecting on my daily tech habits these days, Slack has grown to take up a significant portion of my spend, and I don't mean tech spend, rather where I spend most of my day when using tech tools. I spend 90% less in my email account and all file sharing and communication is there. Slack has become the glue and replaced many products I once used.

Bottom line is, I need significantly less other technology tools during my day, and this means some of those tech products and startups will simply disappear.

The judges on Shark Tank are exactly that. They have broken the rules and redistributed the wealth, but right now they are preaching the complete opposite and contradicting what they originally set out to achieve. Those deals they offer are dirty. Sure, they have a wealth of knowledge, but they are setting norms for future Australian investors and entrepreneurs.  

Sure some of the ideas and pitches are entertaining to watch, and the show provides lots of exposure for the panel. The buck stops there. If you're an entrepreneur or investor I recommend you look beyond the shores of Australia and meet some Investors and entrepreneurs in the valley. Bring back what you learn from those relationships and base decisions on these.

And doing deals like these, not only reduces the chances of entrepreneurs shifting wealth around, but also means less people are going to take a risk and start a company.

And while most people treat those links as a matter of course, the links themselves aren't perfect,  at least not yet.

We are witnessing the evolution  of a communications network. As mobile devices become more sophisticated and applications ever more varied, there are still dead zones and annoying pop-up ads.  Even so,  the quality of communication continues to increase as branded content takes the place of traditional advertisements and cloud storage increases the ability to share information. Branding has shifted from slogans and logos to generating value by providing timely information to consumers and the need for branded content will only increase as technology improves.

Connectivity continues to increase and dead zones are fewer every year. It won't be long before the average person is swimming in an ocean of communication.  As the number of applications grows, the mobile device is slowly becoming the universal everything tool of the future.  And the Internet, coupled with mobility,  is constantly opening up new possibilities  and aiming toward a seamless user experience.

As the seamless user experience grows, communication will become a blend of different technologies all contributing to providing engaging and relevant information that is accessible at the touch of a mobile screen.

Perhaps the biggest reason for seamless mobile is that society is becoming increasingly decentralized. Where once you had to go into a store to make a purchase, or into an office to do business, now these things can be done from a mobile device.  As the need for brick-and-mortar locations decreases, the need for mobile communication will increase, and it's seamless mobile that will keep the future communicating.

Many people in organisations believe that the quality of a digital product is owned by the testing team, not digital product owners. This is surprising given that many organisations have shifted their entire focus of their business to digital from the board level down, digital quality must become front and centre of everything you do.

Most testing departments see Bugwolf as a threat. We deliver cost savings, which in some cases means less jobs. We accelerate testing which means there might be less work for the testing teams, and we change the traditional way of doing things, which commonly threatens most people in organisations because of the fear of change can mean instability.

When we are pushed to the testing teams, there’s a very good chance we will not get the support we need to make real change in your organisation, reduce cost of testing, or accelerate releases.