import test blog

While some types of testing require detailed planning, exploratory testing has a minimum of planning and a maximum of execution and so relies heavily on the experience and skill of the tester.  It is the tester who determines what the analysis will be and generates logs accordingly.  Logging is done as testing is executed, with concentration on documenting the key elements of what is under test as well as outlines regarding what the tester thinks might be useful in further testing.

Exploratory testing does not substitute for other more formal tests.  There is sometimes the temptation, especially when nearing the end of time or budget, to substitute exploratory testing for more formal test procedures.  Exploratory testing is meant to complement such other forms as regression tests and user experience testing.  It is not intended as a substitute and should never be substituted for other types of detailed testing.

Exploratory testing is sometimes referred to as ad hoc testing.  Ad hoc testing is running random tests.  Test cases are chosen randomly in an effort to locate possible bugs.  Exploratory testing generally requires imagination, the tester is actually trying to break the application by pushing it, if necessary, beyond design limits.  The idea is to determine if the application can operate under unusual or high stress circumstances, and while exploratory testing can certainly find bugs, locating bugs is not always its first priority.  Under such circumstances, there may not even be established cases to test. You must then imagine a possible defect and test for it.  Basically, exploratory testing is unstructured and the quality of such testing depends entirely upon the qualities of the tester or testers involved.  Part of that quality is knowing what to test, when to test and how far to test.  It's rather easy to become a bit too enthusiastic and start testing to a point of unreality.  This is why you need skilled testers.  They are, in a sense, artists.  And the greatest gift of the artist is to know when to stop.

Agile methodology works well with exploratory testing because the two share similar basic principles.  These are:

• The importance of interactions and individuals over tools and processes.  Both agile methodology and exploratory testing place the same kind of emphasis on the individual.

• Interaction is senior to planning.  Rather than following a plan, agile methodology and exploratory testing are about responding to changes as they happen.

• Exploratory testing isn't about simply meeting contractual obligations, but is more interested in producing the highest quality software possible, this is also a part of the agile method.

• Like the actual method, exploratory testing is more concerned with obtaining working software rather than detailed documentation.

Exploratory testing isn't really concerned with coding or acceptance testing and is not really a part of user experience, at least not directly.  It is more of a way of critiquing the application by placing it in as difficult a situation as possible and then seeing how it performs.

Exploratory testing can be considered a type of user testing in that its purpose is to use the system as a user would in order to identify bugs.  While exploratory testing can be a bit more severe than normal user testing it still falls into that category.  However, there are some important differences that make this style of testing unique.  Unlike other forms of UX testing, exploratory testing is not interested in wide coverage.  Its purpose is to find the defects that don't show up under normal testing procedures.  These are defects that occur at the extreme edge of functionality, but just because they're rare doesn't mean they have little impact when they arise.  Defects can often be more severe at the edge of performance, as operations outside standard usage patterns are the most likely to push the edge of the envelope and result in system crashes.

While automated testing exists to test the system based upon how the development team intends it to perform, exploratory testing operates outside of the box and focuses on those elements of the system that don't fall so readily within standard usage patterns, and so are less likely to be tested in detail.  Consequently, exploratory testing should be used in concert with automation, but not as a substitute for automated testing.

Whether we're talking about non functional or performance testing, automation belongs in those areas of high predictability while exploratory testing exists to measure the least predictable.  Nevertheless it is important that both types of testing be carefully coordinated in order to extract the maximum value from software testing.  For example exploratory testing can find a bug that can then be added to an automated testing regime in order to prevent the defect from occurring again.

The greatest benefit of exploratory testing is the ability to find problems that cannot be found by any other means.  Of course, this requires some planning, and so exploratory testing is not totally and utterly random, it is instead a skillful investigation of the software.

While exploratory testing certainly has its place prior to release, it isn't just the development cycle phase.  It is a technique that can be applied throughout development.  It is a type of flow testing and can be applied to any module which has an established flow through.

Ultimately the best exploratory testing is done with the user in mind and reflects realistic scenarios that could happen, even if they are not likely to happen.  While exploratory testing is sometimes considered ad hoc, this does not mean that it is careless or sloppy.  It is the union of a knowledgeable tester and the software under test.  This tester then uses his or her imagination, skill and experience to push the software in ways that it could be logically pushed by users themselves.  If the exploratory tester has done his job, the software will not only have fewer bugs it will also have a greater chance of passing user experience testing with minimal difficulties.

Performance is a type of black box testing.  We don't really care what the code says, what we're interested in is how well the application or website interacts with the environment when centered around certain concepts. These concepts are operation, breakability, data volume, scalability,  and reliability.

Operation is covered by load testing, determining how the software operates under normal load. A major consideration is to determine the maximum load that can be handled without generating erratic behavior. As well as testing network and normal database interaction.

Breakability is basically how much pressure an application can take before it breaks. This is covered by stress testing which looks at what caused the system to break down, thereby identifying weaknesses that could cause unexpected or improper performance.

VOL testing looks at how much data an application can handle.  This is often done by incrementally increasing the amount of information the application must deal with.  It's an important form of testing because a database will almost always grow in size as time goes on.  An application may only need to handle a small amount of data when it goes online for the first time, however, as data accumulates the application must be able to handle an increasingly heavy database.

Scalability answers the question, " Will the application be able to support increasingly large loads?" The idea is to determine how many users a given application can support.  Scalability also addresses resources such as bandwidth, disk capacity, processor capacity and memory usage.

The purpose of reliability testing is to determine if an application can bounce back after an unexpected incident and how long it takes to return to normal operation.  This is important because there are many applications, especially those online, which are critical operations.  This is true of banking and government software that must continually operate in critical areas.

The concepts behind performance testing are the foundations on which performance testing is built.  Understanding these concepts makes it easy to comprehend the purposes of performance testing, you can then build your testing scenarios accordingly.

A native app is an application that is written in the native programming language of the device.  For example, this would be Java for Android and Objective-C for Apple.  Native apps have certain advantages, they can easily access other elements of the device such as a camera.  Native apps also tend to be faster and more reliable.

The hybrid app is built around web tech like HTML-5 and then wrapped in a shell designed to fit a specific platform.  Normally, there are also APIs that enable access to device features and hardware.

Which is best, native or hybrid, depends on a number of factors including the purpose of the application as well as cost and time available for development.  Each has advantages and disadvantages.

Native apps generally offer better performance, such as better transitions and load times, which makes them a good option for slower devices.  They provide complete access to the device operating system and hardware and they can also store more data offline.  They have disadvantages as well.  Build cost is initially higher and cost continues to increase for each additional platform.  It also takes more time to build a native app, especially when you consider that it will need to be rebuilt for each new device.

Hybrid apps use web technology which can be accessed by any mobile device with a browser.  This means shorter development time and faster deployment.  Development costs are therefore lower.  The fact that hybrid apps use interpreted code means that they will be a bit slower.  It also means that they will not have full access to the operating system on the device.

As mobile applications become more sophisticated the pros tend to stay, while the cons diminish.  Native and hybrid apps are looking and operating more and more like each other all the time.  There may come a point in the near future when it really doesn't matter anymore.  However, that is still a few years away, and whether you choose to develop a native or hybrid app depends a great deal on what you want the application to do and what your business goals are.

Applications will continue to be developed as more and more companies use apps to bring their businesses into the lives of customers through mobile devices, and those customers increasingly appreciate the convenience that mobile applications provide.

While usability standards have generally been rising, there is still considerable discrepancy between one website and another.  This opens up an avenue for increasing conversions for any company willing to put a little extra skill and care into the design of its website.

The first step is knowing what people consider to be a usable site.  This may seem easy to understand and sometimes it is. It can be common sense, such as don’t hide order buttons, present quality information and make navigation easy.  Or, it can be more sophisticated, such as the skillful use of white space. But usability can also be counter intuitive.  This is where user experience testing comes in handy.

It's also a good idea to check what keywords people are searching for.  Not only will this improve SEO, it will also help you to understand what information should be front and center on your site, and so knowing what people want is a vital first step,

Next comes  usability testing, the purpose of which is to provide important information regarding how well people can achieve their goals when using your website.  It's important to test how people relate to and interact with your site.  This can be done through user experience testing and through software that can give you real time information on how your customers interface with your site.

Remember that no matter what you are selling, you will always be selling to a particular niche of the population. So it’s important to develop an accurate definition of your average customer, in other words, to develop an accurate customer persona. This is usually done through customer interviews, and surveys can also provide backup information.  This can then be used to better match your website to the kind of user behavior that is common to your customer base.  

Collected data can then be combined with user experience to create the web design that can best influence user behavior in a positive way.  It's important to know where your target audience is in order to make certain that you are leaning the design of your web page toward the proper demographic.

Visual elements are also important.  The Internet is first and foremost a visual marketplace.  The very capacity of the Internet to present massive amounts of information means that potential customers will spend only a few seconds evaluating your website to determine if they are interested in doing business with you.  Consequently quality website design is a vital element in the process of converting a visitor to a customer.

As digital technology spreads,  people are becoming faster at evaluating what they see. They tend to make snap judgments, often staying only a few seconds on a given site.   Any part of a website that strikes the visitor the  " wrong way"  can have an adverse effect on conversions.  So, it is vital to make certain that first impressions are favorable impressions.  It's very easy to make the mistake of assuming that the visual style of the website is less important than other usability issues, such as navigation.  However, it's important to realize that user interaction is strongly related to the first few seconds of viewing your site. Navigation then becomes important once visual appeal has attracted and retained visitor attention.

Quality design encourages trust and invites the visitor to remain.  Trust is created through such things as a site that is arranged in a visually appealing and interesting way, followed by usability in such forms as ease of navigation, mobile friendliness and by providing quality information.  

In fact, usability is more than just meeting the standard criteria for ease of use.  It is the bringing together of everything from colour scheme and page arrangement to integrated apps and the call to action.  Transitions must be smooth, actions must be easy to understand and execute and the whole thing should be placed in an attractive package. And  don’t be ambiguous in your call to action.

The purpose of usability is primarily to make the potential customer feel confident and relaxed when using the site, and improving usability can affect conversions in a number of ways.  We have already mentioned how usability can increase the trust factor and it increases credibility as well.  

A well thought out website with easy navigation and a number of different ways for the customer to get his or her questions answered conveys the idea that the potential customer is dealing with an established and credible business. The best websites are the ones that put the user in control.  It doesn't matter whether we're talking about expert users or not.  

Anything that can be done to make it easier for the user to convert is a good thing.  For example, purchases should be easy to make and have as few steps as possible. The more steps there are the more likely it is that the potential customer will abandon the purchase.

Also, it's a good idea to arrange information so that the user never has to click more than twice to get that information.  Information should also be presented based on priority, and it may require user experience testing to determine correct prioritisation.

Design should be simple and straightforward and the use of graphic elements should never be confusing.  For example, the graphic presentation of information should never be easily mistaken for an advertisement, as people tend to ignore these.  

Calls to action should be straightforward and yet subtle.  Never put calls to action in big red letters or scary fonts, it makes your site look tacky and lowers credibility.  Avoid using words and phrases like " get it now" or " deal" in calls to action.  This makes the call look more like a sales pitch than a directive, and the modern jaded consumer will usually reject it.  Besides, the sales pitch should be in the text and not in the call to action.

While certain usability standards are universal, you shouldn’t rely only on doing it by the book. The book provides the foundation, but increasing conversions through usability has a lot to do with with the type of business you are in and understanding your potential customers. Ultimately, there is no substitute for a little imagination guided by careful research.

Upon making contact with some “in-the-know” IBM executives, they quickly explained the value proposition of Bluemix. We were sold. Within 12 hours we had a version of Bugwolf running on IBM Bluemix. We did this with lightning speed and gained valuable lessons along the way.

Having gone through the process, we wanted to share a couple of things that should be considered when thinking of deploying an application to IBM Bluemix. This is especially relevant when deploying between Amazon AWS, Microsoft Azure, and IBM Bluemix.

Preparing your application

This step was one of the most important parts of the process and was the real reason we were able to deploy the application at lighting speed. You don’t build a house without great foundations. So ensuring your application is decoupled and stateless failure is a key start. Because we built Bugwolf for the cloud from the start, it can scale up and down as necessary. This makes it way more flexible than if we had built a traditional monolithic architecture.

Deploying the changes

When you put the right foundations in place then it makes deploying the changes even easier and faster. This is important when you’re talking about moving your Ruby application to a new home. One of the many benefits of Cloud Foundry, which is what IBM Bluemix is built from, is just how easy it is to push new versions of our application. We can do this multiple times a day - it only takes a few minutes and has zero downtime. This means we can move as fast as our customers need us to. Particularly when there is that new “must have” feature they want.

Building resilience

Before deploying an application, it’s important to build some resilience into your architecture design to plan for unexpected things to happen. This makes so much sense for your staging and production environments. We take advantage of things like a stateless architecture, so that we can both scale as needed and also tolerate the unexpected. Plus, our database layer is a master-slave configuration with automatic failover, live replication and automated backup.

Monitor and diagnose

We had already put in place a range of devops tools and notifications including Bugsnag and Pingdom from our migration from Heroku to Amazon AWS. This made it easy to establish our platform in the new IBM Bluemix environment in no time. Because we consume services such as these via an API rather than binding them in too closely into our code, it does not matter where Bugwolf is running. Also, this means that if we want to use a different service, we can do this with minimal changeover.

Agile Testing began after computer interface had graduated from toggle consoles, green screens, paper cards and static input fields. Computer interface was no longer a matter of skill on the part of the user, special training was no longer required. Once computers ended up in the hands of non-specialists, it was inescapable that software would be called on to do more and more and testing would have to evolve.

Testing greatly increased in complexity, once computers became the universal tools they are today. It use to be that there were really only two types of errors, those internal to the machine or those made in the presentation of data, either by the machine to the user or by the user to the machine. While these are still the main types of errors tested for, the way they can manifest has greatly multiplied. And the need for improved interface continues to increase. Testing can no longer be regimented and confined to a few simple rules.

The birth of truly interactive applications has forced regimented testing to follow Cobol and RPG into the dustbin of history. This doesn’t mean there aren’t strict protocols. It just means that software testing has become as interactive as the applications being tested.

User Acceptance Testing became inevitable once the Agile Testing principles of evolutionary development, continuous improvement, adaptive planning and flexibility were accepted as part of the testing environment. UAT simply includes the end user and some previously excluded stakeholders in the testing process.

User Acceptance Testing also follows the Agile idea that presenting working software is more useful than simply handing someone documentation. While User Acceptance Testing generally happens toward the end of the development cycle, customer collaboration is important throughout as software has become so complicated that requirements can never be fully fleshed out at the beginning. There is always the need to supply working modules and prototypes as they are developed.

What was once a step by step process undertaken by small dedicated teams is becoming increasingly a matter of connecting APIs and libraries. UAT has counterbalanced the modern tendency of software development to become more iterative and impersonal by emphasizing  the importance of the human beings who use the software. As software development becomes easier, the temptation to get it done and get it out quickly must be modified by making sure that applications actually serve the people they were intended to serve. User Acceptance testing is the best way to ensure that future applications keep their priorities straight.

Users of enterprise software, for instance, have long complained about the poor user experience, the inflexibility, and the lack of usability of existing tools. A survey by Forrester found that:

• 75% of users couldn’t easily access information from existing enterprise systems.

• 69% of enterprise employees want an engaging mobile first experience but only 55% enterprises have implemented three or less mobile apps.

• Because of poor design 62% employees delay tasks which need them to log into multiple systems, affecting overall efficiency and outcomes.

The usability issues with existing enterprise tools have contributed to the shadow IT phenomenon where enterprise users are increasingly using user friendly third party tools like Dropbox or Slack instead of sticking to officially approved software, sometimes with serious security and data governance repercussions.

However, user-centred design shouldn't be viewed as a "nice-to-have". Software design and the product itself are increasingly inseparable. A confusingly designed internet banking application will make customers jump ship even if the bank offers attractive interest rates or better perks compared to the competition.

Poorly designed software has real world implications beyond a user spending twice as much time trying to understand how a system works.

For instance, last year alone, Tricentis reported that software failures within the transport industry resulted in the recall of 21,228,066 cars, grounding of 8,831 planes, and affected 22,712,987 people.

The government sector has seen the maximum number of fails (for example: the Australian Census blunder or the US Healthcare.gov debacle), with far reaching impact for millions of people who depend on public sector services. The global cost of government software failure has been estimated to be $5,703,579,938 in 2016.

On the whole, the cost of software failure has risen from 2015 to 2016 in terms of people affected (up 2.3% to 4.4 billion), assets impacted (up 260% to 1.1 trillion), and companies afflicted (up by 52% to 363).

The seeds of software failure are sown early in a project, when business requirements are not managed properly (CIO magazine found the numbers of failed projects to be as high as 71%) or when the end user doesn’t have a say in the design and execution.

Prioritising business requirements

So if you want to set up your project for success you will have to focus on getting your requirements right.  In the context of UAT the sponsor is in charge of setting the business requirements which will then be made into test cases.

The usability tests will have both functional as well as non-functional (stress, reliability, performance, speed etc.) requirements to be tested.

One way to  prioritise business requirements and user stories is to use the MoSCoW method, which Wikipedia defines as:

“A prioritization technique used in management, business analysis, project management, and software development to reach a common understanding with stakeholders on the importance they place on the delivery of each requirement.”

The MoSCoW acronym breaks down as:

Mo: Must have this test done.

S: Should run this test, if possible.

Co: Could run this test if other issues are fixed.

W: Would run this test if possible in the future.

This arrangement makes it easy for sponsors to eliminate any kind of confusion while drawing up business requirements.

This prioritisation ensures that the most important tests are conducted first, and more importantly, tests which don’t really matter in the larger scheme of things are deferred for a later date. Given the expense and time requirement of the UAT process, this formula guarantees the highest impact and keeps UAT cycles short and manageable.

UAT acceptance criteria

The UAT acceptance criteria (UAC) is a series of simple statements that distill the business requirements and give stakeholders an idea of the time and costs involved in the entire project.

When you get your UAC right you will be laser focused on your testing processes and not embark on a wild goose chase. If you were to use a decision tree, this is what it would look like:

Acceptance Criteria

1. Given

   1. Input

   2. Preconditions

2. When

   1. Triggers

   2. Actions

3. Then

   1. Output

   2. Consequences

The scope of UAT

Because UAT deals with user experience, it should ideally cover:

• Operational Requirements: Are the requirements around data capture, data processing, data distribution, and data archiving met?

• Functional Requirements: Are all business functions met as per expectations?

• Interface Requirements: Is data passing through the system as per business requirements?

UAT isn’t about testing whether the radio buttons on a particular form function properly. These tests fall in the entry criteria of UAT, which also include:

• Completion of unit testing, integration testing and systems testing

• Absence of dealbreakers, high or medium level defects during integration testing

• Fixing of all major errors, except for cosmetic errors

• Defect free completion of regression testing

• Complete Requirements Traceability Matrix (RTM)

• Communication from systems testing team certifying that the system is ready for UAT.

Diligently following this process will provide much greater clarity and certainty around the purpose of UAT within your organisation. This will not only help fast-track and streamline the process as you move forward, but also help win support from stakeholders in other areas of the organisation. 

What Next?

If you are new to Bugwolf and would like to learn more about how we help with user acceptance testing, the quickest and easiest way to find out more is to Request A Demo by clicking HERE.

Only a very small percentage of organisations are running continuous deployment models including MYOB, REA, and Envato. This approach allows companies to quickly iterate development rather than shipping large releases. However, due to internal security, risk and compliance in large organisations, continuous deployment may never be a reality.  

To reduce these cycles from weeks to days, Bugwolf has created a pay for performance model for testing. We do this by gamifying the test process and assembling elite teams of digital professionals who compete to discover usability, user experience and functional defects in software, over significantly shorter timeframes, all for less cost.

Ultimately this means your products and innovation can be released to market faster than your competitors, allowing you to spend more time and money on producing quality (and effective) products.

Engaging content is more than just telling an interesting story or commenting on the latest news, it must add value to the customers life in order to generate anything but a passing interest.  This means that your content must be customer centered.  It must reflect the customer's needs, wants and even emotions.  Ideally, it should supply the potential customer with the information that he or she can use.  It's also important to remember that it is the customer and not you who defines usefulness.  So, the first question that must be asked is what do you know about the people who have already purchased your product or services.  What was important to them?  What motivated them to buy?  In other words what problems did they come to you with that motivated them to buy from you?  Your present time customers are a tremendous resource.  The first step in drawing in future customers with content is to understand why your customers bought from you.

People often try to create engaging content by presenting what they think is important.  In reality, it's what the customer thinks that's important.  You can often get good information about importances from repeat customers because you must be doing something right that keeps them coming back.  It is highly unlikely that your repeat customers have problems that are unique only to them.  Which means that you can learn a lot about the needs and wants of potential customers by understanding your own satisfied customers.  The basic concept of good content is that it solves problems or, at the very least, helps people understand their problems.  It's up to the CMO and his or her team to understand the customer and what those problems are.

If you can address the customer's needs and wants then you have an excellent chance of engaging that customer in a positive way.  It's not necessary and indeed it's impossible to solve all problems with content.  But, it is possible to lighten the customer’s load a bit and then direct the customer to that product or service that can actually solve the problem.  Even if the customer walks away and doesn't  purchase, you have still communicated an important message. That message is "we've helped you and you can come back when you realize what you're up against”. It sometimes takes people a while to determine what they really need and as long as you have supplied even a partial solution, they will remember you.

There are a number of ways to gain insight into the needs of potential customers. These include but are certainly not limited to online or e-mail surveys, visiting forums and blogs where your target audience hangs out, and incentive driven questionnaires presented to customers following purchase.  And it's a good idea to integrate information capture into all marketing and sales efforts.

Competition for attention can be considerable and this is why it is important to invest in great content. Interestingly enough, one of the best ways to provide good content is to stop selling. Keep your marketing message mild and avoid aggressive sales techniques. The modern consumer is a rather jaded person who has heard it all before and prefers a company that knows what it's doing to one with a hefty pitch. Show that you know what you are doing by providing content that not only presents solutions, but also guides the discussion with thought leadership. Show that you know and your product or service will practically sell itself.

One of the challenges about these periods is not the agreement itself but the time in which the warranty is made available. Typically it’s during a very busy time for the organisation, product owner, and team. The focus is around meeting delivery timeframes, launching and marketing, and the organisation has limited resources to test the product with real users themselves.

To maximise these warranty period, Bugwolf provides highly accelerated and on-demand testing cycles during this time. We provide a fresh set of eyes looking at your applications before launch, provide you greater visibility of the quality of your products, and rapidly expand test coverage.

And this means, the you get the very best investment from the agreement with your software developers, while delivering a premium quality experience to your customers.