Tuesday, April 28, 2020

The testing effort growth

Just wondering how a proof for "Number of test cases grows exponentially" could look like...

Define a program to be a function from a set of input variables to a set of output variables,

P = Input x P|Output = I x O 
  = { (j_1,...,j_n) x (o_1,...,o_m) }
  = { (j_1,...,j_n,o_1,...,o_m) }.

Each component of Input is supposed to have at least 2 elements. (If not, the input variable will never change the image value, in other words, the program's behavior, and can therefore be eliminated. The case where an input value is determine to be defined or not, Input ≃ { 0, {0} }.)

Also, I need to restrict to the case where the dim(I) > 1 because if not each extra value to test adds exactly one test case (linear growth).

Define a feature to be a selection of a subset of input variables combined with their image, that is, it's a restriction of the program to a subset of the domain, F = P|J, J < I.

We take a test case to be the a random variable

T : Input x Ω -> Input x Output
T(i, ω) = (i, T_1(i, ω),..., T_m(i, ω)).

Each T_j is an expected output or simply expectation or post-condition.

A test passes if T_j(ω) = P(i)_j for all j, or fails otherwise, for a test execution ω.

A test case of a feature is then simply T|F = T|J x Ω where F = P|J.

For a given feature we can add new behavior minimally by
  1. extending the domain of an input variable J_i by an additional value j' that defines a new behavior of the program, that is for some i, J_i' = J_i + { j' } and Input = J_1 x ... x J_i' x ... x J_n
  2. extending the set of input variables by an additional dimension J_n+1
Both will result in an extended feature F'.

For 2. it is easy to see that

#T|F' = #{ (J_1,...,J_n) x J_n+1 x O } 
      = #J * #J_n+1 * # O 
      = #T|F * #J_n+1
      ≥ #T|F * 2.

As for 1., given that dim(J) > 1 each new value j' creates another full set of combinations of input variables (j_1,...,j',...,j_n), that is, the number of added test cases is

#T|F' - #T|F = #J_1,...,^J_i,...,J_n
                               ≥ 2^(n-1)

where ^J_i denotes not selecting this component.

Therefore,

#T|F'  #T|F + 2^(n-1).

So, in general adding a new value to test, the lower bound for growth is 2^(n-1).  ⃞

Considering that the effort E|F of testing a feature depends on the selected test suite S|F < T|F, we might dare say that selecting S|F and the methods to evaluate T(i, ω) is a very important activity in testing.

I hope this makes sense...

Thursday, March 26, 2020

ssh into libvirt guest

Libvirt sets up NAT mode per default allowing guests to communicate, for example, with the internet, with each other and with the host. A host interface vnetX will be created and normally an IP will be assigned automatically.


# virsh domifaddr vm
 Name       MAC address          Protocol     Address
-------------------------------------------------------------------------------
 vnet0      52:54:00:6f:dc:90    ipv4         192.168.122.84/24

# virsh dumpxml --inactive vm
...
    <interface type="network">
      <mac address="52:54:00:6f:dc:90">
      <source network="default"></source>
      <model type="virtio">
      <address bus="0x00" domain="0x0000" function="0x0" slot="0x03" type="pci">
    </address>
...

However, vnetX won't get the IP assigned really, that's not how it's intended. So, per default neither the ip nor any name are available to ssh

What you can do instead is use the NSS module libvirt-nss.

If you want to easily ssh into the guest using its libvirt name, set
# /etc/nsswitch.conf:
hosts:       files libvirt_guest dns
and make sure to have the correct sshd configuration in your guest.

Tuesday, February 21, 2017

Sunk Cost Fallacy - Let's talk about bias 3

We continue our review of  biases we all have intrinsecally implemented in our own brains that lead us to make certain decisions, as Rolf Dobelli describes in his bestseller The Art of Thinking Clearly.

Sunk Cost Fallacy - Are you Lean enough?

This really is a very useful bias. I believe if we're honest enough with ourselves it helps us to really do a better job saving unnecessary costs for our company.

Without much introduction, two cases that immediately come to my mind:

  1. You see this huge beautiful implementation arriving testing. Verification passes without any flaw. You then notice: validation fails! This is not really what we wanted. Do you dare to create a bug and vote for throwing away / reworking all that hard and good work? Do you try to bend in and persuade any opposing mind to believe that that's what they wanted? Sometimes this is not the worst choice - sometimes clients just don't know what they want XD
  2. You've convinced everybody you need those automated tests / manual test scripts / test management system - whatever - to improve your testing process and all over quality of your product(s). It becomes a nuissance in time, things change, tempora mutantur. Are you willing to let go, to not hold it back anymore? Throw away what lacks purpose and be really lean.

Wednesday, January 11, 2017

Social Proof - Let's talk about bias 2

We continue our review of  biases we all have intrinsecally implemented in our own brains that lead us to make certain decisions, as Rolf Dobelli describes in his bestseller The Art of Thinking Clearly.

Social Proof

SCRUM planning session. A tester (role) points out that a certain US should include a some test to minimize risk of failure (functional, non-functional, however). The team and the product owner discuss the risk and convince themselves that the test is not necessary. The tester role has given in and will rest in peace.

The social proof makes you believe a decision is right because everybody else seems to believe it's right.
As tester you then have to ask yourself critically: is it their arguments that convince me or is it them being so convinced that does? If it's their arguments you might rest in peace, really. If not, argument against it or forever remain silent.

Survivorship bias - Let's talk about bias 1

You'll probably remember that a tester's profile has one of its main reasons for existence in avoiding author bias.

As QA and testers dive into Agile environments it's hard not to become part of the problem.

Interesting enough, there are well known biases we all have intrinsecally implemented in our own brains that lead us to make certain decisions, as Rolf Dobelli describes in his bestseller The Art of Thinking Clearly.

This post starts a series of checks of those biases with QA specific examples, so any tester or QA role can improve their work. Besides Agile, I like the move forward Quality Assistance because I experience that it works. So anybody in the team should have some advantage by knowing these biases.

Survivorship Bias

You as QA are confronted with developers and business representatives telling you "product X didn't have all of those design patterns, automated tests, branching policies, etc."
They strike you hard. You feel like earth's gonna swallow you. You're in China and only speak Hopi. Shit!

This is not your bias! Let'em check out their bug reports and time (and money - YEEEES, business speaks MO-O-NEY!) spent on fixing them. Speak to them about your customers serving as testers. But we're not facebook, we sell other stuff under different conditions. And if you really can't find none of those arguments to have effect present them the worst case, find examples for companies who failed with much embarrassment or quit your job, right now there are great developers at work!


Saturday, November 19, 2016

Fast Frontend Development and HUI Testing with mockyeah

The backend mocking library mockyeah has recently been added new features inspired by WireMock. The library has the big advantage of allowing you to re-use mock data during development for testing.
By mocking the backend there's no immediate need to set up and maintain backend data thus speeding up development.
You can find working sample code here.

Fast frontend development - backend mocking

In order to be independent from a real backend instance during frontend development we'll organize our expected responses and add them to a default setup. We'll then configure our frontend to point to the mockyeah instance, start the mock server with the default setup and we're ready to develop our web application!

Organizing the mock data

Organizing our mock data we'll have later easy access in your test definitions:
// mock-data.js
var usersGet = {
    pattern: /users$/,
    ok: {
        status: 200,
        json: [
            {
                id: 1,
                username: 'user1',
                email: 'user1@email.com'
            },
            {
                id: 2,
                username: 'user2',
                email: 'user2@email.com'
            }
            ,
            {
                id: 3,
                username: 'user3',
                email: 'user3@email.com'
            }
        ]
    },
    ko: {
        status: 500,
        json: {errorCode: "UnexpectedException"}
    }
};

Collecting mock data

mockyeah has a record-and-play feature that allows it to run as a proxy and save data you want to mock. Have a look at the library's documentation to find out more.

Define a default setup

//default-setup.js

var mockyeah = require('mockyeah');
var mockdata = require('./mock_data');

var init_mock = function () {
    mockyeah.get(mockdata.usersGet.pattern, mockdata.usersGet.ok);
    mockyeah.get(mockdata.userGet.pattern, mockdata.userGet.ok);
    mockyeah.post(mockdata.userPost.pattern, mockdata.userPost.ok);
};

init_mock();

exports.init_mock = init_mock;
exports.mockyeah = mockyeah;
exports.mockdata = mockdata;
Now we run node default-setup.js and have a backend available during development.

Hermetic User Interface (HUI) testing

Frontend by itself is a subsystem of your application. HUI tests strive to have more stable and maintainable tests by mocking out any dependency to a service. This means you would normally implement system level tests with Selenium WebDriver or Protractor.

Starting mockyeah for testing

As mockyeah is automatically started as configured in .mockyeah when required, your test suite must simply import your default setup as defined above.
var setup = require('./../default-setup');

describe('Handle users', function () {
    describe('Users view', function () {
        beforeAll(setup.init_mock);
        afterAll(setup.mockyeah.close);

Changing mockyeah behaviour during test definition

Simply set the response in the test definition itself. Notice that we re-use the mockyeah exported by the default setup.
describe('Handle users', function () {
    describe('Users view', function () {
        // ...
        
        it('should load the users list', function () {
            return usersPage.navigateToUsersView().then(function () {
                expect(usersPage.getUserList()).toContain({id: 1, username: 'user1'});
                expect(usersPage.getMessage()).toEqual("SUCCESS");
            });
        });

        it('should show the error code if list cannot be loaded', function () {
            setup.mockyeah.get(setup.mockdata.usersGet.pattern, 
            setup.mockdata.usersGet.ko);
            return usersPage.navigateToUsersView().then(function () {
                expect(usersPage.getMessage()).toEqual("UnexpectedException");
            });
        });

Request verification and logging

During test development we'll want to inspect the received requests (the "request journal") when something doesn't come out as expected. We can do this by setting
{
    output: true,
    journal: true
}
giving the output
      ✓ should send the correct request
[mockyeah][12:31:46][SERVE][MOUNT][GET] /users$/
[mockyeah][12:31:46][REQUEST][JOURNAL] {
  "callCount": 1,
  "url": "/users",
  "fullUrl": "http://localhost:4001/users",
  "clientIp": "127.0.0.1",
  "method": "GET",
  "headers": {
    "host": "localhost:4001",
    "accept": "application/json",
    "connection": "close"
  },
  "query": {},
  "body": {}
}
[mockyeah][12:31:46][REQUEST][GET] /users (2ms)
If our application code composes a complex request from several sources sometime we find it useful to verify the sent request:
it('should send the correct request', function() {
    return usersPage.navigateToUsersView().then(function () {
        usersPage.enterNewUserDetails("user1", "user1@email.com");
        var expectation = setup.mockyeah.post(setup.mockdata.userPost.pattern, 
                                              setup.mockdata.userPost.ok)
            .expect()
            .body({
                username: 'user1',
                email: 'user1@email.com'
            })
            .once();
        return usersPage.confirm().then(function () {
            expectation.verify();
        });
    });
});
The standard configuration of mockyeah will write a standard request log which is very helpful during test development.
[21:59:59] I/local - Selenium standalone server started at http://192.168.0.155:48608/wd/hub
Spec started
[mockyeah][SERVE] Listening at http://127.0.0.1:4001
[mockyeah][REQUEST][GET] /users (2ms)

  Handle users

    Users view
      ✓ should load the users list
[mockyeah][REQUEST][GET] /users (1ms)
      ✓ should show the error code if list cannot be loaded
[mockyeah][REQUEST][GET] /users (1ms)
[mockyeah][REQUEST][POST] /users (1ms)
      ✓ should load user details when created successfully

Executed 3 of 3 specs SUCCESS in 0.062 sec.
[mockyeah][SERVE][EXIT] Goodbye.
[22:00:00] I/local - Shutting down selenium standalone server.
However, for test reporting you might prefer to switch logging of by setting
{ ...
  "output": false,
  "verbose": false
}
which will give the following less cluttered output.
[21:59:59] I/local - Selenium standalone server started at http://192.168.0.155:48608/wd/hub
Spec started

  Handle users

    Users view
      ✓ should load the users list
      ✓ should show the error code if list cannot be loaded
      ✓ should load user details when created successfully

Executed 3 of 3 specs SUCCESS in 0.062 sec.
[22:00:00] I/local - Shutting down selenium standalone server.

Sunday, June 26, 2016

WireMock for your Dependant-On Http Service

Recently I needed to mock an http service our appplication depends on. A developer recommended me WireMock he uses at unit level.

I target the system level. But - wow - WireMock running as standalone server gives me the same feature set thanks to it's great JSON API for configuration while running.

Here I give you an overview and some advice about how to get started mocking an HTTP service at system level.

Collection of the Mocked Data

First thing you need to know is:
  • the url you call
  • the data returned by those calls.
WireMock has a great record and playback feature. WireMock proxies any calls to the DOS and automatically creates files for the responses and mappings.

Example: GetWeather

We'll record responses for calling the global weather API of WebserviceX.NET.

Run
java -jar wiremock-1.58-standalone.jar --proxy-all="http://www.webservicex.net/" --record-mappings --verbose
and make a sample call to the GetWeather method, for example:
curl --header "Content-Type: text/xml;charset=UTF-8" --header "SOAPAction:her.asmxww.webserviceX.NET/GetWeather" --data @request.xml http://localhost:8080/globalweather.asmx
(request.xml containing a valid request, obviously).

This will create a file containing the response in __files and a sample mapping in mappings in your current working directory.

If you open the mapping just created you'll see something like:

{
  "request" : {
    "url" : "/globalweather.asmx",
    "method" : "POST",
    "bodyPatterns" : [ {
      "contains" : "<soapenv:Envelope xmlns:soapenv=\"http://schemas.xmlsoap.org/soap/envelope/\" xmlns:web=\"http://www.webserviceX.NET\">\n   <soapenv:Header/>\n   <soapenv:Body>\n      <web:GetWeather>\n         <!--Optional:-->\n         <web:CityName>Madrid</web:CityName>\n         <!--Optional:-->\n         <web:CountryName>Spain</web:CountryName>\n      </web:GetWeather>\n   </soapenv:Body>\n</soapenv:Envelope>"
    } ]
  },
  "response" : {
    "status" : 200,
    "bodyFileName" : "body-globalweather.asmx-8VMpb.json",
    "headers" : {
      "Cache-Control" : "private, max-age=0",
      "Content-Type" : "text/xml; charset=utf-8",
      "Content-Encoding" : "gzip",
      "Vary" : "Accept-Encoding",
      "Server" : "Microsoft-IIS/7.0",
      "X-AspNet-Version" : "4.0.30319",
      "X-Powered-By" : "ASP.NET",
      "Date" : "Sun, 26 Jun 2016 12:49:57 GMT",
      "Content-Length" : "691"
    }
  }
}
Woohoo! your first configuration for the JSON API.

You can easily check your configuration / mock data, by restarting WireMock, but this time in standalone mode

java -jar wiremock-1.58-standalone.jar

and rerunning the above curl command.

Configuration of the Mock via JSON API

While you can have a static mock configuration in the mappings and __files folders, the really cool stuff is changing the configuration during runtime / testing on the fly calling the JSON API. This way we can even control delay for performance testing.

Let's suppose you're running WireMock on it's standard port at your localhost:8080 without any previous static configuration. You then create a new configuration by posting
{
  "request" : {
    "url" : "/globalweather.asmx",
    "method" : "POST",
    "bodyPatterns" : [ {
      "contains" : "GetWeather"
    } ]
  },
  "response" : {
    "status" : 200,
    "body" : ...,
    "fixedDelayMilliseconds": 2000
  }
}
to http://localhost:8080/__admin/mappings/new causing this way the mock to respond after 2 seconds.

WireMock has many well documented features any tester could whish for.

JSON API Cheat Sheet

Here a listing for easy reference during test development.

MethodDescriptionNotes
__admin/reset Removes all stub mappings and deletes the request log. Great for clearing mock behaviour before setting up anything
__admin/mappings Lists all configured mappings Helps while developing your tests
__admin/mappings/save Saves all mappings This way you can create a static default setup easily, and refresh it if your mappings change.
__admin/mappings/new Creates a new mapping Create a new mapping in setup or during a test, switching for example from success to error. By default, WireMock will use the most recently added matching stub to satisfy the request. But several mappings can even be prioritised.
__admin/mappings/reset Removes all non-static mappingsLike full reset, but preserving your default setup.
__admin/settings
__admin/socket_delay
Set the global settings for your stubbing You can vary over delays for your performance testing.
__admin/scenarios/reset Reset all states to START. Again, great for resetting your test environment if you use stateful behaviour.
__admin/requests/count
__admin/requests/find
__admin/requests/reset
Lets you manage and check the requests log Check requests during test development and verify DoS have been called.
find with body { "urlPattern" : "/" } let's you inspect all recorded requests.
You can find the full API reference here.