Our team uses Avocado-VT to implement Virtualization system tests from Libvirt or QEMU level.
Avocado-VT is highly-configurable and as such we've seen many people struggling getting started with it despite the documentation.
The following slides provide a hands-on step-by-step setup, execution and development overview that might help get you started. It focuses on an audience for testing on mainframes but most parts are arch-independent. If there's anything missing or wrong, please let me know.
kill -15 1
Some of our automation jobs sometimes seems to break systemd on our hosts resulting in the following error when interacting with it (e.g. trying to reboot):
System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to connect to bus: Host is down
Failed to talk to init daemon.
Clearly this is a lie as ps confirms.
"Host is down" solutions recommend usually to issue some systemctl command but that just caused the same behavior.
However, you can issue those commands by sending signals as the manpage reveals. In our case sending SIGTERM to PID 1 solved the issue (though not sure what causes the issue in the first place :/).
SIGNALS
SIGTERM
Upon receiving this signal the systemd system manager serializes its state, reexecutes itself and deserializes
the saved state again. This is mostly equivalent to systemctl daemon-reexec.
With v2.22.0 the s390-tools (aka s390utils on RHEL) have an important addition that helps to pass through crypto domains to KVM guests: they allow for persistence and help the user to avoid invalid configurations. (Thanks to Matthew Rosato for explaining the details to me.)
As you might remember from my previous post, crypto device passthrough (with libvirt) consists of three main steps:
Mainframes allow for shared memory communications between LPARs on the same box through ISM - internal shared memory. More information about the Linux device driver can be found here.
IBM provides open-source tools including smc_run which easily converts an application's usage of TCP/IP sockets to SMC (shared memory connection) sockets.
Let's suppose you want to check two of your LPARs can communicate with each other.
First you'd want to check if ISM is available. As ISM are made available as virtual pci devices, you can simply run lspci on each LPAR.
# lspci 00:00.0 Non-VGA unclassified device: IBM Internal Shared Memory (ISM) virtual PCI deviceIf your admin tells you have been provided the ISM device but you don't see it you might have to power it on.
# echo 1 > /sys/bus/pci/slots/0000032/powerNow, smc_run will convert any TCP/IP socket usage to an SMC socket. So let's suppose you have an echo server using the AF_NET protocol that you can start via commandline; you'd simply run the same command.
[host1]# smc_run python3 echo_server.py --host my_host_name.example.org --port 12345You can then simply send data from a client in the same way.
[host2]# smc_run python3 send_data.py --host my_host_name.example.org --port 12345 some dataLooks like it's working right? But does it really?
SMC socket capabilities are negotiated at connection setup. If one peer is not SMC capable, further socket processing falls back to TCP usage automatically.
So, how can we make sure that our LPARs really communicate through the ISM?
The s390-tools luckily deliver another tool smcss. It shows details for AF_SMC socket connections. The Mode column shows how data is exchanged:
SMCD The SMC socket uses SMC-D for data exchange.
SMCR The SMC socket uses SMC-R for data exchange.
TCP The SMC socket uses the TCP protocol for data exchange, because an SMC connection could not be established.
And really, the difference can be confirmed while the connection is open depending on the availability of ISM on both LPARs.
[host1]# smcss State UID Inode Local Address Peer Address Intf Mode ACTIVE 00000 22045079 192.168.0.10:12223 192.168.0.12:37060 0000 SMCDvs.
[host1]# smcss State UID Inode Local Address Peer Address Intf Mode ACTIVE 00000 22049662 192.168.0.10:12223 192.168.0.12:37058 0000 TCP 0x05000000/0x03030000
Finally, the s390-tools since version 1.5 also offers another tool that helps to check the ISM live-connectivity without a TCP application, smc_chk. You can shortly run:
[host1]# smc_chk -S Server started on port 37374
[host2]# smc_chk -C 192.168.0.12 -p 37374
Test with target IP 192.168.0.12 and port 37374
Live test (SMC-D and SMC-R, EXPERIMENTAL)
Success, using SMC-DIn MSDOS 6.22 there are alignment restrictions for partitions. This means a partition of size capacity = /start - end/ = end - start, partition boundaries (start, end) must coincide with certain boundaries; in this case cylinder boundaries.
The following alignment algorithm is taken from the libvirt virtualization API.
In short, the algorithm will make sure allocated continuous space is aligned on the right, that is on the end. If the available free space for alignment already starts at a given boundary value, it will be fully aligned [1].
We'll have:
<hostdev mode='subsystem' type='mdev' model='vfio-ap'>
<source>
<address uuid='$uuid'/>
</source>
</hostdev>
Input ≃ { 0, {0} }.)
# virsh domifaddr vm
Name MAC address Protocol Address
-------------------------------------------------------------------------------
vnet0 52:54:00:6f:dc:90 ipv4 192.168.122.84/24
# virsh dumpxml --inactive vm
...
<interface type="network">
<mac address="52:54:00:6f:dc:90">
<source network="default"></source>
<model type="virtio">
<address bus="0x00" domain="0x0000" function="0x0" slot="0x03" type="pci">
</address>
...
# /etc/nsswitch.conf: hosts: files libvirt_guest dnsand make sure to have the correct sshd configuration in your guest.
// mock-data.js
var usersGet = {
pattern: /users$/,
ok: {
status: 200,
json: [
{
id: 1,
username: 'user1',
email: 'user1@email.com'
},
{
id: 2,
username: 'user2',
email: 'user2@email.com'
}
,
{
id: 3,
username: 'user3',
email: 'user3@email.com'
}
]
},
ko: {
status: 500,
json: {errorCode: "UnexpectedException"}
}
};//default-setup.js
var mockyeah = require('mockyeah');
var mockdata = require('./mock_data');
var init_mock = function () {
mockyeah.get(mockdata.usersGet.pattern, mockdata.usersGet.ok);
mockyeah.get(mockdata.userGet.pattern, mockdata.userGet.ok);
mockyeah.post(mockdata.userPost.pattern, mockdata.userPost.ok);
};
init_mock();
exports.init_mock = init_mock;
exports.mockyeah = mockyeah;
exports.mockdata = mockdata;node default-setup.js and have a backend available during development.mockyeah is automatically started as configured in .mockyeah when required, your test suite must simply import your default setup as defined above.var setup = require('./../default-setup');
describe('Handle users', function () {
describe('Users view', function () {
beforeAll(setup.init_mock);
afterAll(setup.mockyeah.close);
describe('Handle users', function () {
describe('Users view', function () {
// ...
it('should load the users list', function () {
return usersPage.navigateToUsersView().then(function () {
expect(usersPage.getUserList()).toContain({id: 1, username: 'user1'});
expect(usersPage.getMessage()).toEqual("SUCCESS");
});
});
it('should show the error code if list cannot be loaded', function () {
setup.mockyeah.get(setup.mockdata.usersGet.pattern, setup.mockdata.usersGet.ko);
return usersPage.navigateToUsersView().then(function () {
expect(usersPage.getMessage()).toEqual("UnexpectedException");
});
});
{
output: true,
journal: true
} ✓ should send the correct request
[mockyeah][12:31:46][SERVE][MOUNT][GET] /users$/
[mockyeah][12:31:46][REQUEST][JOURNAL] {
"callCount": 1,
"url": "/users",
"fullUrl": "http://localhost:4001/users",
"clientIp": "127.0.0.1",
"method": "GET",
"headers": {
"host": "localhost:4001",
"accept": "application/json",
"connection": "close"
},
"query": {},
"body": {}
}
[mockyeah][12:31:46][REQUEST][GET] /users (2ms)
it('should send the correct request', function() {
return usersPage.navigateToUsersView().then(function () {
usersPage.enterNewUserDetails("user1", "user1@email.com");
var expectation = setup.mockyeah.post(setup.mockdata.userPost.pattern,
setup.mockdata.userPost.ok)
.expect()
.body({
username: 'user1',
email: 'user1@email.com'
})
.once();
return usersPage.confirm().then(function () {
expectation.verify();
});
});
});
[21:59:59] I/local - Selenium standalone server started at http://192.168.0.155:48608/wd/hub
Spec started
[mockyeah][SERVE] Listening at http://127.0.0.1:4001
[mockyeah][REQUEST][GET] /users (2ms)
Handle users
Users view
✓ should load the users list
[mockyeah][REQUEST][GET] /users (1ms)
✓ should show the error code if list cannot be loaded
[mockyeah][REQUEST][GET] /users (1ms)
[mockyeah][REQUEST][POST] /users (1ms)
✓ should load user details when created successfully
Executed 3 of 3 specs SUCCESS in 0.062 sec.
[mockyeah][SERVE][EXIT] Goodbye.
[22:00:00] I/local - Shutting down selenium standalone server.{ ...
"output": false,
"verbose": false
}[21:59:59] I/local - Selenium standalone server started at http://192.168.0.155:48608/wd/hub
Spec started
Handle users
Users view
✓ should load the users list
✓ should show the error code if list cannot be loaded
✓ should load user details when created successfully
Executed 3 of 3 specs SUCCESS in 0.062 sec.
[22:00:00] I/local - Shutting down selenium standalone server.java -jar wiremock-1.58-standalone.jar --proxy-all="http://www.webservicex.net/" --record-mappings --verbose
curl --header "Content-Type: text/xml;charset=UTF-8" --header "SOAPAction:her.asmxww.webserviceX.NET/GetWeather" --data @request.xml http://localhost:8080/globalweather.asmx
{ "request" : { "url" : "/globalweather.asmx", "method" : "POST", "bodyPatterns" : [ { "contains" : "<soapenv:Envelope xmlns:soapenv=\"http://schemas.xmlsoap.org/soap/envelope/\" xmlns:web=\"http://www.webserviceX.NET\">\n <soapenv:Header/>\n <soapenv:Body>\n <web:GetWeather>\n <!--Optional:-->\n <web:CityName>Madrid</web:CityName>\n <!--Optional:-->\n <web:CountryName>Spain</web:CountryName>\n </web:GetWeather>\n </soapenv:Body>\n</soapenv:Envelope>" } ] }, "response" : { "status" : 200, "bodyFileName" : "body-globalweather.asmx-8VMpb.json", "headers" : { "Cache-Control" : "private, max-age=0", "Content-Type" : "text/xml; charset=utf-8", "Content-Encoding" : "gzip", "Vary" : "Accept-Encoding", "Server" : "Microsoft-IIS/7.0", "X-AspNet-Version" : "4.0.30319", "X-Powered-By" : "ASP.NET", "Date" : "Sun, 26 Jun 2016 12:49:57 GMT", "Content-Length" : "691" } } }Woohoo! your first configuration for the JSON API.
java -jar wiremock-1.58-standalone.jarand rerunning the above curl command.
{
"request" : {
"url" : "/globalweather.asmx",
"method" : "POST",
"bodyPatterns" : [ {
"contains" : "GetWeather"
} ]
},
"response" : {
"status" : 200,
"body" : ...,
"fixedDelayMilliseconds": 2000
}
}
to http://localhost:8080/__admin/mappings/new causing this way the mock to respond after 2 seconds.| Method | Description | Notes |
|---|---|---|
| __admin/reset | Removes all stub mappings and deletes the request log. | Great for clearing mock behaviour before setting up anything |
| __admin/mappings | Lists all configured mappings | Helps while developing your tests |
| __admin/mappings/save | Saves all mappings | This way you can create a static default setup easily, and refresh it if your mappings change. |
| __admin/mappings/new | Creates a new mapping | Create a new mapping in setup or during a test, switching for example from success to error. By default, WireMock will use the most recently added matching stub to satisfy the request. But several mappings can even be prioritised. |
| __admin/mappings/reset | Removes all non-static mappings | Like full reset, but preserving your default setup. |
| __admin/settings __admin/socket_delay |
Set the global settings for your stubbing | You can vary over delays for your performance testing. |
| __admin/scenarios/reset | Reset all states to START. | Again, great for resetting your test environment if you use stateful behaviour. |
| __admin/requests/count __admin/requests/find __admin/requests/reset |
Lets you manage and check the requests log | Check requests during test development and verify DoS have been called. find with body { "urlPattern" : "/" } let's you inspect all recorded requests. |
package fakeexe
//...
func EncodeArgument(exitCode int) string {
return strconv.Itoa(exitCode) + ext
}
type fakeExe struct {
ExitCode int
}
func (f *fakeExe) DecodeArgument(arg string) {
var err error
if len(arg) > len(ext) {
ecode := arg[:len(arg)-len(ext)]
f.ExitCode, err = strconv.Atoi(ecode)
} else {
err = errors.New("input not valid" + arg)
}
if err != nil {
f.handleError(err)
}
}
func (f *fakeExe) Run(arg string) {
f.DecodeArgument(arg)
}
func (f *fakeExe) handleError(error) {
//...
}
package main
import (
"fakeexe"
"os"
)
func main() {
f := new(fakeexe.fakeExe)
f.Run(os.Args[0])
os.Exit(f.ExitCode)
}
package configuring
import (
"fmt"
"io/ioutil"
"net"
)
var ConfigFilename = "Config.txt"
//...
type server struct {
getListener func(protocol, port string) (net.Listener, error)
ln net.Listener
protocol string
port string
}
func (srv *server) Start() {
var err error
srv.ln, err = srv.getListener(srv.protocol, ":"+srv.port)
if err != nil {
panic(err)
}
for {
conn, err := srv.ln.Accept()
if err != nil {
fmt.Println(err)
continue
}
go srv.sendConfig(conn)
}
return
}
func (srv *server) sendConfig(conn net.Conn) {
bytes, err := ioutil.ReadFile(ConfigFilename)
if err != nil {
panic(err)
}
if len(bytes) > MAX_MESSAGE_LENGTH {
panic("Config message too long.")
}
_, err = conn.Write(bytes)
if err != nil {
panic(err)
}
return
}
func (srv *server) Stop() {
if srv.ln != nil {
srv.ln.Close()
}
}
package logging
import (
"net"
"fmt"
)
//...
type server struct{
getListener func(protocol, port string) (net.Listener, error)
ln net.Listener
protocol string
port string
msgs []string
}
func (srv *server)Msgs() (msgs []string){
msgs = srv.msgs
return
}
func (srv *server)Start() (err error){
srv.ln, err = srv.getListener(srv.protocol, ":" + srv.port)
if(err != nil){
panic(err)
}
for {
conn, err := srv.ln.Accept()
if (err != nil){
fmt.Println(err)
continue
}
go srv.appendMessage(conn)
}
return
}
func (srv *server)appendMessage(conn net.Conn){
defer conn.Close()
buf := make([]byte, MAX_MESSAGE_LENGTH)
msg_length, err := conn.Read(buf)
var msg string
if(err != nil){
msg = err.Error()
}else{
msg = string(buf[:msg_length])
}
srv.msgs = append(srv.msgs, msg)
}
func (srv *server)Stop(){
if(srv.ln != nil){
srv.ln.Close()
}
}
package fakeexe
import (
"lisp" //https://github.com/bobappleyard/golisp
"io"
"strings"
)
//...
func (f *fakeExe)Run(script string){
i := lisp.New()
i.Repl(strings.NewReader(script), io.Stdout)
//...
}
public void Init()
{
var content = File.ReadAllText(this.Path);
...
}
public void Init()
{
this.Init(File.ReadAllText);
}
internal void Init(ReadAllText readAllText)
{
var content = readAllText(this.Path);
...
}
internal delegate string ReadAllText(string path); using System.IO;
using System.Xml;
...
internal void Init()
{
var file = new FileInfo(this.Path);
var dir = new DirectoryInfo(this.DirPath);
var xdoc = new XmlDocument();
...
}
internal void Init()
{
var file = new System.IO.FileInfo(this.Path);
var dir = new System.IO.DirectoryInfo(this.DirPath);
var xdoc = new System.Xml.XmlDocument();
...
}
public void TestCaseConstructorSetsProperties()
{
var to = new TestObject(anyParameter());
AssertThatPropertiesAreSet(to);
}
public void TestCaseConstructorSetsProperties()
{
var to = new TestObject(someParameter());
AssertThatPropertiesAreSet(to);
}
public void TestCaseUsingFileSystem()
{
var file = createTestFile();
var to = new TestObject();
to.doSomething(file);
AssertThatSomethingHoldsOn(to);
file.Delete();
}
public void TestCaseUsingFileSystem()
{
var file = createAndRegisterForCleanupTestFile();
var to = new TestObject();
to.doSomething(file);
AssertThatSomethingHoldsOn(to);
}
public void TearDown()
{
foreach(var item in this.trash)
{
try
{
var file = item as File;
if(file != null) file.Delete();
...
}catch(Exception e){
reportToTestRunner(e.Message);
}
}
}
private File createAndRegisterForCleanupTestFile()
{
var file = createTestFile();
this.trash.Add(file);
return file;
} public void TestCaseSomeMethodRaisesEvent()
{
var eventHasBeenRaised = false;
testObject.SomeEventHandler = (sender, args) => eventHasBeenRaised = true;
testObject.SomeMethod();
AssertThat(eventHasBeenRaised);
}