Verification & Validation using Ingescape

Verification & Validation using Ingescape

Ingescape brings a formal model to distributed software architecture. Together with this model comes a domain-specific language (DSL) dedicated to running Verification & Validation (V&V) tests on Ingescape platforms and generating reports based on these tests. This article shows the principles and examples of the Ingescape DSL dedicated to V&V.

Before reading this article, one should already be familiar with the core concepts of Ingescape such as agents, peers, inputs/outputs/parameters (IOP) and their types, logging in Ingescape, and channels.

General principles

The DSL is structured according to various families of commands:

  • Instructions
    • Allocations
    • Service call
    • Shout & whisper
    • Agent start & stop
    • Quitting an agent (a peer)
    • User actions
    • Sleep
    • System commands
    • HTTP REST queries
    • Mouse & keyboard emulation
  • Verifications
    • Output assertions
    • Silence assertions
    • Service assertions
    • Channel assertions
    • Channel silence assertions
    • User-based assertions
    • Agent enter/exit assertions
    • Log verifications
    • Compare
  • Blocks
  • Report entries
    • Manual report entries
    • Report entries per instruction
    • Report entries per verification
    • Report entries for blocks
    • Verification timeouts inside blocks
  • Commenting the scripts

A typical test script is a succession of instructions and verifications. Instructions enable to inject information and events in a given platform. And verifications enable to check that the behaviors of the platform and its agents regarding the injected information and events is correct. To create proper test scripts, one shall be familiar with the agents of the tested platform. For example, it is important to know which inputs will trigger which outputs, so that the script logic reflects the specified behaviors.

Most errors when using Ingescape V&V come from a lack of knowledge of the underlying platform, which induce erroneous sets of instructions and verifications. To avoid this kind of problems, try to start with minimal scripts or subparts of larger scripts and always test your scripts in a controlled environment. We recommend that the test scripts focusing on a given agent are written during the development of the agents by the developers themselves. Then, the integration team can rely on the unit tests to build larger tests involving many agents at once.

Parallel events in a distributed environment

Ingescape supports the execution of distributed decentralized environments. As such, on an Ingescape platform, events happen in parallel, sometimes in an unpredictable order. Ingescape V&V handles most of the resulting complexities for the user. However, the user must keep these characteristics in mind when having to design complex verifications. From the point of view of the script itself, here are the events happening in parallel :

  • Receiving information from the agents of the platform (outputs and channels),
  • Checking log flows from the agents,
  • Running system commands,
  • Executing a blocking sleep command or a user action,
  • Managing instructions and verifications timeouts.

This document provides extended information about when and how to deal with complex verification patterns.

Properly alternating instructions and verifications

Except for sleep and user actions, instructions are non-blocking. They are executed as fast as possible so that the script continues to the next instruction. At the opposite, verifications are blocking with possibilities to set timeouts. Ingescape V&V is designed to activate one or more verifications just before the instruction that precedes them is executed. This way, verifications are already active when the preceding instruction produces its effects. When an instruction is followed by a verification, this verification and all the following ones until the next instruction are put in a stack and are ready to be activated when their turn comes. By default, the stack is emptied with a first in, first out logic. But some tests require complex series of instructions and validations. Blocks are designed to help building theses complex structures by providing additional elements for the control of the execution, including parallelism of the various verification types, indefinite order of the events to be verified, multiple verifications at the same time or tolerance regarding unexpected events until the expected event occurs. Please read further to go into details.

This principle is very important to keep in mind: if one erroneously designs successions of instructions and verifications, a given verification might be activated too late compared to the expected effects of a corresponding instruction. In the same way, a series of instructions shall always be written carefully to avoid a following instruction to block the proper activation of a verification. Said otherwise, it is not possible to design scripts so that all the instructions come first and then come the verifications. All successions of instructions and verifications are to be designed ones corresponding to the others.

Instructions

This section presents the various types of instructions and their extended options and behavior.

Allocations

Allocations are the most obvious instructions on an Ingescape platform. They enable to inject information on an input of an agent, depending on the type of this input.

Here is a series of allocations, one for each type of IOP. The agent is called ‘myAgent’ and each input has a name indicating its type.

myAgent.myImpulsion = 0 //by convention for impulsion IOPs, we affect 0 but any value is valid
myAgent.myInt = 345
myAgent.myBool = 1 //0 means false and 1 means true
myAgent.myDouble = 23.56
myAgent.myString = "my string to be injected"
myAgent.myData = "0xaa22eeff" //hexadecimal binary dataCode language: JavaScript (javascript)

Service call

Service calls enable to call services on agents. Their syntax is very close to calling a function on an agent with the parameters corresponding to the ones of the service. Arguments are passed without quotes if they are numeric, or with quotes if they carry strings or data.

The following example calls the myService service on myAgent with the expected arguments.

myAgent.myService(1, 23, 45.213, "this is a string", "aa32eed4")Code language: JavaScript (javascript)

NB: calling services on an agent requires that this agent is known to us. This requires either passing a platform to the script engine or waiting for the agent to have connected to us with the “knows us” event. If one wants to call services on an agent, it is wise to wait a reasonable amount of time, i.e. 500 to 1000ms, by example by using the sleep instruction at the beginning of the script.

NB: if the passed arguments do not fit the ones expected by the service, the service is not called.

Shout & whisper

These two instructions enable to use Ingescape channels to send information to a channel or to a given peer.

Shout

The following example sends the provided string of characters to the MY_CHANNEL channel.

shout MY_CHANNEL "shout message with several words"Code language: JavaScript (javascript)

NB: at the moment, sending binary data is not supported.

Whisper

The following example sends the provided string of characters to the ‘my_peer’ peer.

whisper my_peer "shout message with several words"Code language: JavaScript (javascript)

NB: at the moment, sending binary data is not supported.

NB: remember that peers are an underlying construct of the agents. By default, an agent gives its name to the peer when you set it using the igs_set_agent_name function. But when using several agents in the same process, the peer name is the one of the so-called main agent, i.e. the one named using the igs_set_agent_name function.

NB: in addition to the peer name, the dynamically allocated peer id can also be used but is poorly relevant in predefined test scripts.

Agent start & stop

This instruction is used to start or stop agent. Be aware that starting or stopping and agent is different from stopping the underlying peer and its process on the hosting computer. These instructions shall be reserved to advanced scenarios when one desire to start or stop agents in advanced Ingescape architectures.

In the following example, the first instruction starts the agent ‘my_agent’ on the peer ‘my_peer’. The second instruction stops all the existing instances of ‘my_agent’.

start my_peer my_agent
stop my_agent

NB: in addition to the peer name, the dynamically allocated peer id can also be used but is poorly relevant in predefined test scripts.

Quitting an agent (a peer)

This instruction is equivalent to stopping an agent from Ingescape Circle. It stops not only the agent but its underlying peer and my result in stopping the peer process itself, depending on how the developer has decided to handle Ingescape STOP event.

The following instruction stops the agent ‘my_agent’ and its underlying peer:

quit my_agent

NB: in addition to the peer name, the dynamically-allocated peer id can also be used but is poorly relevant in predefined test scripts.

User actions

User actions are the way to interactively interrogate a human user through the command line interface. It supposes that the script is executed in an interactive console and not in the background. If run in a non-interaction console, such instructions terminate immediately.

Please keep in mind that these instructions are blocking, i.e. the next instruction and the possible following verifications are executed/enabled only after the user has pressed enter.

There are three ways to interrogate the user, by order of complexity:

  • Just pressing enter,
  • Entering a character from a specified list and pressing enter,
  • Entering a free text and pressing enter.

The following instructions give one example for each type:

ask "question 1 without text"
ask "question 2 with a specific character from the following list" y n ?
ask "question 3 with free text" textCode language: JavaScript (javascript)

Sleep

This instruction forces the script to wait for a certain amount of milliseconds. Please keep in mind that these instructions are blocking, i.e. the next instruction and the possible following verifications are executed/enabled only after the wait is over. Because this instruction is blocking, it is useless to introduce it to give more delay to a previous instruction to produce its effects. The sleep instruction is reserved to giving more time to external behaviors that are generally independent from the test script itself.

The following instruction makes the script block and wait for one second:

sleep 1000

System commands

System commands enable to run commands on the computer on which the script is running. System commands can be anything runnable in a terminal and depend on the rights of the user running it on the hosting computer.

System commands can be attached a delay after which the command stops automatically. The delay is also mandatory to allow an embedding block to stop the system command on the block timeout. In any case, system commands are blocking, just like sleep and user actions.

System command may be run in background. In this case, the System command terminates immediately and the run command is detached from the V&V script. Command run in background should terminate on their own. If not, many processes may keep running on the system.

The following command runs the provided command line and stops either when the executed command ends or when the timeout of 10 000 milliseconds is reached. If the timeout is reached, the process attached to the command is killed by the script.

system 10000 "/usr/local/bin/igsTime --device en0 --port 5669 --verbose"Code language: JavaScript (javascript)

The following command runs the provided command line and stops only when the executed command terminates.

system "/usr/local/bin/igsTime --device en0 --port 5669 --verbose"Code language: JavaScript (javascript)

The following command runs the provided command line in the background, i.e. immediately detached from the script, and the script moves to the next command.

system bg "/usr/local/bin/igsTime --device en0 --port 5669 --verbose"Code language: JavaScript (javascript)

HTTP REST queries

HTTP REST queries are a way to send HTTP REST commands to a web server. Supported queries are of type GET, POST, PUT and DELETE. Only PUT and POST types require a payload. Payloads may only be provided as text.

The following command runs a GET query to the designated server.

http get "https://httpbin.org/anything"Code language: JavaScript (javascript)

The following command runs a POST query to the designated server including the payload between quotes.

http post "https://httpbin.org/post" "this a test \"payload\""Code language: JavaScript (javascript)

NB: PUT and DELETE queries use the ‘put’ and ‘delete’ keywords.

Mouse commands

Mouse commands enable to trigger mouse press, move and release commands on the machine hosting the script. Mouse commands use absolute screen coordinates. When multiple screens are in use, absolute coordinates also apply and require knowing the global coordinates between all the available displays.

The following command executes a mouse press event at the x = 22 and y = 34 coordinates on the screen.

mouse press 22 34

The following command executes a mouse move event at the x = 400 and y = 500 coordinates on the screen.

mouse move 400 500

The following command executes a mouse release event at the x = 22 and y = 34 coordinates on the screen.

mouse release 22 34

Various mouse events can follow one another. For example, a drag event consists in the combination of press, move and release events from the drag origin to the drag destination.

NB: Depending on the operating systems, a release event without a previous press event or other normally impossible combinations may cause problems.

Keyboard commands

Keyboard commands enable to trigger keyboard press, release and press & release events. Special keys are supported with their usual name: RETURN, TAB, SPACE, DELETE, ENTER, ESCAPE, COMMAND, SHIFT, RIGHTSHIFT, LEFTSHIFT, CAPLOCK, OPTION, ALT, CONTROL, VOLUMEUP, VOLUMEDOWN, MUTE, F1 to F20, LEFT, UP, DOWN, RIGHT, etc. [to be completed and refined for each operating system].

The following command executes a keyboard press event on the a key.

keyboard press a

The following command executes a keyboard press event on the B (upper case) key.

keyboard release B

The following command executes a keyboard press and release event on the F1 key.

keyboard press_release F1

NB: Depending on the operating systems, a release event without a previous press event or other normally impossible combinations may cause problems.

Verifications

This section presents the various types of verifications and their extended options and behavior.

One may notice that the first verifications are named assertions and the log verifications keep the generic term. This is to emphasize on the different ways the logs are treated compared to assertions. By default, i.e. outside a block explicitly changing the local behavior, assertions wait for the first valid information emitted by an agent, check it, generate an OK or NOK entry in the report and then are deactivated for the benefit of the next instruction or verification. At the opposite, log verifications operate during a specified amount of time and terminate either with an OK report entry when a log matches the verification before the timeout, or with a NOK report entry when the timeout is reached. The log verifications behave differently by default in order to deal with the large amount of log information, and the fact that logs are hard to control strictly in software projects.

Output assertions

Output assertions enable the verification of published values on agent outputs. They are at the core of the Ingescape input/output logic and are the most used of all assertions. Output assertions work for any type of output except for data. They offer the following comparators:

  • Equality : ==
  • Superior and superior or equal : >, >=
  • Inferior and inferior or equal : <, <=

A sixth comparator is reserved to strings, which also support the five first comparators based on the return value of the standard strcmp function. This sixth comparator is named ‘~’ and enables to match the published string against a regular expression based on the GNU C library.

Here are examples of output assertions:

assert myAgent.myDoubleOutput >= 10.234
assert myAgent.myIntOutput == 42
assert myAgent.myStringOutput == "expected message with value = 45"
assert myAgent.myStringOutput ~ "expected [^ ]+ with value = (\d+)"Code language: JavaScript (javascript)

Silence assertions

Silence assertions are the opposite of the output assertions. They verify that an agent does not publish any output for a certain amount of time.

The following commands runs a silence assertion for five seconds on myAgent.

assert silence myAgent 5000 //all the outputs of myAgent
assert silence myAgent.myOutput 5000 //silence on myOutputCode language: JavaScript (javascript)

Service assertions

Service assertions enable to verify the reply to a service called on another agent. Optionally, they enable to check one argument or more towards an expected value.

Let’s suppose that we call a service on myAgent and that myAgent will call back the getAnswer service on us, with five arguments named arg1 to arg5.

The following examples show how to write the various possible service assertions.

assert getAnswer from myAgent //simple verification that we received the call
assert getAnswer from myAgent with arg1 = 1 //checking both the service call and the value of the 1st argument
assert getAnswer from myAgent with arg2 = 45 and arg4 = "fourth argument value is a string" //checking service, arg2 and arg4
assert getAnswer from myAgent with arg1 = 1 and arg2 = 45 and arg3 = 32.134 and arg4 = "string" and arg5 = "aa23e4f4" //checking everythingCode language: JavaScript (javascript)

In the case where an argument is a string, it is possible to use regular expressions in addition to a fully defined text. To setup a regular expression, just add “~ ” – without forgetting the space after the tilde – at the beginning of the regular expression you want to check.

assert getAnswer from myAgent with arg4 = "~ str(.*)" //checking the service with arg4 starting with 'str'Code language: JavaScript (javascript)

Channel assertions

Channel assertions enable to verify data sent on a given channel. Please be aware that only string messages are supported at the moment. Channel assertions can be strict (==) or can match a regular expression (~).

The following examples show how to write the various possible channel assertions.

assert channel MY_CHANNEL == "this is my expected message" //full message
assert channel MY_CHANNEL ~ "this is my (.*)" //message starting with "this is my "Code language: JavaScript (javascript)

Channel silence assertions

Channel silence assertions enable to verify that a given channel remains silent for a certain amount of time.

The following example shows the syntax for channel silence assertions.

assert channel silence MY_CHANNEL 5000 //duration expressed in millisecondsCode language: JavaScript (javascript)

User assertions

User assertions enable the user to manually answer ‘y’ (i.e. OK) or ‘n’ (i.e. NOK) to a verification through a textual question.

The following example shows the syntax for user assertions.

assert user "are you ready ?"Code language: JavaScript (javascript)

Agent entered assertions

“Agent entered” assertions enable to verify that a given agent has entered the network. If the agent is already present, this assertion will not verify. It is designed to detect actual entering of the agent.

The following example shows the syntax for “agent entered” assertions.

assert agent entered myAgent

Agent exited assertions

“Agent exited” assertions enable to verify that a given agent has exited the network. If the agent is already gone, this assertion will not verify. It is designed to detect actual exiting of the agent.

The following example shows the syntax for “agent exited” assertions.

assert agent exited myAgent

Log verifications

Log verifications target a specific agent and a specific log level for a specified duration. Log verifications can be strict (==) or can match a regular expression (~).

The following command expects a strict log from myAgent at the DEBUG level and waits for it during 15 seconds.

expect log 15000 myAgent DEBUG == "model_write_iop;set input myDouble to double 13.890000"Code language: JavaScript (javascript)

The following command expects a regular expression from myAgent at the DEBUG level and waits for it during 5 seconds.

expect log 5000 myAgent DEBUG ~ "model_write_iop;set output value1 to (.*)"Code language: JavaScript (javascript)

Compare

The compare verification is in fact an instruction which creates OK or NOK entries in the test report, depending on the results of the comparison. At the moment, comparisons are for images only. Being treated as instructions, compare verifications are not affected by specific block behaviors.

The compare verification requires two arguments:

  • The file to be compared, e.g. “file1.png”
  • The reference to be compared to, e.g. “reference.png”

The compare verification proposes one optional argument to analyze only a subpart of the image:

  • The geometry of the subpart of the image in the format width*height+x_offset+y_offset, e.g. 2400*1200+200+100 with a width of 2400 pixels, a height of 1200 pixels, and from the top-left corner with the y axis going from up to down, an x offset of 200 pixels and a y offset of 100 pixels.

The order between the two first arguments is important because, in the case when images differ, the first argument is used to generate two additional files exhibiting the differences:

  • file1.png-diff.gif : based on the first file location and name, a new image with the GIF extension is created, providing an animation of the two images to help detect the differences visually,
  • file1.png-diff.png : based on the first file location and name, another new image with the PNG extension is created, showing a superposition of the two images and highlighting the differences.

NB: Ingescape V&V must have writing rights to the directory where the file to be compared is stored, in order to properly create the two additional images, should differences be detected.

The following command triggers a comparison between two PNG files.

compare "/Users/steph/Desktop/original.png" "/Users/steph/Desktop/reference.png"
compare "/Users/steph/Desktop/original.png" "/Users/steph/Desktop/reference.png" 2400*1200+200+200Code language: JavaScript (javascript)

NB: if the comparison fails in the above example, the two generated images will be:

  • /Users/steph/Desktop/original.png-diff.gif
  • /Users/steph/Desktop/original.png-diff.png

Understanding the concurrency between the different types of verifications

Ingescape V&V operates in a distributed parallel environment. Moreover, most verifications are network-based without any guarantee about the order of reception of the messages from various agents. The only guarantee is the order per agent.

By design, Ingescape V&V manages the following separated threads :

  • A thread dedicated to receiving data from ingescape : outputs, agent events, service calls,
  • A thread dedicated to receiving log streams from the relevant agents,
  • A possible thread dedicated to running the current system command based on a system instruction.

This means that, if one is not careful, depending on the order of the verifications involving outputs and logs, a single event from an agent may validate one or more verifications.

Consider the following example:

assert myAgent.value1 >= 10
expect log 500 myAgent DEBUG ~ "model_write_iop;set output value1 to (.*)"Code language: JavaScript (javascript)

When myAgent publishes a value on its value1 output, one or the two verifications above could be validated, depending on the reception order between the output and the log message:

  • If the output is received before the log, the two verifications are validated one after the other, based on the same initial event in myAgent.
  • If the log arrives before the output, only the output assertion will be validated because the log arrived first in the script.

Most the time, logs and outputs do not need to be mixed and should not be.

Blocks

Block are used to add a structure into complex test scripts and separate series of tests into independent pieces. Blocks are created with two optional informations:

  • A title,
  • A description.

The following example shows three empty block, all with a valid syntax.

{ //first block all empty

}

"my title" { //second block, with a title only

}

"my title" "my description" { //third block, with a title and a description

}Code language: JSON / JSON with Comments (json)

Block may contain instructions and verifications in any order. By default, inside a block, the logic of succession between instructions and verifications is the same as outside a block. Please read the General principles again if needed.

In addition, blocks offer ways to change the way instructions and verification to give more flexibility to the script writer. Four options are provided:

  • use_pool
    • By default, the first active verification in the current verifications stack is blocking the next ones. If the pooling is activated and the first verification in the stack cannot be checked immediately, because the received event (output publication, log entry, etc.) does not match its type, Ingescape V&V tries to find in the stack a verification that has the correct type and activates it. If a verification with the proper type is found, it executes and the event is discarded.
    • This option is useful to introduce flexibility in the order of the verifications, either because the order cannot be guaranteed or because the order is not important.
  • multi_check
    • This option is to be used with use_pool. When combined with use_pool, the multi_check option enables to activate as many verifications as possible. When an event is received and this option is active in combination with use_pool, all verifications in the current stack are scanned and executed if they fit the type of the received event.
    • This option is useful to run multiple verifications at once based on a given event. For example, if one wants to verify that a published output is included between a minimum and a maximum value, one simply needs to enable use_pool and multi_check, and to add the min and max verifications in the stack, without any specific order.
    • If use_pool is not enabled, multi_check has no effect on the block because the verifications stack will be run through one verification at a time.
  • timeout
    • When a timeout is set in a block, the block terminates automatically when the timeout is expired and a verification is enabled. This verification and the remaining ones are automatically set to NOK, the remaining instructions are ignored and timed out if needed, and the block exits.
    • This option is useful when verifications may not be verifiable or when one wants to abort a block in some situations.
    • Be careful, sleep and user actions are not interruptible and will thus block independently from a block timeout. The block timeout will take effect before an interrupting command is executed or after an interrupting command is over.
  • wait_for_ok
    • By default, when a verification is executed, it results in either an OK or a NOK report entry. When the wait_for_ok option is enabled, a NOK verification will not be discarded and remains in the current verifications stack. At the next occurrence of a compatible event, the verification will be checked again. Note that such an option, without a proper timeout activated, avoids the termination of the block until all verifications in it are finally OK.
    • This option is useful to ensure that, among a series of events, at least one validate one or more verifications. This makes verifications more permissive and allow for invalid events to occur without triggering a NOK report entry.

These four options can be combined freely. Here is a summary exposing some relevant combinations.

use_poolThe first verification in the current stack with the proper type matching the current event (log, output, service, channel) is executed, even if it is not at the head of the current stack.
use_pool + multi_checkAll verifications in the current stack matching the event type are executed.
timeoutThe block terminates after the provided delay and discards all remaining instructions and verifications.
wait_for_okAn executed verification is discarded only if its status is OK. If not, it remains in the current stack.
wait_for_ok + timeoutAn executed verification is discarded only if its status is OK. If not, it remains in the current stack. After the timeout is reached, all remaining instructions and verifications are discarded in the current stack.
use_pool + multi_check + wait_for_okAll verifications in the current stack matching the event type are executed and are discarded only if they are OK.
use_pool + multi_check + wait_for_ok + timeoutAll verifications in the current stack matching the event type are executed and are discarded only if they are OK, until the timeout is reached, which discards all remaining instructions and verifications in the current stack.

These four options may be added into any block, using the following syntax.

"my block" {
block.use_pool = true
block.multi_check = true
block.timeout = 1500 //in milliseconds
block.wait_for_ok = true
}Code language: JavaScript (javascript)

Notes:

  • Options can be placed anywhere in a block but are generally placed at its beginning.
  • If an option is used multiple times in a block only the last occurrence is used.
  • It is not possible to change a block option during the execution of the script.

Report entries

Report entries are the entries generated by the instructions and the verifications into a test report. Every entry is timestamped to the microsecond (µs) and provides all the useful information about the results of the instructions, and more importantly of the verifications. The following section details the possible report entries per type of instruction and verification.

All entries are built according to the same format:

dd/mm/yyyy;hh:mm:ss.µµµµµµ; entry type               | entry details

Here is an example.

21/12/2021;08:39:34.949661; assert OK | macosAgent.value1 >= 10

Manual report entries

There is a last instruction provided by Ingescape V&V which enables to print information into the report. This instruction only requires one parameter: the text to be written in the report.

The following command writes a manual entry in the test report.

-> "entry to be added into the report"Code language: JavaScript (javascript)

This command generates the following entry in the report.

21/12/2021;08:39:31.133662; message | entry to be added into the report

Extending the reports and including metadata

Ingescape V&V keeps the provided information as minimal as possible. However, it is easy to include metadata inside the reports by using manuel report entries. Their syntax is free and even allows for multi-line text.

Report entries per instruction

Report entries for instructions are for information only. They act as the context of the verifications inside the report.

Allocations

Allocations show the name of the agent, the name of the allocated input and the actual value.

21/12/2021;08:39:26.996414; allocate | myAgent i myDouble 23.56

Service call

Service calls show the name of the agent, the name of the service and the parameters passed for the call.

21/12/2021;08:39:46.879426; call | myAgent myService 324 23.654 1 "string argument" "0xaa22cc"Code language: JavaScript (javascript)

Shout & whisper

Shout and whisper instructions show the name of the channel or the agent, and the text message that is sent.

21/12/2021;08:40:14.142332; shout | MY_CHANNEL | message with several words
21/12/2021;08:40:14.142397; whisper | otherAgent | message with several wordsCode language: JavaScript (javascript)

Agent start & stop

Agent start provides the name of the peer and the name of the agent to start on this peer. Agent stop provides the name of the agent to stop.

21/12/2021;08:40:21.201047; start | myMainAgentOrPeer agentInPeer
21/12/2021;08:40:22.203968; stop | myAgent

Quitting an agent (a peer)

Agent quit provides the name of the agent to be terminated.

21/12/2021;08:40:23.206787; quit | myAgent

User actions

Depending on their type, user actions respectively provide the message, the answered character and the message, the entered text and the message.

21/12/2021;08:39:50.458258; ask anykey | question 1 without text
21/12/2021;08:39:52.102515; ask keypress | y | question 2 with specific characters
21/12/2021;08:39:53.238046; ask text | typed text as an answer | question 3 with free textCode language: JavaScript (javascript)

Sleep

Sleep instructions provide a start and a stop entry with the planned sleep duration.

21/12/2021;08:40:14.144517; sleep start | 123 ms
21/12/2021;08:40:14.272572; sleep end | 123 ms

NB: sleep instructions on any non-realtime system are not always exact. The planned duration can be compared to the start and stop timestamps if needed.

System commands

System instructions provide a start, a timeout and a stop entry with the used command line. The timeout entry provides the timeout duration. The stop entry also provides at least the first characters of the console output of the executed command line.

21/12/2021;08:40:44.775001; system start | /usr/local/bin/igsTime --device en0 --port 5669 --verbose
21/12/2021;08:40:42.497918; system timeout | 1500 ms | /usr/local/bin/igsTime --device en0 --port 5669 --verbose | KILL SIGINT
21/12/2021;08:40:46.515922; system stop | /usr/local/bin/igsTime --device en0 --port 5669 --verbose | using log file /Users/steph/Documents/IngeScape/logs/igsTime.logCode language: JavaScript (javascript)

Report entries per verification

Output assertions

If OK, output assertions provide the successfully achieved comparison. If NOK, the comparison is followed by the value actually published.

21/12/2021;08:39:32.712784; assert OK | myAgent.myOutput >= 10
21/12/2021;08:39:31.130711; assert NOK | myAgent.myOutput <= 10 | 454233502.000000

Silence assertions

Silence assertions provide the expected silence duration, the name of the agent and if needed the precise output.

21/12/2021;08:39:40.748168; silence OK | 1500 ms | myAgent
21/12/2021;08:39:40.748168; silence OK | 1500 ms | myAgent.myOutput
21/12/2021;08:39:44.747424; silence NOK | 1500 ms | myAgent
21/12/2021;08:39:44.747424; silence NOK | 1500 ms | myAgent.myOutput

Service assertions

If OK, service assertions provide the name of the called service, the name of the calling agent and the list of the arguments with their expected values, which are identical to the received values. If NOK, service assertions add the list of the received values that do not match expected values.

21/12/2021;08:39:46.883546; service OK | someService from myAgent with arg1 = 3240 and arg2 = 23.6543 and arg3 = 0 and arg4 = call parameter2
21/12/2021;08:39:46.883546; service NOK | someService from myAgent with arg1 = 3240 and arg2 = 23.6543 and arg3 = 0 and arg4 = call parameter2 | arg1 = 324 and arg3 = 1Code language: JavaScript (javascript)

Channel assertions

Channel assertions provide the name of the channel, the expected message, and if the assertion is NOK, the message actually received.

21/12/2021;08:41:24.244037; channel OK | MY_CHANNEL | expected message
21/12/2021;08:41:24.244037; channel NOK | MY_CHANNEL | expected message | received message

Channel silence assertions

Channel silence assertions provide the name of the channel, the silence duration and the assertion status.

21/12/2021;08:41:24.244037; channel silence OK | 5000 ms | MY_CHANNEL
21/12/2021;08:41:24.244037; channel silence NOK | 5000 ms | MY_CHANNEL

User assertions

User assertions recall the question asked to the user which has been answered by ‘y’ (i.e. OK) or ‘n’ (i.e. NOK).

21/12/2021;08:41:24.244037; user OK | are you ready ?
21/12/2021;08:41:24.244037; user NOK | are you ready ?

Agent entered assertions

“Agent entered” assertions provide the name of the agent and the assertion status.

21/12/2021;08:41:24.244037; agent entered OK | myAgent

Agent exited assertions

“Agent exited” assertions provide the name of the agent and the assertion status.

21/12/2021;08:41:24.244037; agent exited OK | myAgent

Log verifications

Log verifications provide the duration, the name of the logged agent, the log level and the expected log message or expression. If an expression is used, log verifications add the log message actually received. If the log times out, a NOK entry is written.

21/12/2021;08:39:33.829083; log OK | 15000 ms | macosAgent DEBUG model_write_iop;set output value1 to double 329863108.000000
21/12/2021;08:39:33.829083; log OK | 15000 ms | macosAgent DEBUG model_write_iop(.*) | macosAgent;DEBUG;model_write_iop;set output value1 to double 329863108.000000
21/12/2021;08:39:33.829083; log NOK | 15000 ms | macosAgent DEBUG model_write_iop(.*) | TIMEOUTCode language: JavaScript (javascript)

Compare

If OK, compare verifications provide the name of the compared file, followed by the name of the reference file. If NOK, the file names are preceded by the number of pixels that differ between the two images.

21/12/2021;08:40:23.743363; compare OK | /Users/steph/Desktop/reference.png | /Users/steph/Desktop/reference.png
21/12/2021;08:40:23.743363; compare NOK | 83803 pixels differ | /Users/steph/Desktop/reference.png | /Users/steph/Desktop/reference.pngCode language: JavaScript (javascript)

Report entries for blocks

Blocks provide a start and a stop entry. The start entry provides the title and the description of the block. The stop entry also provides the title and description but also indicates the global status of the block, i.e. if all verifications inside it are OK or if at least one of the verifications is NOK.

The following entries are for an OK block.

21/12/2021;08:40:23.208481; block start | title | description
...
21/12/2021;08:40:27.821993; block OK | title | description

The following entries are for a NOK block.

21/12/2021;08:40:23.208481; block start | title | description
...
21/12/2021;08:40:27.821993; block NOK | title | description

The difference between an OK and a NOK block are indicated simply with the “OK” and “NOK” texts.

The following entries are for a NOK block because of a block timeout.

21/12/2021;16:30:50.825328; block start | title | description
...
21/12/2021;16:30:52.800997; block NOK | title | description | TIMEOUT

Verification timeouts inside blocks

If blocks enable a timeout any verification inside a block can be discarded prematurely, except for user assertions which are blocking with user input in the console. All discarded verifications still create an entry in the test report with a NOK status and a “BLOCK TIMEOUT” indication.

21/12/2021;08:39:31.130711; assert NOK | myAgent.myOutput <= 10 | BLOCK TIMEOUT
21/12/2021;08:39:44.747424; silence NOK | 1500 ms | myAgent.myOutput | BLOCK TIMEOUT
21/12/2021;08:39:46.883546; service NOK | someService from myAgent | BLOCK TIMEOUT
21/12/2021;08:41:24.244037; channel NOK | MY_CHANNEL | expected message | BLOCK TIMEOUT
21/12/2021;08:39:33.829083; log NOK | 15000 ms | macosAgent DEBUG model_write_iop(.*) | BLOCK TIMEOUT
21/12/2021;08:40:23.743363; compare NOK | /Users/steph/Desktop/reference.png | /Users/steph/Desktop/reference.png | BLOCK TIMEOUTCode language: JavaScript (javascript)

Commenting the scripts

We encourage the script writers to fill their scripts with useful and relevant comments regarding the test strategies. Ingescape V&V scripts support C-like comments, whether on a single line or on multiple lines.

//single line comment

/* 
* multiple lines comment
*/

assert silence myAgent 5000 //comment after an instruction, a verification or a manual report entryCode language: JavaScript (javascript)

If some parts of the comments deserve to be part of the test report, simply use manual report entries. This type of entries is displayed in the code, the console and the report, making them easy to find and read.