Testing Asynchronous Reactive Systems: Beyond the ioco framework

Manual testing can be rather time consuming and prone to errors specially when testing asynchronous reactive systems. Model based testing is a well-established approach to verify reactive systems speciﬁed by input output labeled transition systems (IOLTSs). One of the challenges stemming from model based testing is verifying conformance and, also, generating test suites, primarily when completeness is a required property. In order to check whether an implementation under test is in compliance with its respective spec-iﬁcation one resorts to some form of conformance relation that guarantees the expected behavior of the implementations, given the behavior of the speciﬁcation. The ioco relation is an example of such a conformance relation. In this work we study another conformance relation based on formal languages. We also investigate how to generate complete test suites for IOLTS models, and discuss the complexity of the test generation mechanism under this new conformance relation. We also show that ioco is a special case of this new conformance relation. Further, we relate our contributions to more recent works, accommodating the restrictions of their classes of fault models as special cases, and we expose the complexity of generating any complete test suite that must satisfy their restrictions.


Introduction
Software testing has been an important part in system development processes, with the goal of improving the quality of the final products. The systematic use of formal methods and techniques has taken prominence when accuracy and critical guarantees are of paramount importance to the development process, such as when failures can cause severe damages. Testing approaches based on formal models have the added advantage that test suite generation, with proven completeness guarantees, can be effectively and precisely automated, for certain classes of specification models.
Some formalisms, such as Finite State Machines (FSMs) [1,2,3], capture some aspects of systems' behaviors. However, in FSM models, input actions from the environment and output actions produced by the implementations under test are strongly related, and must occur synchronously. This may limit the designers' ability to model more complex asynchronous behaviors.
This work studies aspects of a more powerful formalism where the exchange of input and output stimuli can occur asynchronously, namely, the class of Input/Output Labeled Transition Systems (IOLTSs) [4,5]. In this context, model based testing [6,7] has been widely used as a formal framework to verify whether an implementation under test (IUT) is in conformance to a given specification, according to a given fault model and a given conformance relation [8,9,4]. In particular, the ioco relation has been proposed as a suitable conformance relation for testing IOLTS models [5]. We propose a new notion of conformance relation for testing IOLTS models. Under this new notion, it is possible to accommodate wider classes of IOLTS models, thus removing some of the structural restrictions imposed by other approaches. Our main contributions can be summarized as follows: -The new notion of conformance allows for the specification of arbitrary desired, as well as undesired, behaviors that an IUT must comply to. When these specifications can be cast as regular languages, we show that a certain regular language is a complete test suite that can be used to detect the presence of all desired behaviors and the absence of any undesired behaviors in any IUT.
-In a "white-box" testing scenario, when one has access to the internal structure of the IUTs, we prove the correctness of a polynomial time algorithm that can be used for checking conformance against the new relation. If the specification model is fixed, then the algorithm runs in linear time in the number of states in the implementation.
-In a "black-box" testing environment, when the tester does not have access to the internal structure of the implementations, we show that the classical ioco relation [5] is a special case of this new conformance relation. We also show how to generate complete test suites that can be used to verify ioco-conformance under the same set of fault models considered by Tretmans [5], and for implementations with any number of states, independently of the number of states in the specification. We remark that our models and definitions in Section 5 are co-extensive with those found in [5]. Further, in this setting, we prove that 1.61 m is an asymptotic worst case lower bound on the size of any ioco-complete test suite, where m is the number of states of the largest implementation to be put under test. We also show that our approach attains such a lower bound.
-In a recent work, Simão and Petrenko [10] discussed how to generate complete test suites for some classes of restricted IOLTSs. We prove that our method can construct a complete test suite for such classes of models, when the same restrictions must be satisfied. Also, in [10] the complexity of the generated test suites was not studied. Here we prove an asymptotic exponential worst case lower bound for any ioco-complete test suite that must satisfy the same restrictions as in [10] and, further, we prove that our method also attains this lower bound.
We briefly comment on works that are more closely related to our study. See Section 7 for a more expanded view. Tretmans [5] proposed the ioco-conformance relation for IOLTS models, and developed the foundations of an ioco-based testing theory, where IUTs are treated as "black-boxes". In this testing architecture the tester is seen as an artificial environment that drives the exchange of input and output symbols with the IUT during test runs. Some restrictions must be observed by the specification, implementation and tester models, such as input-completeness and output-determinism. Simão and Petrenko [10] also described an approach to generate finite complete test suites for IOLTSs. They, however, also imposed a number of restrictions upon the specification and the implementation models in order to obtain finite complete test suites. They assumed test purposes to be single-input and also output-complete. Moreover, specifications and implementations must be input-complete, progressive, and initially-connected, so further restricting the class of IOLTS models that can be tested according to their fault model. Here, we remove many of such restrictions.
Section 2 establishes notations and preliminary results. A new notion of conformance relation is defined in Section 3. Generating complete test suites for verifying adherence to the new conformance relation, and its complexity, is described in Section 4. Section 5 visits the ioco-conformance relation and establishes exponential lower bounds on the size of ioco-complete test suites. Section 6 looks at another class of IOLTS models and show how to obtain complete test suites for this class, and discusses complexity issues when working with models of this class. Section 7 comments on related works, and Section 8 offers concluding remarks.

Preliminaries and Notation
In this section we define Labeled Transition Systems (LTSs) and Input/Output Labeled Transition Systems (IOLTSs). For completeness, we also include standard definitions and properties of regular languages and finite state automata (FSA). Some preliminary results associating LTSs and FSA are given.

Basic Notation
Let X and Y be sets. Then P(X) = {Z | Z ⊆ X} is the power set of X, and X − Y = {z | z ∈ X and z ∈ Y } is the set difference. We let X Y = X ∪ Y . When no confusion can arise, we write X y instead of X {y} . If X is a finite set, the size of X will be indicated by |X|.
An alphabet A is any non-empty set of symbols. A word over A is any finite sequence σ = x 1 . . . x n where n ≥ 0 and x i ∈ A, for i = 1, 2, . . . , n. When n = 0 we have the empty sequence ε. The set of all finite words over A is denoted by A . When we write x 1 x 2 . . . x n ∈ A , it is assumed that n ≥ 0 and that x i ∈ A, 1 ≤ i ≤ n, unless noted otherwise. The length of α ∈ A is indicated by |α|. Hence, |ε| = 0. Let σ = σ 1 . . . σ n and ρ = ρ 1 . . . ρ m be words over A. The concatenation σρ of σ and ρ is the word σ 1 . . . σ n ρ 1 . . . ρ m . Clearly, |σρ| = |σ| + |ρ|. A language G over A is a set G ⊆ A . Let G 1 , G 2 ⊆ A . The product G 1 G 2 is {σρ | σ ∈ G 1 , ρ ∈ G 2 }, and the complement is G 1 = A − G 1 .
Definition 1 Let A, B be alphabets. A homomorphism from A to B is any function h : A → B .
A homomorphism h : A → B can be extended to a function h : A → B where h(ε) = ε, and h(aσ) = h(a) h(σ) where a ∈ A. We can further lift h to a function h : P(A ) → P(B ), by letting h(G) = ∪ σ∈G h(σ), for all G ⊆ A . We often write h in place of h, or of h, when no confusion can arise. When a ∈ A is any symbol, we define the simple homomorphism h a : A → (A − {a}) by letting h a (a) = ε, and h a (x) = x when x = a. So, h a (σ) erases all occurrences of a in σ.

Labeled Transition Systems
A Labeled Transition System (LTS) is a formal model that is convenient to express asynchronous exchange of messages between participating entities, in the sense that outputs do not have to occur synchronously with inputs, but are generated as separate events. It consists of a set of states and a transition relation between states. Each transition is guarded by an action symbol. We first give the finite syntactic description of LTSs. A semantic structure that attributes meaning to a syntactic description will be given shortly in the sequel.
Definition 2 A Labeled Transition System is a tuple S = S, s 0 , L, T : 1. S is a finite set of states or locations; 2. s 0 is the initial state, or initial location; 3. L is a finite set of labels, or actions; τ / ∈ L is the internal action symbol; The symbol τ is used to model any actions that are not exchanged as messages, that is, actions that cause only an internal change of states. The class of all LTSs over an alphabet L will be denoted by L(L).
The semantics of an LTS is given by its traces, or behaviors. But first, we need the notion of paths in an LTS.
Definition 3 Let S = S, s 0 , L, T be an LTS and p, q ∈ S. Let σ = σ 1 , . . . , σ n ∈ L τ . We say that σ is: 1. a path from p to q if there are states r i ∈ S, 0 ≤ i ≤ n and, additionally, we have (r i−1 , σ i , r i ) ∈ T , 1 ≤ i ≤ n, r 0 = p and r n = q; 2. an observable path from p to q if µ is a path from p to q and σ = h τ (µ).
We say that the paths start at p and end at q.
A path σ from p to q is just a sequence of symbols that allows one to move from state p to state q. Note that a path may include transitions over the internal symbol τ . An observable path arises from any ordinary path from which internal symbols have been erased. An external observer will not see the internal transitions, as the model moves from state p to state q. It is also clear that an observable path is not necessarily a path.
Example 2 Consider the LTS of Figure 1. The following are paths starting at s 2 : ε, b, bcτ , cτ bbbtτ b. The following are observable paths starting at s 2 : ε, b, bc, cbbbtb. There are observable paths of any length from s 2 to s 1 . Note that, starting at s 2 , the path cτ b may lead to either s 1 or back to s 2 . 2 If σ is a path from p to q we write p σ → q. We write p σ → when there is some q ∈ S and p σ → q; likewise, p → q means that there is some σ ∈ L τ such that p σ → q. Also p → means p σ → q for some q ∈ S and some σ ∈ L τ . To emphasize the underlying LTS we write p σ → S q. The symbol ⇒ will indicate observable paths, with the same abbreviations used with the → relation. Note that s ⇒ p if and only if s → p.
Paths starting at a state p are traces of p. The semantics of an LTS are the traces of the initial state.
Definition 4 Let S = S, s 0 , L, T be an LTS and let p ∈ S. The set of traces of p is tr(p) = {σ | p σ →}, and the set of observable traces of p is otr(p) = {σ | p σ ⇒}. The semantics of S is the set tr(s 0 ), and the observable semantics of S is the set otr(s 0 ).
We may write tr(S) for tr(s 0 ) and otr(S) for otr(s 0 ). If S has no τ -labeled transitions, then otr(S) = tr(S). Figure 1. the sequences ε, bbc, btτ bcτ bbb are in the semantics of S. The observable semantics of S includes: ε, bcbt, bbbtbcbtb.

Example 3 Consider the LTS of
2 We can restrict the class of LTS models somewhat, with no loss of descriptive capability. First, we assume that all τ -moves change state, and we do away with states that are not reachable from the initial state.
Remark 1 In Definition 2 we will always assume that (s, τ, s) ∈ T and that s 0 → s holds, for any s ∈ S.
Internal actions can facilitate the specification of formal models. For example, specifying that "after delivering money, the ATM returns to the initial state", can be done by a τ -move back to the initial state, if there is no need to exchange messages with a user. Note that Remark 1 does not prevent the occurrence of cycles labeled by τ -moves, which would lead a livelock. Sometimes, however, we may not want such behaviors, or we might want that no observable behavior leads to two distinct states. This motivates the next definition.
Definition 5 We say that S = S, s 0 , L, T is deterministic if s 0 σ ⇒ s 1 and s 0 σ ⇒ s 2 imply s 1 = s 2 , for all s 1 , s 2 ∈ S, and all σ ∈ L .
The following result is immediate.
Proposition 1 A deterministic model has no τ -moves.

Finite State Automata
An LTS induces a finite automaton with simple properties. In particular, τ -labeled transitions correspond to nondeterminism induced by ε-moves in the automaton. For completeness, we give here the basic definitions.
1. S is a finite set of states; 2. s 0 ∈ S is the initial state;

3.
A is a finite non-empty alphabet; 4. ρ ⊆ S × (A ∪ {ε}) × S is the transition relation; and 5. F ⊆ S is the set of final states.
A transition (p, ε, q) ∈ ρ is called an ε-move of A.
The semantics of a FSA is the set accepted words.
Determinism requires no ε-moves and the transition relation is a function. Completeness means that from any state there is an outgoing transition on any symbol.
The subset construction [11,12] gives: Proposition 2 Let A be FSA. We can effectively construct a deterministic FSA B such that L(A) = L(B).
Note that B may have up to 2 k states if A has k states.
The next constructions require simple algorithms.
Proposition 3 Let A and B be FSAs. We can easily obtain and FSA C such that 2. L(C) = L(A) and C is complete.
Also, if A and B are deterministic, then so is C.
We can convert τ -moves of an LTS into ε-moves of a FSA, and turn all states of the LTS into final states of the FSA, and vice-versa. Then algorithmic constructions for FSAs can be used to transform the LTSs.

Definition 9
We have the following two associations: 1. Let S = S, s 0 , L, T be an LTS. The induced FSA is A S = S, s 0 , L, ρ, S where (p, , q) ∈ ρ if and only if (p, , q) ∈ T, (p, ε, q) ∈ ρ if and only if (p, τ, q) ∈ T.
2. Let A = S, s 0 , A, ρ, S be a FSA. The induced LTS is S A = S, s 0 , A, T where (p, a, q) ∈ T if and only if (p, a, q) ∈ ρ, (p, τ, q) ∈ T if and only if (p, ε, q) ∈ ρ.
An easy induction establishes the following Proposition 4 The associations in Definition 9 give otr(S) = L(A S ) and otr(S A ) = L(A).
If S is an LTS, Definition 9(1) gives a FSA A where all states are final and L(A) = otr(S). Using Proposition 2 we can get a deterministic FSA B with L(B) = L(A). The subset construction [11,12] says that all states in B are final, except possibly for the state representing the empty set. Removing this state we get a FSA C in which all states are final and such that L(C) = L(B). Now we can use Definition 9(2) and get a LTS T with otr(T) = L(C). Moreover, since B was deterministic, it is easy to see that T is also deterministic.
Proposition 5 Let S be an LTS. We can construct a deterministic LTS T such that otr(S) = tr(T) = otr(T).
Note that in the worst case T can grow exponentially.

Input Output Labeled Transition Systems
Sometimes we wish to say that symbols are "received" from the environment as inputs, while others are "sent back" as outputs. An Input/Output Labeled Transition System (IOLTS) allows for this.
Definition 10 An Input Output Labeled Transition System (IOLTS) is a tuple I = S, s 0 , L I , L U , T , where: 1. L I is a finite non-empty set of input actions; 2. L U is a finite non-empty set of output actions; 3. L I ∩ L U = ∅, and L = L I ∪ L U is the set of actions; 4. S I = S, s 0 , L, T is the underlying LTS of I.
IO(L I , L U ) is the class of all IOLTSs with input alphabet L I and output alphabet L U . Other works impose additional restrictions to the basic IOLTS model, but we do not need any restrictions at this point. We look at more restricted variations in Sections 5 and 6. Several notions involving IOLTSs will be defined by a direct reference to their underlying LTSs.

Definition 11
The semantics of an IOLTS I is the set otr(I) = otr(S I ).
When I is an IOLTS the notation → is the underlying LTS for I. IOLTSs generalize the simpler formalism of Meally machines [13], where communication is synchronous. In an IOLTS, inputs and outputs happen asynchronously, which facilitates the specification of more complex behaviors, like in reactive systems.  Example 4 Figure 2 was adapted from [5]. It describes a strange coffee machine. When the user hits the start button, indicated by the input label but?, the machine chooses to go either to s 1 or to s 2 . If it goes to s 1 , no matter how many extra times but? is hit the loop at s 1 keeps the machine at s 1 ; then, asynchronously, the machine dispenses a cup of tea, signaled by the output label tea!, reaching state tea. Next, the machine performs an internal action, and returns to the start state. The left branch indicates a similar behavior, but now it dispenses a cup of coffee, and by another internal action returns it to the start state. 2

Conformance testing
We define a new and generalized notion of conformance, allowing for sets of desired and undesired behaviors in IOLTS models. We study the relationship of this notion to ioco-conformance [5], and show that the latter is a special case of this new conformance relation.

Remark 2
We deal with the notion of quiescent states in Section 5. For now we note that the treatment of quiescent states will require the addition of a new symbol δ to the output alphabet, and some δ-transition to the set of transitions. Since the results obtained in this section are valid for any LTS model, they will remain valid for those variations that treat quiescent states explicitly.

The New Conformance Relation
We consider a language D, the set of "desirable" behaviors, and a language F , the set of "undesirable", behaviors. We say that an implementation I conforms to a specification S according to (D, F ) if no undesirable behavior observable in I is specified in S, and all desirable behaviors observable in I are specified in S. For an equivalent way of expressing these conditions write otr(S) for the complement of otr(S). Example 5 Let S be as in Figure 3(a) and I as shown in Figure 3(b). Let D = (a + b) ax and F = ba b. We want to check if I conf D,F S holds.
Since baab ∈ F ∩ otr(S)we get F ∩ otr(S) = ∅. Because baab ∈ otr(I), we conclude that conf D,F S does not hold. Alternatively, it is easy to see that ababax ∈ otr(S), and clearly ababax ∈ D, so that   In this case, either F or D is enough to guarantee non-conformance. Now take F = abb x and D = aaa b(bb) ax. By inspection, we get σx ∈ otr(I) for all σ ∈ abb , so that otr(I) ∩ F ∩ otr(S) = ∅. Moreover, for any σ ∈ D we get s 0 σ ⇒ s 2 so that D ⊆ otr(S). Hence, D ∩ otr(S) = ∅. Therefore, otr(I) ∩ (D ∩ otr(S)) ∪ (F ∩ otr(S)) = ∅, and we see that I conf D,F S holds.
In Subsection 3.2 we will use this example to relate the new relation to ioco relation [5]. 2. Let C and E be disjoint subsets of locations of I. The set of allowed behaviors, H C , must lead to a location in C, and the set of forbidden behaviors, H E , lead to a location in E. Then, I conf H C ,H E S if and only if any allowed behavior of I is also specified in S and no forbidden behavior of I is specified in S. Clearly, D and F are regular in all cases listed above. In general, if I conforms to S according to (D, F ), then we can assume D and F to be finite languages.

The ioco Conformance Relation
The ioco-conformance relation [5] essentially requires that if σ is any observable trace of both an implementation I and of a given specification S, then if σ leads I to a location from which it can move on a symbol , then σ must also lead S to a location from which it can move on . This motivates the following definitions.  Proof From Definition 9 we see that otr(S) is regular, and so D is also regular. Assume I conf D,∅ S. From Definition 12 it is clear that I conf D,∅ S is equivalent to otr(I) ∩ D ⊆ otr(S). In order to prove that I ioco S, let σ ∈ otr(S) and let ∈ out(q 0 after σ). We must show that ∈ out(s 0 after σ). Because ∈ out(q 0 after σ) we get σ, σ ∈ otr(I). Since ∈ L U , we get σ ∈ otr(S)L U and so σ ∈ D. Hence, σ ∈ otr(I) ∩ D, and then σ ∈ otr(S). So, ∈ out(s 0 after σ), as desired.
Example 7 Let S be as in Figure 3(a), and I as in Figure 3(b) with the extra transition q 3 From Figure 3(a) and the new I it is apparent that s 0 aa ⇒ s 3 and also q 0 aa ⇒ q 3 . Also, x ∈ out(q 3 ), but x ∈ out(s 3 ). So, by Definition 13, I ioco S does not hold. Now take σ = aax. Since aa ∈ otr(S), we get σ ∈ otr(S)L U = D. Also, σ ∈ otr(I) and σ ∈ otr(S), so that σ ∈ otr(I) ∩ D ∩ otr(S). From Proposition 6, we see that I conf D,∅ S does not hold, as expected. 2 The next example shows that the new conformance, conf D,F , is able to capture non-conformance situations where the ioco relation would yield positive verdicts. Figure 3(a) and I as in Figure 3

Example 8 Let S as in
, but x ∈ out(s). By Definition 13, I ioco S holds. Now let F = ∅, D = (a + b) ax, and take σ = ababax ∈ D. We see that σ ∈ otr(S) and σ ∈ otr(I). So that σ ∈ otr(I) ∩ D ∩ otr(S). From Proposition 6, I conf D,∅ S does not hold, whereas I ioco S would always hold. 2 We know that ioco-conformance is equivalent do I conf D,F S when F = ∅. By taking F = ∅, however, gives the test designer even more freedom to check whether some behaviors that occur in the specification are, or are not, also represented in the implementation. In particular, when D = ∅ and a verdict of conformance is obtained, the behaviors specified in F are not present, whereas a verdict of non-conformance would say that some behavior of F is present in the implementation.
Lemma 1 and Corollary 1 lead to the next result.

Test Generation for IOLTS models
We show how to generate complete test suites for checking conformance of IOLTS models according to the new relation defined in Section 3, and for arbitrary sets of behaviors D and F . For deterministic IOLTS models, the algorithm has time complexity linear on the number of states of the implementation, regardless of the specification. Recall Remark 2.

Complete Test Suite Generation
A test suite is a set of words over the IOLTS alphabet.

Definition 14
Let L be a set of symbols. A test suite is any set T ⊆ L . Each σ ∈ T is called a test case.
Loosely speaking, an IUT I conforms to a specification S when behaviors of I do not deviate from those specified by S. Test suites are designed to flag any such deviations, or guarantee that none is possible.
Given a pair of languages (D, F ) and a specification S, we want to generate test suites T that are sound, that is, if I adheres to T then I also (D, F )-conforms to S. The converse is also desirable, that is, when I conf D,F S then adherence of I to T should also be guaranteed.
Definition 16 Let S ∈ IO(L I , L U ) and T ⊆ L . Also, let D, F ⊆ L . We say that: 3. T is complete for S and (D, F ) if it is both sound and exhaustive for S and (D, F ).
We view adherence as a syntactic notion, as it is verified just by the emptiness of a set intersection. On the other hand, conformance is closer to a semantic notion, since it is based on output events generated by the intrinsic behavior of the IUT. So, here, we preferred to use soundness in the tradition in mathematical logic, in the sense that once a sentence is (syntactically) proven, it is guaranteed to be (semantically) true. On the other direction, also as in mathematical logic, when every (semantically) true sentence has a (syntactic) proof, the theory is said to be complete. Here instead, we used the term exhaustive to indicate that if an IUT (semantically) conforms to a specification, then it must (syntactically) adhere to the test suite, reserving the term complete to indicate situations when both soundness and exhaustiveness are guaranteed to be present. Our notions of soundness and exhaustiveness might have their meaning reversed when compared to other definitions in the literature. But, of course, when both soundness and exhaustiveness are guaranteed, as is the case with our results, this reversal of notation is innocuous.
We can always have complete test suites. Furthermore, in a sense they are also unique. Because we have no control over the size of the witness σ ∈ otr(I), it was crucial that we had no restrictions on the size of I. This indicates that the size of the implementations will affect the complexity of the test suites can verify (D, F )-conformance. We investigate the complexity of complete test suites in the next subsection. Lemma 2 says that T = (D ∩ otr(S)) ∪ (F ∩ otr(S)) is complete for S and (D, F ). So checking if I conf D,F S is equivalent to checking if otr(I) ∩ T = ∅. If D and F are regular, that is D = L(A D ) and F = L(A F ) for some FSAs A D and A F , then using Propositions 4 and 3 we can construct a FSA A T such that T = L(A T ). Further, by the same propositions we know that otr(I) ∩ T is also regular. Moreover, if I is a "white-box", that is, we have access to its the syntactic description, we can construct the FSA A I such that otr(I) = L(A I ) and so, using Proposition 3 we can obtain a FSA A such that L(A) = otr(I) ∩ T . Now, a simple breadth-first algorithm can check if L(A) = ∅, so that we can effectively decide if I (D, F )-conforms to S. Proposition 7, in the sequel, details the time complexity of such an algorithm.

On the Complexity of Test Suites
An important issue is the size of test suites. If S has n states and t transitions, then n − 1 ≤ t ≤ n 2 , and it is reasonable to take t as the size of S.
Let D, F ⊆ L , and S = S, s 0 , L, T be deterministic with n S states. Lemma 2 says that T = (D ∩ otr(S)) ∪ (F ∩ otr(S)) is complete for S and (D, F ). Assume that D and F are regular with L(A D ) = D and L(A F ) = F for deterministic FSAs A D and A F with n D and n F states, respectively. By Proposition 4 we can construct a deterministic FSA A 1 with n S + 1 states and L(A 1 ) = otr(S). The proof of Proposition 3 shows that we can get a deterministic FSA A 2 with at most (n S + 1)n F states and such that L(A 2 ) = F ∩ otr(S).
Proof The preceding discussion and Lemma 2 yield T. A breadth-first search over T gives the algorithm. 2 We can state a similar result for checking ioco-conformance when a description of the IUT is available.
Theorem 9 Let S, I ∈ IO(L I , L U ) deterministic with n S and n I states, respectively, and n L = |L I ∪ L U |. There is a O(n S n I n L ) algorithm to check I ioco S.
Proof Let A be the FSA with n S and L(A) = otr(S) given by Proposition 4. Starting with A we construct a deterministic FSA B with n S + 2 as follows. Let s I and s U be new states. For any state s of A and any symbol ∈ L I ∪ L U : if there is no transition (s, , p) in A, for any state p of A, we add to B the transition (s, , s I ) when ∈ L I , and the transition (s, , s U ) when ∈ L U . Finally, we turn B into a deterministic FSA by adding the transitions (s U , , s I ) and (s I , , s I ) for all ∈ L I ∪ L U . Let the initial state of B to be as in A, and s U be the only final state of B. It is easy to see that L(B) = otr(S) ∩ otr(S)L U . Use Proposition 4 again to get a deterministic FSA C with n I + 1 states and L(C) = otr(I). Proposition 3 gives a deterministic FSA T with (n S + 2)(n I + 1) states and L(T) = otr(I) ∩ otr(S) ∩ otr(S)L U . Now use Corollary 3. A simple breadth-first search on T gives the algorithm.   Proof Follows from Theorem 9.

Testing IOLTS with Test Purposes
In Section 4 the testing architecture presupposed that one has access to a syntactic description of the IUTs. In a contrasting setting, where IUTs are "black-boxes", we do not have access to their syntactic structure. In this case, we can imagine a scenario where there is a tester or "artificial environment", T, and an IUT, I, which are connected by a "zero-capacity" bidirectional and lossless channel. At each step either of two moves may occur: (i) T issues an output action symbol x to I and changes state. At once, I accepts x as an input symbol and also changes state; or (ii) the move is reversed with I sending an output symbol y and changing its state, while T accepts y as an input symbol and also changes state. Clearly, a sequence of type (i) moves can occur before a type (ii) move occurs; and vice-versa. We see that the input and output sets of symbols are interchanged in T and I. Hence, one should write I = S, s 0 , L I , L U , T and T = Q, q 0 , L U , L I , R . We will always refer to a symbol x as an input or an output symbol from the perspective of the IUT I, unless there is an explicit mention to the contrary.
It may happen, however, that I reaches a so called quiescent state. Informally, those are states from which there are no transitions labeled by some output symbol [7,5]. From this point on I could no longer respond to T, and the latter will have no way of "knowing" whether I is rather slow, has timed out, or will not ever respond. If we want to reason about this situation, within the formalism, it will be necessary to somehow signal to the tester that the IUT is in a quiescent state. A usual mechanism [5] is to add a special symbol δ as an output symbol to I, and hence as an input symbol to T. Then I sends δ to T when it reaches a quiescent state s. Since I is not changing states in this situation, we also add the self-loop s δ → s to the transitions of I. On the tester side, being on a state q and upon receiving a δ symbol it decides whether that is appropriate or not, depending on the fault model it is supposed to describe. If that was adequate, the tester may then move to another state through a move q δ → q 1 . In this section we apply our results of Section 4 to the architecture described in Tretmans [5], and where IUTs are black-boxes. But first we must formally prepare our models to deal with quiescent states. Further, when we consider models with quiescent states, we must ensure that we are treating the same class of models as in [5], and that ioco-conformance, as in Definition 13, is the same as the ioco relation studied in [5].
We proceed as follows: 1. We define a variation of IOLTS models, where the special symbol δ is used to indicate quiescence.
2. We formalize the notion of an external tester in order to reason precisely about test runs. For that, we define test purposes in Subsection 5.1.
3. When IUTs are black-boxes, it is customary to impose a series of restrictions over the formal models that describe specifications, IUTs and test purposes [5], so that some guarantees about the exchange of messages can be stated. Although our methods impose almost no restrictions on the formal models, except for regularity of the D and F sets, in Subsection 5.2 we look at the extra model restrictions imposed by Tretmans [5]. 4. In Subsection 5.3 we look at the complexity of complete test purposes under these restrictions, and we establish a new asymptotic worst case exponential lower bound of the size of these test suites. Other works hinted at possible exponential upper bounds on the size of complete test suites for these models, but we are not aware of any precise lower bounds, under the same restrictions as mentioned here.
We start with the following variation of Definition 10 incorporating quiescence in IOLTS models.
We indicate the class of all δ-IOLTSs with input alphabet L I and output alphabet L U by IO δ (L I , L U ). The following example illustrates the situation. In the test architecture studied in [5], given an IOLTS S = (S, s 0 , L I , L U , T ), a state s ∈ S is said to be quiescent if, for all x ∈ L U ∪{τ } we have s x → in S, a fact there indicated by δ(s). Then, assuming δ ∈ L U , the extended model S δ = S, s 0 , L I , L U ∪ {δ}, T ∪ T δ is defined, with T δ = {(s, δ, s) | δ(s)}, so that S δ includes self-loops on the new output symbol δ at any quiescent state. Since in this section we are applying our results of Sections 3 and 4 to the test architecture described in [5], it is important to guarantee that the class of δ-IOLTS models from Definition 17 is coextensive with the class of extended models in [5]. Also, the ioco-relation used in [5] must coincide with that at Definition 13. All details of these considerations are given in Appendix 8, specially Proposition 17 for the guarantee that both classes of models are the same, and Proposition 21 that shows that both ioco-relations coincide.
In this section all IUTs are from IO δ (L I , L U ).

A Class of Fault Models
An IUT is a model from IO δ (L I , L U ). A tester, however, is a model from IO(L U , L I ), that is, from the perspective of the tester δ is now an ordinary input symbol. Upon receiving a δ-symbol, signaling quiescence on the IUT side, the test designer is free to specify any δ-transitions to drive the tester model, as deemed appropriate to the design of the test goals.
We will refer to testers as test purposes. We require that test purposes have two special states: pass and fail [5]. Once the fail or pass state is reached we have a test verdict, so it is reasonable to require that there are no paths from pass to fail, or vice-versa. A fault model is a finite set of test purposes, allowing for several conditions for acceptance and rejection to be verified. Since , it is more profitable to express some notions and results using the full IO(L I , L U ) class, and specialize only when needed. The exchange of symbols between two models can be described by their cross-product 2 .
Definition 19 Let T = Q, q 0 , L U , L I , R and I = S, s 0 , L I , L U , T be in IO(L I , L U ). Their cross-product is the LTS T × I = Q × S, (q 0 , s 0 ), L I ∪ L U , P , where ((q 1 , s 1 ), x, (q 2 , s 2 )) ∈ P if and only if either 1. x = τ , s 1 = s 2 and (q 1 , τ, q 2 ) ∈ R, or 2. x = τ , q 1 = q 2 and (s 1 , τ, s 2 ) ∈ T , or Behavior in the cross-product is tightly tied to behaviors in the participating IOLTSs, and conversely.  Proof Inductions on k ≥ 0 and |µ| ≥ 0, with h τ (µ) = σ. 2 Assume we want to verify if an IUT I ioco-conforms to a specification S using a test purpose T. That is, we want T to act as an "external environment" and drive a run of I, so that if T reaches a fail state in this run then I does not ioco-conform to S. Because the simultaneous run of T and I is described by their cross product, it is sufficient to check that T × I cannot reach a (fail, s) state. Conversely, when I does ioco-conform to S we need that T × I never reaches a (fail, s) state. Further, we may require that T correctly verifies ioco-conformance only when I is taken from a subclass of models.
Definition 20 Let I ∈ IO(L I , L U ), T ∈ IO(L U , L I ). We say that I passes T if (fail, q) is not reachable in T ×I, for any state q of I. We say that I passes a fault model T P if I passes all T in T P . Let S ∈ IO(L I , L U ), IMP ⊆ IO(L I , L U ). We say that T P is ioco-complete for S relative to IMP if I ioco S if and only if I passes T P , for all I ∈ IMP. When IMP = IO(L I , L U ) we say that T P is ioco-complete for S.
The following implies a single test purpose suffices. As an illustration we have the following: Example 12 Let S as in Figure 4(a) and recall that L I = {a, b} and L U = {x}. To avoid cluttering, quiescent δ-loops were not inserted in Figure 4(a). If needed, just add δ-self-loops at states s 0 , s 2 and s 3 . Also, this should pose no difficulty since a δ is treated as just any other symbol in L U . The argument in Lemma 3 yields the structure in Figure 5, but: (i) s I and all transitions into it are removed, and (ii) s U is relabeled fail and we do not need final states in T.
We have (t 0 , q 0 ) axbx ⇒ (fail, q 2 ) in T × I with I in Figure 4(b), so that I does not pass T. Since Lemma 3 says {T} is complete for S, then I ioco S should not hold. To check this, we have s 0 ba ⇒ s 3 in S and q 0 ba ⇒ q 3 in I. Thus x ∈ out(s 3 ) and and x ∈ out(q 3 ), so that out(q 0 after β) ⊆ out(s 0 after β). Hence, I ioco S does not hold according to Definition 13. 2

A Specific Family of Formal Models
Some extra restrictions are imposed by Tretmans [5] so that test runs can be adjusted to more practical situations. First, one requires test purposes to be acyclic, so that any test run is a finite process. Secondly, since one cannot predict in advance which output symbol a black-box IUT will be sending back at any instant, test purposes must be able to synchronize on any of such symbols, that is, the tester must be input-enabled. Further, since the tester drives the IUT, at any state it must be able to send at least one symbol to the IUT. Also, in order to avoid arbitrary choices and non-determinism, the tester is required to be output-deterministic, that is, at any state it can emit only one of its output symbols. Moreover, because fail and pass states already hold a verdict, it is required that the tester can only have self-loops at these states. Recall Definition 13. Input-enabledness is no serious restriction.
Corollary 5 For any specification S there is an ioco-complete fault model {T} where T is deterministic, input-enabled and has a single fail and no pass states.
Proof The test purpose constructed in proof of Lemma 3 is deterministic and input-enabled. 2 It is no surprise that the acyclic requirement imposes restrictions on the size of IUTs. Proof We see that a n x ∈ otr(S) ∩ (otr(S) L U ) for all n ≥ 0. Consider an IUT I n with transitions (q i−1 , a, q i ) for i = 1, . . . , n, (q n , x, q), where q 0 is the initial state. Clearly, I n is deterministic. We have a n x ∈ otr(I n ) so that otr(I n ) ∩ otr(S) ∩ otr(S) L U = ∅. Then Corollary 3 says that I n ioco S does not hold, for all n ≥ 0. Because T P is complete, we must have T ∈ T P such that I n does not pass T, that is, (fail, q) is reachable in T × I n . By Proposition 8 we have t 0 σ ⇒ fail in T and q 0 σ ⇒ r in I n , where t 0 is initial in T and q 0 is initial in I. By construction of I n , either σ = a k for some k ≤ n, or σ = a n x. We can not have σ = a k for any k ≥ 0 because then (fail, s 0 ) is reachable in T × S, implying that S does not pass T. Since T P is ioco-complete for S we get that S ioco S does not hold with S deterministic, a contradiction. Let σ = a n x. Since T is acyclic, except for self-loops at fail, and n can be arbitrarily large, t 0 a n x ⇒ fail in T imply again t 0 a n ⇒ fail in T for n large enough, and we reach a contradiction again. 2 In the preceding proof we can take S where a state s has a self-loop on any input symbol. This makes the proposition much more widely applicable. Next, we investigate the case where there is an upper bound on the number of states in the IUTs. The situation is more amenable in these cases. Since it is not possible to construct fault models comprised only of acyclic test purposes, and that are ioco-complete in general, we turn to the problem of obtaining such fault models that are m-ioco-complete, for a given m.
Proposition 10 Let S ∈ IO(L I , L U ) be deterministic, and let m ≥ 1. Then, there is a fault model T P which is m-ioco-complete for S, and such that all test purposes in T P are deterministic and acyclic.
Proof Let S = S, s 0 , L I , L U , T and S = {s i : 0 ≤ i < n}. Construct a directed acyclic multi-graph D with mn + 1 levels. At level i, 0 ≤ i ≤ mn, list all nodes in S as s 0,i , s 1,i , . . . , s n−1,i . Consider s j,k , k < mn. For each transition (s j , , s i ) of S: (i) if i > j, add an arc from s j,k to s i,k ; (ii) if i ≤ j add an arc from s j,k to s i,k+1 . Let label the new arc. Next, add a fail node to S and for any ∈ L U and any s i,k , if (s i , , p) ∈ T for every p ∈ S, add an arc labeled from s i,k to fail. Finally, let s 0,0 be the root and discard any node not reachable from it. Clearly, D is acyclic. Also, s 0 σ ⇒ s i with |σ| = k ≤ mn if and only if there is a path labeled σ from s 0,0 to some s i,r in D.
For I as in Figure 4(b), α leads from q 0 to q 2 . So that IUT does not pass the test purpose induced by α and so I ioco S does not hold, as expected again. 2 A careful observation of the construction at Proposition 10, with minor adjustments, reveals that one can get m-ioco-complete fault models whose test purposes are deterministic, output-deterministic, input-enabled and acyclic, except for self-loops at pass and fail states.
Proposition 11 Let S ∈ IO(L I , L U ) be deterministic, and m ≥ 1. Then, there is a fault model T P which is ioco-complete for S relatively to IO(L I , L U )[m], and such that all test purposes in T P are deterministic, input-enabled, output-deterministic, and acyclic except for self-loops at fail and pass states.
Proof By Proposition 10 we get T P m-ioco-complete for S, with all T ∈ T P acyclic and deterministic.
Fix T = S, t 0 , L U , L I , T ∈ T P . In order to secure input-enabledness, add a pass state to S. For all ∈ L U , all t = fail, if (t, , t ) ∈ T for any t ∈ S, add (t, , pass) to T . Also, add transitions (pass, , pass) and (fail, , fail) to T , for all ∈ L U . In the end, we get T which is deterministic, input-enabled and acyclic, except for self-loops at pass and fail. . By so adjusting all T ∈ T P we get T P which is m-ioco-complete for S, and such that all T ∈ T P is deterministic, input-enabled and acyclic, except for self-loops at pass and fail.
Let T ∈ T P as in Proposition 10, and T ∈ T P obtained from T as in the previous paragraph. Fix some a ∈ L I . Let t be a state in T, with fail = t = pass. We have at exactly one transition (t, , t ) in T, and on passing to T we added no transitions on symbols from L I . Thus, either there is exactly one transition (t, , t ) in T with ∈ L I , or there is none. If there is none, add (t, a, pass) to T , so that now t is output-deterministic. Apply this transformation to all states in T and complete the transformation by adding (fail, a, fail) and (pass, a, pass) to T , yielding a new test purpose T . Clearly, T is deterministic, input-complete, output-deterministic, and acyclic except for self-loops at fail and pass. Further, it is easy to see that I passes T if and only if I passes T , for all I ∈ IO(L I , L U ). Apply this transformation to all T ∈ T P to get T P . Then T P is m-ioco-complete for S, because T P is.   The next example illustrates the construction. Note that the quiescent δ symbols were not included to keep the example uncluttered. They offer no extra difficulty since they are treated as any plain symbol in L U .

Example 14
The outer path from s 0,0 to s 3,3 to fail in Figure 8 represents a test purpose extracted according to Proposition 10 and already illustrated in Example 13.
Recall that L U = {x} and L I = {a, b}. Following Proposition 11, in order to secure input-enabledness, we added a new pass state. Proceeding, for all states s i,j for which there were no outgoing transition on x, we added transitions (s i,j , x, pass). We completed this step by adding (fail, x, fail) and (pass, x, pass). Next, we fixed a ∈ L I , and for all states s i,j for which there were no outgoing transition on a symbol of {a, b} we added transitions (s i,j , a, pass). Adding the transitions (pass, a, pass) and (fail, a, fail) completed the construction. The test purpose depicted in Figure 8 is easily seen to be deterministic, input-enabled, output-deterministic and acyclic, except by self-loops at fail and pass.
Fix the IUT I in Figure 4(b), and recall that T at Figure 8 was constructed from the original specification S in Figure 4(a). In T, the shortest path from s 0,0 to fail is σ = axba 2 b 2 x. We also see that σ ∈ otr(I), so that in the cross-product T × I we get (s 0,0 , q 0 ) σ ⇒ (fail, q 2 ). This shows that I ioco S does not hold, and T is sufficient to reach a negative verdict. Clearly, from this verdict alone we can not conclude that {T} is 4-complete for S.
As another illustration, Figure 9 shows T , the result obtained after extracting another test purpose from the same multi-graph D in Example 13, and transforming it according to Proposition 11. The set of paths from s 0,0 to fail is P = {a 2 b 2 axbabx n : n ≥ 1}. For the same IUT I in Figure 4(b) we see that P ∩otr(I) = ∅, so that we do not have (s 0,0 , q 0 ) σ ⇒ (fail, q) for any state q of I and any σ ∈ P . Since I ioco S does not hold, we conclude that {T } is not 4-ioco-complete for S. 2 These partial results show that we effectively construct test purposes that satisfy all requirements listed by Tretmans [5] when testing black-box IUTs.
Theorem 15 Let S ∈ IO(L I , L U ) and m ≥ 1. We can effectively construct a finite fault model T P which is m-ioco-complete for S, with all test purposes in T P deterministic, input-enabled, output-deterministic, and acyclic except for self-loops at fail and pass states.

Proof From Propositions 5 and 11. 2
It is not hard to see that all our models given by Theorem 15 satisfy all restrictions that must be obeyed by test cases in [5], Definition 10. A detailed, step by step argument can be seen in Appendix 8, specially Proposition 24. Also, if one has a different set of characteristics, stemming from another kind of testing architecture, and with somewhat different requirements to be satisfied by test purposes as compared to those proposed by Tretmans [5], one might try to proceed as discussed here, and transform each basic test purpose so as to make it adhere to that specific set of new requirements.  Figure 10: A simple IOLTS that whose fault model is super-polynomial.

On the Complexity of Test Purposes
We look at the complexity of the specific family of test purposes constructed in Subsections 5.1 and 5.2. Let S = S, s 0 , L I , L U , T be a deterministic specification, with |S| = n. Consider the acyclic multi-graph D, described in the proof of Proposition 10, and that was used to obtain acyclic test purposes that are mioco-complete for S. Since S is deterministic, it is clear that D is also deterministic and acyclic. Moreover, since D has nm + 1 levels with at most n nodes per level, we conclude that D has at most n 2 m + n nodes. Although the number of nodes and levels in D are polynomial on n and m, the number of traces in D might be super-polynomial on n and m, in general. Since we extract the test purposes from such traces, the fault model that is so generated might also be of super-polynomial size on n and m. We argue that, in general, this is unavoidable, even when we restrict all models to the smaller class of deterministic input-enabled IOLTS models. Proof Clearly, S is deterministic and input-enabled. Let R = (0 + 11) . Define σ = 1 when σ = 0, and let σ = 0 when σ = 1. Let α = y 1 . . . y r ∈ R, with 1 ≤ r ≤ m − 3. It is clear that α ∈ otr(S), αx ∈ otr(S)L U , and αx ∈ otr(S), and so αx ∈ otr(S). Now let I α = S I , q o , L I , L U , T I be an IUT given by (q i−1 , y i , q i ) for 1 ≤ i ≤ r, and (q r , x, q r+1 ). For input-enabledness, add a new pass state, the transitions (q i−1 , y i , pass) (1 ≤ i ≤ r), as well as (q r , σ, pass) and the self-loops (q r+1 , σ, q r+1 ), (pass, σ, pass), for σ ∈ {0, 1}. See Figure 11. Clearly, I α has r + 3 ≤ m states, is deterministic and input-enabled.
Assume that T P is a fault model which is m-ioco-complete for S, relatively to the class of all deterministic input-enabled IUTs. Since αx ∈ otr(I), we get otr(I α ) ∩ otr(S) ∩ (otr(S)L U ) = ∅, so Corollary 3 says that I α ioco S does not hold. Hence, t 0 σ ⇒ fail in T α and q 0 σ ⇒ q in I α for some q ∈ S I , σ ∈ (L i ∪ L U ) . If σ ∈ {0, 1} we get s 0 σ ⇒ s and then (t 0 , s 0 ) σ ⇒ (fail, s) in T α × S. and S does not pass T P . Since T P is m-ioco-complete for S, it must be that S ioco S does not hold, a clear contradiction. Hence σ ∈ {0, 1} . Thus, since σ ∈ otr(I α ), we must have σ = αxα with x ∈ L I and α ∈ {0, 1} .
Consider words of length m in R.
are at least Φ m / √ 5 distinct elements in T P . For later reference, note that both transitions from q r to pass, as well as both self-loops at q r+1 , were never necessary in the proof. Their only function here is to make states q r and q r+1 input-enabled. 2 It is clear that Theorem 16 applies to any specification S in which Figure 10 is a sub-model.

Another Restricted Class of IOLTS Models
As another illustration of our approach, in this section we look at a subclass of IOLTS models that were studied more recently [10]. In that work, Simão and Petrenko considered a more contrived subclass of IOLTS models, and showed that it is possible to generate ioco-complete test suites for specifications in that subclass, although they did not study the complexity of the test suites that were generated. In this section we show how to construct ioco-complete test suites for that same subclass of IOLTS models in a more unified and direct way, and we also look at the complexity of the test suites that are generated using our approach. As one of the main results of this section, we also establish a precise exponential lower bound on the worst case asymptotic size of any test suite that is required to be complete for the class of IOLTS models treated here.
Since there are several restrictions that IOLTS models must satisfy in Simão and Petrenko's approach [10], we introduce them in stages, as needed. Motivation for considering these restrictions can be found in their work [10]. Recall Definitions 13 and 21. First we use the inp and out functions that collect inbound and outbound transitions, respectively, in order to characterize the notions of input-complete and outputcomplete states, among others. Also we need the notion of init, where init( We say that the IOLTS S is single-input, input-complete, or output-complete if all states in S are, respectively, single-input, input-complete, or output-complete. We also say that S is initially-connected if every state in S is reachable from the initial state, and we say that S is progressive if it has no sink state and for any cycle q 0 x k → q k , with q 0 = q k we have x j ∈ L I for some 1 ≤ j ≤ k. Let IOIP(L I , L U ) ⊆ IO(L I , L U ) denote the class of all IOLTSs which are deterministic, input-complete, progressive and initially-connected.
Note the difference in the notions of input-completeness that appear in Definitions 23 and 21.
In Definition 14, the terms test case and test suite refer to words and languages over L I ∪L U , respectively. To avoid confusion in this section we will use the terms schemes and scheme suites, respectively, when referring to words and languages, as in Definition 3 of Simão and Petrenko [10]. Note that, contrary to the notion of a test purpose in Definition 18, test schemes in Simão and Petrenko's approach do not have their sets of input and output symbols reversed with respect to the sets of input and output symbols of specifications and IUTs.
Definition 24 Let L I and L U be sets with L I ∩ L U = ∅ and L = L I ∪ L U . A scheme over L is an acyclic single-input and output-complete IOLTS T ∈ IO(L I , L U ) which has a single sink state, designated fail. A scheme suite SS over L is a finite set of schemes over L.
Proceeding, recall Definition 19, of the cross-product operator S × I. Simão and Petrenko [10] denote the exactly same operator by S ∩ I, with the proviso that internal τ -moves are not considered. In this section we will continue to use S × I to denote synchronous execution. We also remark now that Definition 20, saying when an IUT I passes a scheme suite SS, exactly matches Definition 4 of Simão and Petrenko. So, we also have the same notion of ioco-completeness, as stated in our Definition 20 and their Definition 4. In what follows we want to show that our approach can also be used to construct ioco-complete scheme suites, but with the advantage that we do not need to further constrain specification and IUT models.
Theorem 17 Let S ∈ IO(L I , L U ) with L = L I ∪ L U , and let m ≥ 1. We can effectively construct a scheme suite SS over L which is m-ioco-complete for S.
Proof If needed, use Proposition 5 to transform S into an equivalent deterministic IOLTS. From Proposition 10 we get a scheme suite SS which is ioco-complete for S relatively to IO(L I , L U )[m], and such that all schemes in SS are deterministic and acyclic, and have a single fail state, which is also a sink.
Let T = S T , t 0 , L I , L U , R T in SS and let s ∈ S T . From that proof, we know that there is at most one transition (s, , p) in R T , for any ∈ L U ∪ L I and any p ∈ S T . If ∈ L I , from Definition 23, we get | inp(s)| ≤ 1 and out(s) = ∅, and so s is single-input and output-complete. If ∈ L U then s is already single-input. If |L U | = 1, then s is also output-complete. Else, we construct T by adding a new pass state and, for any x ∈ L U with x = , we add (s, x, pass) to R T . Clearly, s is now output-complete in T .
Fix any IUT I = S I , q 0 , L I , L U , R I ∈ IO(L I , L U ). Since pass is a sink in T , we get (t 0 , q 0 ) ⇒ (fail, q) in T × I if and only if (t 0 , q 0 ) ⇒ (fail, q) in T × I. Thus, I passes T if and only if I passes T .
Apply transformation to all T in SS to get a new scheme suite SS . Since SS is m-ioco-complete for S then so is SS . Clearly, SS satisfies all the requirements in Definition 24.  Simão and Petrenko [10] consider specifications and IUTs that are further restricted to be input-stateminimal, in the sense that any two distinct input-states are always distinguishable. In their work, two states r and p are said to be distinguishable when there are no sink state in the cross-product S /r × S /p , where S /p is the as S, but now with p being the initial state.

Definition 25 ([10])
Let S = S S , s 0 , L I , L U , R S and Q = S Q , q 0 , L I , L U , R Q , and let s ∈ S S , q ∈ S Q . We say that s and q are distinguishable if there is a sink state in S /s × Q /q , else we say that they are compatible. We say that S is input-state-minimal if any two distinct input-states r, s ∈ S S are distinguishable. Also, IOMIN(L I , L U ) ⊆ IOIP(L I , L U ) denotes the subclass of all models which are input-state-minimal.

Recall Definition 23.
IUTs are yet further constrained by Simão and Petrenko [10] to have at most as many input-states as the specification. Let k ≥ 1, and let IMP(L I , L U ) ⊆ IO(L I , L U ). We denote by IMP(L I , L U , k) the subclass of IMP(L I , L U ) comprised by all models with at most k input-states. The main result in [10] is their Theorem 1, which shows that for any S ∈ IOMIN(L I , L U ) we can construct scheme suites that are ioco-complete for IUTs in IOMIN(L I , L U , k), where k is the number of input-states in S. This result also follows easily from Theorem 17.
Corollary 7 Let m ≥ 1 and S ∈ IOMIN(L I , L U ) with k ≥ 0 input-states. We can effectively construct a finite scheme suite SS over L I ∪ L U which is m-ioco-complete for S relatively to the sub-class IOMIN(L I , L U , k).  The complexity of the generated test suites were not analyzed by Simão and Petrenko [10]. However, as we argued in Subsection 5.3 and in Theorem 16, we cannot, in general, avoid scheme suites to asymptotically grow very large, even when specifications are confined to the subclass IOMIN(L I , L U ), and IUTs are restricted to the subclass IOMIN(L I , L U , k), where k is the number of input-states in S. The next result establishes a worst case exponential asymptotic lower bound on the size of the test schemes that can be generated using their Theorem 1 of [10] or, equivalently, using our Corollary 7. Proof We argue almost exactly as that in the proof of Theorem 16, with a few adjustments to appear later on. Note that S, as used in the proof of Theorem 16 and depicted in Figure 10, is deterministic, inputcomplete, progressive and initially-connected, that is, S ∈ IOIP(L I , L U ). Also, the IUT I α , constructed in that proof and illustrated in Figure 11, is also deterministic and in the class IOIP(L I , L U ). Recall that we write y = 0 when y = 1 and y = 1 when y = 0. The argument now proceeds just as in the proof of Theorem 16, and we get a scheme T α = S α , t 0 , L I , L U , T α in SS and that I α does not pass T α . We continue as in Theorem 16. Now we have (t 1 , x 1 , t 2 ) and (t 1 , x 2 , t 2 ) in T α , with x 1 = x 2 , and x 1 , x 2 ∈ {0, 1} = L I . The input alphabet for T α is L I = {0, 1} but, according to Definition 24, any scheme must be single-input. This forces x 1 = x 2 and we reach a contradiction again, as in the proof of Theorem 16. As, before, this will imply T α = T β when α, β ∈ R, and α = β, with |α| = |β| = r ≤ m − 3. From this point on, the argument follows the proof of Theorem 16, establishing that SS must be of size Ω(Φ m ).
We would be done if S ∈ IOMIN(L I , L U ) and I α ∈ IOMIN(L I , L U , k)[m], where k is the number of inputstates in S. We will now extend Figures 10 and 11 to meet these conditions, while preserving the validity of the previous argument. First note that S has 3 input-states, whereas I α has r + 3 ≤ (m − 3) + 3 = m ≤ k input-states. Next, extend the specification in Figure 10 as in Figure 12, with states s 3 , . . . , s k .
State s 0 is repeated to avoid the clutter. Note that transitions on 0 and 1 out of states s 0 and s 1 , as well as the transition on x out of state s 1 were not touched, so that the argument above is still valid when we consider this new specification.
Call this new specification S . States s i , 0 ≤ i ≤ k − 1, are the input-states, so S has k input-states. It is also easy to check that S is deterministic, input-complete and initially-connected. Moreover, S is also progressive, since any cycle in S must go through a transition on an input. In order to assert that S is in IOMIN(L I , L U ), we need to check that any two input-states in Figure 12 are distinguishable. As indicated therein, states can be partitioned into blocks B 1 and B 2 . Consider two distinct states s i , s j ∈ B 2 with 2 ≤ i < j ≤ k, and let w = 0 k−j . We see that (s i , s j ) w ⇒ (s , s k ) where = k − (j − i) and 2 ≤ i ≤ ≤ k − 1. Since init(s )∩ init(s k ) = ∅, any two distinct states in B 2 are distinguishable. Then, since (s 0 , s 1 ) a ⇒ (s 2 , s 3 ), it follows that s 0 and s 1 are also distinguishable.
We now argue that s 0 and s 1 are distinguishable from any state Since we already know that s 0 is distinguishable from s k , we conclude that s 0 is distinguishable from any state in B 2 . Likewise, with w = 0 k−i x we see that (s 1 , s i ) w ⇒ (s 2 , s k ), so that s 1 is also distinguishable from any state in B 2 . Hence, any pair of states in B 1 × B 2 are distinguishable, and we conclude that any two distinct states in B 1 ∪ B 2 are distinguishable, that is, S is input-state-minimal. So, S ∈ IOMIN(L I , L U ) with k input-states, as desired.
We now turn to the IUT. In the proof of Theorem 16, we noted that both transitions from state q r to state pass, together with the self-loops at state q r+1 could have been removed, with no prejudice to the argument given therein. See Figure 13, which we also designate by I α . From the proof of Theorem 16 we have 1 ≤ r ≤ m − 3. By inspection, we see that I α is deterministic, input-complete, progressive, initiallyconnected, and has r + 3 ≤ (m − 3) + 3 = m ≤ k states. Also, all states, except for q r and q r+1 , are input-states, so that I has at most k input-states, as needed.
Finally, we show that every pair of distinct input-states of I α are distinguishable. Fix some q j , 0 ≤ j ≤ r − 1, and define w = y j+1 · · · y r . Clearly, q j w ⇒ q r and, since pass w ⇒ pass, we get (q j , pass) w ⇒ (q r , pass). Since init(q r ) ∩ init(pass) = ∅ we conclude that pass is distinguishable from any state q j , 0 ≤ j ≤ r − 1. Lastly, take a state q i distinct from q j with 0 ≤ i < j ≤ r − 1. Now we get q i w ⇒ q where = r − j + i = r − (j − i) ≤ r − 1. Hence, (q i , q j ) w ⇒ (q , q r ) and, because ≤ r − 1, we see that init(q ) ∩ init(q r ) = ∅, thus proving that q i and q j are also distinguishable. Putting it together, we see that any pair of distinct input-states of I α are distinguishable, that is, I α is input-state-minimal. 2 Theorem 18 clearly also applies to any specification S in which the model depicted in Figure 12 appears as a sub-model with state s 0 being reachable from the initial state of S. This is in contrast to Theorem 9 which says that, for a deterministic S over L, we have an algorithm of time complexity O(km) for checking m-ioco-completeness, where k = n S n L , n L = |L| and n S is the number of states in S, and IUTs have at most m states.
We also remark that in Theorem 1 of Simão and Petrenko [10], implementations are further restricted to be "input-eager", although they do not precisely define this notion in that text. On the other hand, in none of their proofs the input-eager hypothesis explicitly used, leading us to infer that constraining implementations to also be input-eager is a practical consideration to render the testing process more controllable. Since inputeagerness is not strictly necessary to establish their Theorem 1, we conclude that Theorem 18 expresses a valid worst case exponential asymptotic lower bound on the size of the test suites claimed by Theorem 1 in [10].

Related Works
IOLTS models are largely used to describe the syntax and the semantics of systems where input and output actions can occur asynchronously, thus capturing a wide class of systems and communication protocols. Several works have studied different aspects of (complete) test suite generation for families of IOLTS models, under various conformance relations. We comment below on some works that are more closely related to our study.
de Vries and Tretmans [14] presented an ioco-based testing theory to obtain e-complete test suites. This variant of test suite completeness is based on specific test purposes that share particular properties related to certain testing goals, and they consider only observable behaviors based on some objective criteria when testing black-box IUTs. It turns out that such specific test purposes somewhat limit the fault coverage spectrum, e.g., producing inconclusive verdicts. Large, even infinite, test suites can be produced by their test generation method. Some test selection criteria need to be used to avoid this problem, at least when applied in practical situations. On the other hand, our approach allows for a wider class of IOLTS models, and a low degree polynomial time algorithm are devised for efficiently testing ioco-conformance in practical applications.
Petrenko et al. [15] studied IOLTS-testing strategies considering IUTs that cannot block inputs, and also testers that can never prevent an IUT from producing outputs. This scenario calls for input and output communication buffers that can be used for the exchange of symbols between the tester and the IUTs. This leads to an entirely different class of testing strategies, where arbitrarily large buffer memories (queues) are allowed.
Tretmans [5] studied the classic ioco-conformance relation for IOLTS models, and developed the foundations of an ioco-based testing theory for these models [16], where IUTs are treated as black-boxes, and with a testing architecture where the tester, having no access to the internal structure of IUTs, is seen as an artificial environment that drives the exchange of input and output symbols with the IUT. In this case, some restrictions must be observed by the specification, the IUT and the tester models, such as input-completeness and output-determinism. The algorithms developed therein, however, may in general lead to infinite test suites, making it more difficult to devise solutions for practical applications. In our work we described a method that, considering the exact same restrictions to the IOLTS models, does in fact generate finite sets of test purposes that can be used in practical situations. In rare situations, the algorithm may lead to exponential sized testers. If the same restrictions are to be obeyed by the specification, the IUT and the tester IOLTS models, we established an exponential worst case asymptotic lower bound on the size of the testers. This shows with those restrictions in order, generating exponential sized testers is, in unavoidable general, being rather an intrinsic to the problem when one requires ioco-completeness.
Simão and Petrenko [10] also described an approach to generate finite ioco-complete test suites for a class of IOLTS models. They, however, also imposed a number of restrictions on the specification and IUT models in order to obtain ioco-complete finite test suites. They assumed the test purposes to be singleinput and also output-complete. Moreover, specifications and IUTs must be input-complete, progressive, and initially-connected, so further restricting the class of IOLTS models that can be tested according to their fault model. They also did not study the complexity of their method for generating ioco-complete test suites under those restrictions. In contrast, we applied our approach to a testing architecture that satisfies the same restrictions, and showed how to generate ioco-complete test suites in a more straightforward manner. Further, we examined the complexity of the problem of generating ioco-complete test suites under the same restrictions, and established an exponential worst case asymptotic lower bound on the size of any ioco-complete test suite that can be generated in this situation.
Noroozi et al. [17] presented a polynomial time reduction from a variation of the SAT problem to the problem of checking ioco-completeness, thus establishing that, under very general assumptions about the IOLTS models -including non-determinism, -that checking ioco-completeness is a PSPACE-complete problem. In a more restricted scenario, treating only deterministic and input-enabled IOLTS models, they proposed a polynomial time algorithm, based on a simulation-like preorder relation. This is the same complexity bound that our method attains but, in contrast, our approach treats a wider class of conformance relations not being restricted to ioco-conformance only. In another work, Noroozi et al. [18] also studied the problem of synchronous and asynchronous conformance testing, when allowing communication channels as auxiliary memories. They treated a more restricted class of IOLTS, the so-called Internal Choice IOLTSs, where quiescent states must be also input-enabled. The notions of ioco-conformance and of traces are also more restricted. When the structure of IUTs are accessible, algorithms to generate test cases are shown to be sound and exhaustive. However, in a setting where IUTs are black-boxes these algorithms are not applicable, thus limiting their practical use.
Roehm et al. [19], in a more recent work, introduced a variation of conformance testing, related to safety properties. Despite being a weaker relation than trace-inclusion conformance, it allows for tunning a tradeoff between accuracy and computational complexity when checking conformance of hybrid systems. Instead of verifying the whole system, their approach searches for counter-examples. They also proposed a test selection algorithm that uses a coverage measure to reduce the number of test cases. However, since the models are hybrid, the continuous flow of time forces discretizations to reduce the test generation problem to discrete models. This imposes a trade-off between accuracy and computational load, which must be tuned by appropriate choices related to some over-approximations.
Other works have considered ioco-based testing for compositional systems, where components of a more complex system can be formally tested using composition operators to capture the resulting behavior of multiple components. Benes et al. [20] have proposed merge and quotient operators in order to check consistency of more complex parts of systems under test. Following a similar line, Daca et al. [21] proposed compositional operators, friendly composition and hiding, applied to an ioco-testing theory in order to minimize the integration of testing efforts. The result of the friendly composition is an overall specification that integrates the component specifications while pruning away any inputs that lead to incompatible interactions between the components. The friendly hiding operation can prune inputs that lead to states which are ambiguous with respect to underspecified parts of the system. In a similar vein, Frantzen and Tretmans [22] presented a method when complete behaviors of components of a system are not available, and applied a parallel operator when integrating different components. They proposed a specific conformance relation for the components and devised an algorithm that constructs complete test suites.

Conclusions
Conformance of an IUT to a specification often needs to be checked, in order to establish a mathematical guarantee of correctness of the IUTs. The ioco framework has been the conformance relation of choice for verifying IOLTS models in several testing architectures.
We addressed the problem of conformance testing and test case generation for asynchronous systems that can be described using IOLTS models as the base formalism. A new notion of conformance relation was studied, one that is more general and encompasses the classic ioco-conformance. It opened the possibility for a much wider class of conformance relations, all uniformly treated under the same formalism. In particular, it allows for properties or fault models to be specified by formal languages, e.g., regular languages. Very few restrictions over the IOLTS models must be satisfied when generating finite and complete test suites under any notion of conformance that fits the more general setting studied herein. We also proved correct a polynomial algorithm to test general conformance in a "white-box" architecture. With a fixed specification, the algorithm runs in linear time on the size of the IUTs.
Equipped with this new notion of conformance, we specialized the test generation process in order to cover other special cases of conformance relations, such as classical ioco-conformance. Complexity issues related to complete test suite generation for verifying ioco-conformance in settings where the IOLTS models were under several specific restrictions were also discussed. For some sets of restrictions we showed that the state explosion problem cannot be avoided, in general, forcing ioco-complete test suites to grow exponentially with the size of the IUTs. In these cases, we proved correct general algorithms with time complexities that attained such lower bounds, while still generating complete test suites. This indicates that other families of specialized IOLTS models could be considered by our approach, leading to similar results.
Other research areas that might be inspired by these ideas are symbolic test case generation, where data variables and parameters are also present [23,24], as well as conformance relations and generation methods for real-time systems [25,26].

Models with inputs and outputs
Let L = L I ∪ L U with L I ∩ L U = ∅ be alphabets. Then, Definition 10 says that (S, s 0 , L I , L U , T ) is an IOLTS with input alphabet L I and output alphabet L U when (S, s 0 , L, T ) ∈ LTS A (L), that is, an IOLTS is a LTS where the alphabet has been partitioned into disjoint sets of input and output action symbols. In this appendix we will designate the class of IOLTSs over L I and L U by IO A (L I , L U ). Likewise, in [5], Definition 6 says that a labeled transition system with inputs and outputs is a system (S, L I , L U , T, s 0 ), where (S, L, T, s 0 ) ∈ LTS T (L), and we will here denote the class of all such models by LTS T (L I , L U ).  )} to stress that these models were constructed from IOLTSs whose output alphabet did not contain δ (so L −δ U in the notation), but the output alphabet of the extended model always contains δ (so LTS δ T () in the notation.) If the original model was from the class LTSR T (L I , L −δ U ), meaning that all states are reachable from the initial state, then the new class of models will be designated by LTSR δ T (L I , L −δ U ). Next proposition states a simple result. Proof Immediate from the definitions. 2 Now we turn to Definition 17 where quiescence in δ-IOLTS models is introduced in this work. To emphasize that δ is always a symbol in the output alphabet of an δ-IOLTS, in this appendix we designate this class of models by IO δ A (L I , L +δ U ) (in the main text it was designated simply by IO δ (L I , L U )). Let S = (S, s 0 , L I , L +δ U , T ) ∈ IO δ A (L I , L +δ U ) be a δ-IOLTS. If q ∈ S is quiescent according to Definition 17, we write δ S A (q), and may omit the index S when no confusion can arise. Proposition 17 is important because it gives the exact relationship between the class of all δ-IOLTS models and the class of all models after they are extended in order to include quiescence, as defined in [5].

The out relation
In [5], Definition 11 introduces the out relation, here denoted out T , thus: Let S = S, s 0 , L I , L U , T ∈ LTS T (L I , L U ). Then, for all s ∈ S and all Q ⊆ S, out T (s) = {x ∈ L U |s x →} ∪ {δ|δ S T (s)}, and out T (Q) = {out T (s)|s ∈ Q}.
In this work, we define the same relation, denoted out A , as follows: Let S = S, s 0 , L I , L U , T ∈ IO A (L I , L U ). Then, for all Q ⊆ S out A (Q) = s∈Q {x ∈ L U | s x ⇒}.
See Definition 13. Next we show that both definitions of out coincide.

Proposition 19
Let S = (S, s 0 , L I , L +δ U , T ) ∈ IO δ A (L I , L +δ U ) be an IOLTS. Then, we have that out A (Q) = out T (Q) for all Q ⊆ S.
Proof We show that out A (s) = out T (s) for all s ∈ S.
From Proposition 12 we have ⇒ T = ⇒ A , and so the indexes may be omitted. Likewise, we can write → instead of → T or → A .
First assume x ∈ out A (s) and x ∈ out T (s). Since S ∈ IO δ A (L I , L +δ U ) it is clear that S ∈ IO A (L I , L +δ U ). Hence, from the definition of x ∈ out A (s) we obtain s x ⇒ and x ∈ L +δ U . So, we also have s x →. From Proposition 17(1) we have IO δ A (L I , L +δ U ) LTS δ T (L I , L −δ U ). Hence, from Proposition 16 we get S ∈ LTS T (L I , L +δ U ). Since s x → and x ∈ L +δ U , from the definition of out T we obtain x ∈ out T (s), contradicting x ∈ out T (s). Then out A (s) ⊆ out T (s). Now assume that x ∈ out T (s) and x ∈ out A (s). Since S ∈ LTS T (L I , L +δ U ), from x ∈ out T (s) we get: (i) δ S T (s), or (ii) s x → and x ∈ L +δ U . First assume (i). From the definition of δ S T (·) we know that for all x ∈ L I ∪ L +δ U ∪ {τ }, if (s, x, p) ∈ T then x ∈ L +δ U ∪ {τ }, i.e., we must have x ∈ L I . Since S ∈ IO δ A (L I , L +δ U ), Definition 17 for δ-IOLTS models implies s = p and (s, δ, s) ∈ T . But such transition contradicts δ ∈ L +δ U ∪ {τ }. Then hypothesis (i) does not hold. Next, assuming (ii) we must have s x → and x ∈ L +δ U . Thus s x ⇒ and x ∈ L +δ U . Since S ∈ IO A (L I , L +δ U ), the definition of out A results in x ∈ out A (s), contradicting x ∈ out A (s). Hence hypothesis (ii) does not hold. We conclude that out T ⊆ out A . 2

Input-enabledness property
In [5], Definition 7 says that the class of all input-output transition systems with inputs in L I and outputs in L U is a restricted subclass of LTS T (L I ∪ L U ). More specifically, (S, L I , L U , T, s 0 ) ∈ LTS T (L I ∪ L U ) is an input-output transition system if any reachable state s is input-enabled, that is, there is a transition out of s for all input symbols. More formally, in Definition 5(6) we have der T (p) = {q | p ⇒ q}. Then Definition 7 decrees that (S, L I , L U , T, s 0 ) ∈ LTS T (L I ∪ L U ) is an input-output transition system when for all p ∈ der T (s 0 ) we have p a ⇒ for all a ∈ L I . The class of all input-output systems is here denoted by IOTS T (L I , L U ). The subclass of all input-output systems S where all states are reachable from the initial states, that is when S is in the subclass LTSR T (L I , L U ), will be designated by IOTSR T (L I , L U ).
In  So, let y ∈ L −δ I ∪ L +δ U ∪ {τ } with s y →. Since T is deterministic, Proposition 1 says that T has no τ -labeled transitions. Hence, y ∈ L −δ I ∪ L +δ U . If y ∈ L +δ U then y ∈ out A (s) ∪ L +δ U . Now assume that y ∈ L −δ I with s y →, so that y ∈ out A (s). Since | out A (s)| = 1 we have out A (s) = {y}, and again y ∈ out A (s) ∪ L +δ U . We have shown that {y ∈ L −δ I ∪ L +δ U ∪ {τ } | s y →} ⊆ out A (s) ∪ L +δ U . Now let y ∈ out A (s) ∪ L +δ U . If y ∈ out A (s) then y ∈ L −δ I and s y ⇒. Since T has no τ -labeled transitions we get s y →. Hence y ∈ {y ∈ L −δ I ∪ L +δ U ∪ {τ } | s y →}. Now assume y ∈ L +δ U . Since inp A (s) = L +δ U we get y ∈ inp A (s), so that s y ⇒. Because T has no τ -labeled transitions we have s y →. Hence, y ∈ L +δ U and s y → give y ∈ {y ∈ L −δ I ∪ L +δ U ∪ {τ } | s y →}. We now have out A (s) ∪ L +δ U ⊆ {y ∈ L −δ I ∪ L +δ U ∪ {τ } | s y →} and the proof is complete. 2 In Theorem 15 we showed that test purposes satisfying a number of restrictions can be constructed for any given specification. We can now verify that those test purpose models are also test cases, in the sense used in [5].
Proposition 24 Let S be a specification in IO A (L −δ I , L +δ U ), and let m ≥ 1. Then the set T P of test purposes constructed in Theorem 15 is ioco-complete for S with respect to any implementation with at most m states. Moreover, any test purpose in T P satisfies conditions (1), (2 ), (3 ) and (4 ) listed above.
Proof By Theorem 15 we know that T P is m-ioco-complete for S.
Let T = S, s 0 , L +δ U , L −δ I , T ∈ IO A (L +δ U , L −δ I ) be a test purpose constructed in T P . By Theorem 15 we know that T is already deterministic, input-enabled and output-deterministic. So, by Proposition 23 conditions (1) and (4 ) are satisfied.
By Theorem 15, T has two distinct pass and fail states, and it is acyclic except for self-loops at these states. The proof of Theorem 15 starts with test purposes constructed at Proposition 10. A simple examination of the proof of Proposition 10 shows that we explicitly add self-loops (fail, , fail) and (pass, , pass) to T, for all ∈ L +δ U . Hence, conditions (2 ) and (3 ) are also satisfied.