Equivalence of Finite-Valued Streaming String Transducers Is Decidable

LIPICS - Leibniz International Proceedings in Informatics, Jul 2019

In this paper we provide a positive answer to a question left open by Alur and and Deshmukh in 2011 by showing that equivalence of finite-valued copyless streaming string transducers is decidable.

A PDF file should load here. If you do not see its contents the file may be temporarily unavailable at the journal website or you do not have a PDF plug-in installed and enabled in your browser.

Alternatively, you can download the file locally and open with any standalone PDF reader:

http://drops.dagstuhl.de/opus/volltexte/2019/10698/pdf/LIPIcs-ICALP-2019-122.pdf

Equivalence of Finite-Valued Streaming String Transducers Is Decidable

I C A L P Equivalence of Finite-Valued Streaming String Transducers Is Decidable Anca Muscholl 0 1 Gabriele Puppis CNRS 0 1 LaBRI 0 1 Bordeaux 0 1 France 0 1 0 LaBRI, University of Bordeaux , France 1 Category Track B: Automata , Logic, Semantics, and Theory of Programming In this paper we provide a positive answer to a question left open by Alur and and Deshmukh in 2011 by showing that equivalence of finite-valued copyless streaming string transducers is decidable. 2012 ACM Subject Classification Theory of computation ? Transducers Related Version A full version of the paper is available at https://arxiv.org/abs/1902.06973. Finite transducers are simple devices that allow to reason about data transformations in an effective, and even efficient way. In their most basic form they transform strings using finite control. Unlike automata, their power heavily depends on various parameters, like non-determinism, the capability of scanning the input several times, or the kind of storage they may use. The oldest transducer model, known as generalized sequential machine, extends finite automata by outputs. Inspired by an approach that applies to arbitrary relational structures [9], logic-based transformations (also called transductions) were considered by Engelfriet and Hoogeboom [12]. They showed that two-way transducers and monadic-second order (MSO) definable transductions are equivalent in the deterministic case (and even if the transduction is single-valued, which is more general than determinism). This equivalence supports thus the notion of ?regular? functions, in the spirit of classical results on regular word languages from automata theory and logics due to B?chi, Elgot, Trakhtenbrot, Rabin, and others. A one-way transducer model that uses write-only registers as additional storage was proposed a few years ago by Alur and Cern? [2], and called streaming string transducer (SST). SST were shown equivalent to two-way transducers and MSO definable transductions in the deterministic setting, and again, even in the single-valued case. In the relational case the picture is less satisfactory, as expressive equivalence is only preserved for SST and non-deterministic MSO transductions [5], which extend the original MSO transductions by existentially quantified monadic parameters. On the other hand, two-way transducers and SST are incomparable in the relational case. Between functions and relations there is however one class of transductions that exhibits a better behavior, and this is the class of finite-valued transductions. Being finite-valued means that there exists some constant k such that every input belonging to the domain has at most k outputs. Finite-valued transductions were intensively studied in the setting of one-way and twoway transducers. For one-way transducers, k-valuedness can be checked in Ptime [17]. In addition, every k-valued one-way transducer can be effectively decomposed into a union of k and phrases String transducers; equivalence; Ehrenfeucht conjecture Introduction 122:2 unambiguous one-way transducers of exponential size [24, 23]. For both two-way transducers and SST, checking k-valuedness is in Pspace. Besides expressiveness, another fundamental question concerning transducers is the equivalence problem, that is, the problem of deciding whether two transducers define the same relation (or the same partial function if we consider the single-valued case). The equivalence problem turns out to be Pspace-complete for deterministic two-way transducers [16], singlevalued two-way transducers, as well as for single-valued SST [5]. For deterministic SST, equivalence is in Pspace [3], but it is open whether this complexity upper bound is optimal. For arbitrary SST, and in fact even for non-deterministic one-way transducers over a unary output alphabet, equivalence is undecidable [13, 18]. The equivalence problem for k-valued one-way transducers was shown to be decidable by Culik and Karhum?ki using an elegant argument based on Ehrenfeucht?s conjecture [10], and the authors noted that the same proof goes through for two-way transducers as well. The decidability status for the equivalence problem for k-valued SST was first stated as an open problem in [5]. Another open problem is whether SST and two-way transducers are equivalent in the finite-valued case, like in the single-valued case. It is worth noting, however, that in the full relational case SST and two-way transducers are incomparable. Concerning this last open question, a partial positive answer was given in [14], by decomposing any finite-valued SST with only one register into a finite union of unambiguous SST. This decomposition result also entails the decidability of the equivalence problem for the considered class. The main result of this paper is a positive answer to the first question left open in [5]: I Theorem 1. The equivalence problem for finite-valued SST is decidable. We show the above result with a proof idea due to Culik and Karhum?ki [10], based on the Ehrenfeucht conjecture. Our proof is much more involved, because SST produce their outputs piece-wise, in contrast to one-way and two-way transducers, that produce output linearly while reading the input. We manage to overcome this obstacle using some (mild) word combinatorics and word equations, by introducing a suitable normalization procedure for SST. We believe that our technique will also allow to solve the second problem left open in [5], which is the expressive equivalence between finite-valued SST and two-way transducers. Related work The equivalence problem for transducers has recently raised interest for more complex types of transducers in the single-valued case: Filiot and Reynier showed that equivalence of copyful, deterministic SST is decidable by showing them equivalent to HDT0L systems and applying [10], which contains the above-mentioned result as a special case. Subsequently, Benedikt et al. showed that equivalence of copyful, deterministic SST has Ackerman complexity, with a proof based on polynomial automata and ultimately on Hilbert?s basis theorem [6]. Interestingly, the use of Hilbert?s basis theorem goes back to the proof of Ehrenfeucht?s conjecture [1, 15]. A similar approach was used by Boiret et al. in [7] to show that bottom-up register automata over unordered forests have a decidable equivalence problem, see also the nice survey [8]. Overview Section 2 introduces the transducer model, then Section 3 sets up the technical machinery that allows to normalize finite-valued SST. Section 4 shows the major normalization result, which holds for left quotients of SST. Finally Section 5 recalls the Ehrenfeucht-based proof for equivalence and the application to finite-valued SST. A full version of the paper is available at https://arxiv.org/abs/1902.06973. 2 Streaming string transducers A streaming string transducer (SST ) is a tuple T = (?, ?, X, Q, U, I, E, F, xout), where ? and ? are finite input and output alphabets, X is a finite set of registers (usually denoted x, x?, x1, x2, etc.), Q is a finite set of states, U is a finite set of register updates, that is, functions from X to (X ? ?)?, I, F ? Q are subsets of states, defining the initial and final states, E ? Q ? ? ? U ? Q is a transition relation, describing, for each state and input symbol, the possible register updates and target states, and finally xout ? X is a register for the output. Note that, compared to the original definition from [2], here we forbid for simplicity the use of final production rules, that perform ad additional register update after the end of the input. This simplification is immaterial with respect to the decidability of the equivalence problem. For example, it can be enforced, without loss of generality, by assuming that all well-formed inputs are terminated by a special marker, say ?, on which the transducer can apply a specific transition. We assume here that all inputs of a transducer are non-empty and of the form u ?, with ? not occurring in u. Copyless restriction and capacity An SST as above is copyless if for all register updates f ? U , every register x ? X appears at most once in the word f (x1) . . . f (xm), where X = {x1, . . . , xm}. For a copyless SST, every output has length at most linear in the length of the input. More precisely, every output associated with an input u has length at most cSuS, where c = maxf?U ?x?X Sf (x)S? is the maximum number of letters that the SST can add to its registers along a single transition (this number c is called capacity of the SST). Hereafter, we assume that all SST are copyless. Register updates and flows Every register update, and in general every function f ? X ? (X ? ?)? is naturally extended to a morphism on (X ? ?)?, by defining it as identity over ?. When reasoning with register updates, it is sometimes possible to abstract away the specific words over ?, and only consider how the contents of the registers flows into other registers. Formally, the flow of an update f ? X ? (X ? ?)? is the bipartite graph that consists of two ordered sequences of nodes, one on the left and one on the right, with each node in a sequence corresponding to a specific register, and arrows that go from the node corresponding to register x to a right node corresponding to register x whenever x occurs in f (x). For example, the flow of the update f defined by f (x1) = a x1 aa x3, f (x2) = b a, and f (x3) = x2 b is the second bipartite graph in the figure on page 5. Note that there are finitely many flows on a fixed number of registers. Moreover, flows can be equipped with a natural composition operation: given two flows F1 and F2, F1 ? F2 is the bipartite graph obtained by glueing the right nodes of F1 with the left nodes of F2, and by shortcutting pairs of consecutive arrows. We call flow monoid of an SST T the monoid of flows generated by the updates of T , with the composition operation as associative product. Transitions, runs, and loops A transition (q, a, f, q?) of an SST T is conveniently denoted by the arrow q ?a?Tf? q?, and ~ the subscript T is often omitted when clear from the context. A run on w = a1 . . . an is a sequence of transitions of the form q0 ??a1?~?f1? q1 ??a2?~?f2? . . . ?a?n?~?fn? qn. Sometimes, a run as above is equally denoted by q0 ?w?~?f qn, so as to highlight the underlying input w and the induced register update f = f1 ? ? ? ? ? fn. A run is initial (resp. final) if it begins with an initial (resp. final) state; it is successful if it is both initial and final. Given two registers x, x? and a run ? ? q ?w?~?f q?, we say that x flows into x? along ? if x occurs in f (x?). Note that this property depends only on the flow of the induced update f . An SST is said to be trimmed is every state occurs in at least one successful run, so every state is reachable from the initial states and co-reachable from the final states. This property can be easily enforced with a polynomial-time preprocessing. When reasoning with automata, it is common practice to use pumping arguments. Pumping will also be used here, but the notion of loop needs to be refined as to take into account the effect of register updates. Formally, a loop of a run ? of an SST is any non-empty factor of ? of the form ? ? q ?w?~?f q, that starts and ends in the same state q, and induces a flow-idempotent update, namely, an update f such that f and f ? f have the same flow. Outputs and finite-valuedness The output of a successful run ? ? q0 ?w?~?f qn is defined as out(?) = (f0 ? f )(xout), where f0(x) = ? for all x ? X. Sometimes, we write out(f ) in place of out(?). The relation realized by an SST is the set of pairs (u, v) ? ?? ? ??, where u is a well-formed input (namely, terminating with ?) and v is the output associated with some successful run on u. An SST is k-valued if for every input u, there are at most k different outputs associated with u. It is single-valued (resp. finite-valued) if it is k-valued for k = 1 (resp. for some k ? N). The domain an SST T , denoted Dom(T ), is the set of input words that have some successful run in T . Two SST T1, T2 are equivalent, denoted as T1 ? T2, if they realize the same relation over ?? ? ??. Register valuations A register valuation is a function from X to ??. Given a successful run ? ? q0 ?a?1~?f1 q1 ?a?2~?f2 . . . a?n?~?fn qn and a position i ? {0, . . . , n} in it, the register valuation at position i in ? is the function val?,i that is defined inductively on i as follows: val?,0(x) = ?, for all x ? X, and val?,i+1 = val?,i ? fi. Note that val?,n(xout) coincides with the final output out(f1 ? ? ? ? ? fn) produced by ?. 3 Normalizations A major stumbling block in deciding equivalence of SST, as well as other crucial problems, lies in the fact that the same output can be produced by very different runs. This phenomenon already appears with much simpler transducers, e.g. with one-way transducers, where runs may produce the same output, but at different speeds. However, the phenomenon is more subtle for SST, as the output is produced piece-wise, and not sequentially: runs with same output may appear to be different in many ways, e.g. in terms of the flows of the register updates, or in terms of shifts of portions of the output. The goal of this section is to provide suitable normalization steps that remove, one at a time, the above mentioned degrees of freedom in producing the same output. Another issue that we will be concerned with is the compatibility of the normalization steps with constructions on transducers that shortcut arbitrary long runs into a single transition. Essentially, we aim at having an effective notion of equivalence w.r.t. final outputs that works not only for transitions but also for runs. Normalization of flows In this section, m will always denote the number of registers of an SST and X = {x1, . . . , xm} the set of registers. It is convenient to equip X with a total order, say x1 < ? ? ? < xm. Accordingly, we let ? = x1 . . . xm be the juxtaposition of all register names, and f (?) = f (x1) . . . f (xm) for every register update f . We say that a register update f is non-erasing if for every register x, f (?) contains at least an occurrence of x (in fact, exactly one, since T is copyless). This can be rephrased as a property of the flow of f , where every node on the left must have an outgoing arrow. In a similar way, we say that f is non-permuting if registers appear in f ? ( ) with their natural order and without jumps, that is, f (?) ? ??x1?? . . . ??xk??, for some k ? m. As before, this can be rephrased by saying that the arrows in the flow of f must not be crossing, and the target nodes to the right must form a prefix of ?. Below are some examples of updates with their flows: the first update f is erasing, the second update g is non-erasing but permuting, and the third update h is non-erasing and non-permuting. f? xxx123 ??? ___ x1 _ x3 _ g? xxx123 ??? ___ xx12 __ x3 _ h? xxx123 ??? ___ xx13 __ x2 _ We say that T is flow-normalized if all its register updates are non-erasing and non-permuting. Note that a flow-normalized SST with m registers can have at most 2m different flows. I Proposition 2. One can transform any SST into an equivalent flow-normalized one. Recall that a register valuation is a function from X to ??. With a flow-normalized SST, one can also define a dual notion of valuation, representing ?gaps? between registers that shrink along the run. For this we introduce m + 1 fresh variables y0, y1, . . . , ym, called gaps. Hereafter, Y = {y0, y1, . . . , ym} will always denote the set of gaps. We use the term valuation to generically denote a register/gap valuation, that is, a function from X ? Y to ??. The idea is that a gap yj represents a word that is inserted between register xj (if j > 0) and register xj+1 (if j < n) so as to form the final output. Formally, given a word w ? ??x1?? . . . ??xk??, with k ? m, and given two registers xi, xj, with i < j, we denote by w?xi, xj? the maximal factor of w strictly between the unique occurrence of xi and the unique occurrence of xj, using the following conventions for the degenerate cases: if i = 0, then w?xi, xj? is a maximal prefix of w; if i > 0 but there is no occurrence of xi, then w?xi, xj? = ?; finally, if there is an occurrence of xi but no occurrence of xj in w, then w?xi, xj? is a maximal suffix. Given a run ? ? q0 ?a?1~?f1 q1 ?a?2~?f2 . . . a?n?~?fn qn and a position i in it, the valuation at position i of ? is the function val?,i ? X ? Y ? ?? such that val?,i restricted to X is the register valuation at position i of ?, val?,i maps every gap yj to the word (fi+1 ? ? ? ? ? fn)(?)?xj, xj+1?. By definition, the image of the word ? = y0 x1 y1 . . . xm ym via the valuation val?,i is always equal to the final output out(?), for all positions i. In this sense, the sequence of valuations val?,0, val?,1, . . . , val?,n can be identified with a sequence of factorizations of out(?). For example, below are the factorizations of the output before and after a transition with register update f such that f (x1) = s x1 u x2 t and f (x2) = v, for s, u, t, v ? ??: y? 0 y0 s x1 x2 y1 u This also suggests the principle that gaps, like registers, are updated along transitions via suitable morphisms, but in a symmetric way, that is, from right to left. For instance, in the above picture, the gaps y0, y1, y2 are updated by the function f ? such that f ?(y0) = y0 s, f ?(y1) = u, and f ?(y2) = t y1 v y2. In general, the function f ?, called gap update, is uniquely determined by the register update f , and vice versa, f is uniquely determined by the gap update f ?. Another perhaps interesting phenomenon is that the gap update f ? is also non-erasing and non-permuting (the notion of non-permuting gap assignment is defined w.r.t. the reverse order ym < ? ? ? < y0). Normalization of states The next normalization step splits the states of an SST in such a way that it becomes possible to associate with each state an over-approximation of the possible register/gap valuations witnessed when the state is visited along a successful run. These over-approximations are very simple languages over the output alphabet ?, e.g. singleton languages like {aba} and periodic languages like {ab}?{a} (often denoted (ab)?a to improve readability). Basically our over-approximations refer to length and period constraints. The period of a word w is the least number 0 < p ? SwS such that w is a prefix of (w[1, p])?. For example, the period of w = abcab is 3. For a given parameter ? ? N we define the family L? that contains: the empty language ?, the singleton languages {u}, with u ? ?? and SuS ? ?, the periodic languages u?v, with u ? ?+ primitive (i.e. u = wk only if k = 1), SuS ? ?, and v ? ?? strict prefix of u, the universal language ??. The languages in L?, partially ordered by containment, form a finite meet semi-lattice, where the meet is the intersection ?. We depict here part of the lattice L? for a parameter ? ? 3: ?? a? ? (ab)? ab ? (ab)?a a aba . . . The semi-lattice structure allows to derive a best over-approximation in L? of any language oLn?si?n?g,letthoant lias:ngLu?a?g=es?.{FLor? ?exLa?m?pLle?, ?foLr}?. =W3e, wwiellhmavoest{lyabuas}e??th=e {aapbpar}o,xi{mabaatiboan}?o?p=er(aatob)r??a?, and {abbb}?? = ??. Note also that if SwS ? ? then w?? = {w}. A useful property is the compatibility of ?? with concatenation, which immediately extends to compatibility with word morphisms: I Lemma 3. (L1 ? L2)?? = (L1?? ? L2??)?? for every ? ? N and L1, L2 ? ??. Recall that X, Y denote, respectively, the sets of registers and gaps of a flow-normalized SST. Given a valuation ? ? X ? Y ? ??, its ?-approximant is the function ??? ? X ? Y ? L? that maps any z ? X ? Y to the language {?(z)}??. The set of ?-approximants is denoted L?X?Y , and consists of all maps from X ? Y to L?. Further let Valq = {val?,i ? ? successful run visiting q at any position i} be the set of possible valuations induced by an arbitrary successful run when visiting state q. A first desirable property is that all valuations in Valq have the same ?-approximant, which is thus determined by the state q. Formally, given a flow-normalized SST T with trimmed state space Q, we say that T admits ?-approximants if every state q ? Q can be effectively annotated with an ?-approximant Aq ? L?X?Y in such a way that ?? ? Valq ? ??? = Aq. (1) This condition is best understood as an invariant on lengths and periods that can be enforced on valuations of registers and gaps when visiting a particular state. For instance, when the approximant Aq guarantees a certain period, then this period will be the same for all valuations occurring at q, independently of the specific initial run that may lead to q (for registers), and of the run that may lead from q to an accepting state (for gaps). The proposition below shows that it is always possible to refine any SST T so as to admit ?-approximants, for any parameter ?. The proof for ?-approximants that concern only registers could be understood as unfolding the SST T , and merging nodes corresponding to any two inputs u and v, with u prefix of v, whenever the induced ?-approximants at u and v are the same for every register x. In general, the resulting SST can be seen a covering of the original SST T , in the sense formalized by Sakarovitch and de Souza in [22]: T ? is a covering of T if the states of T ? can be mapped homomorphically to states of T , while preserving transitions and the distinction into initial and final states, and, moreover, the outgoing transitions of every state of T ? map one-to-one to outgoing transitions of a corresponding state of T . This implies that the successful runs of T and those of T ? are in one-to-one correspondence. I Proposition 4. Let T be a flow-normalized SST, and let ? ? N. One can construct an equivalent flow-normalized SST T ? that admits ?-approximants and that is a covering of T . Notation. Whenever an SST admits ?-approximants Aq as above, it is convenient to denote its states by triples of the form (q, AX , AY ), where AX (resp. AY ) is the restriction of the ?-approximant Aq of state q to registers (resp. gaps). Note that the smaller the parameter ?, the weaker is the property required for ?approximants (in particular, for ? = 0 the lattice L? collapses to ? and ??). Choosing ? to be at least the capacity of the SST is already a reasonable choice, as it gives a nice characterization of equivalence of transitions w.r.t. the produced outputs (cf. Lemma 5 below). However, we will see that it is desirable to have even finer approximants, in such a way that our results will be compatible with left quotients of SST, that shortcut arbitrary long runs into single transitions. We postpone the technical details to Section 4, and only provide a rough intuition underlying the choice of the appropriate parameter ?. We will choose ? much larger than the capacity of the SST, so that, by pumping arguments, one can show that, for every state q and every parameter ? ? ?, the ?-approximant cannot be strictly smaller than the ?-approximant on all valuations from Valq. Normalization of transitions We finally turn to studying a notion of equivalence on transitions that is similar to the twosided Myhill-Nerode equivalence on words. We will only compare transitions that consume the same input letter and link the same pair of states. Instead of using words as two-sided contexts, we will use initial and final runs that can be attached to the considered transitions in order to form successful runs, and instead of comparing membership in a language, we will compare the effect on the produced outputs. Consider two transitions ?1 ? q ?a?~f?1 q? and ?2 ? q ?a?~f?2 q?. We say that ?1 and ?2 are equivalent if for every initial run ? leading to q and every final run ? starting in q?, the outputs out(? ?1 ?) and out(? ?2 ?) are equal. We often refer to (?, ?) as a context for ?1, ?2. In general, two transitions of an SST having the same source, target and ?-label might turn out to be non-equivalent, and still produce the same output within specific contexts. However, Lemma 5 below shows that this is not the case with ?-approximants at hand, provided that ? is at least the capacity of the SST. More precisely, we will show that the equivalence of two a~f1 a~f2 transitions ?1 ? (q, AX , AY ) ??? (q?, A?X , A?Y ) and ?2 ? (q, AX , AY ) ??? (q?, A?X , A?Y ), where f1, f2 have the same flow, only depends on the ?-approximants AX , A?Y that annotate the source and target states. This will imply that ?1, ?2 either always produce the same output or always produce different outputs, independently of the surrounding contexts. To prove the statement, we have to consider register valuations induced by initial runs, and symmetrically gap valuations induced by final runs. It helps to introduce the following: Notation. Given an initial run ?, val?? is the register valuation induced at the end of ?; symmetrically, val?? is the gap valuation induced at the beginning of a final run ?. We also recall a consequence of the flow normalization: the effect on the final output of an update f that occurs in a successful run can be described by a word Effect(f ) over the alphabet X ? Y ? ?, defined as Effect(f ) = y0 f (x1) y1 . . . f (xm) ym+1. Note that in Effect(f ) each register (resp. gap) occurs exactly once, according to the order x1 < ? ? ? < xm (resp. y0 < ? ? ? < ym) ? the occurrences of registers and gaps, however, may not be strictly interleaved. In Effect(f ), an occurrence of xi ? X represents an abstract valuation for register xi before applying the update f , while an occurrence of yj ? Y represents an abstract valuation for the gap yj after applying f . In particular, note that the x?s and the y?s refer to valuations induced at different positions of a run. The maximal factors of Effect(f ) that are entirely over ? represent the words that need to be added in order to get the register valuation after f , or equally the gap valuation before f . For instance, by reusing the example update f from page 5, where f (x1) = s x1 u x2 t and f (x2) = v, the effect of f is described by the word Effect(f ) = y0 s x1 u x2 t y1 v y2, suggestively depicted as Effect(f ) = y0 s x1 u x2 t y1 v y2 (here the lengths of the blocks labeled with variables are immaterial). Note that the yj above are in fact the yj? from the picture at page 5. In the above figure we have also highlighted with dotted rectangles the factors that represent gap valuations before the update (e.g. y0 s), and register valuations after the update (e.g. s x1 u x2 t). Given a valuation ? and two approximants A ? L?X and A? ? LY? , one for register valuations and the other for gap valuations, we write ? ? A ? A? to mean that ?(x) ? A(x) and ?(y) ? A?(y) for all x ? X and y ? Y . I Lemma 5. Let T be a trimmed flow-normalized SST. Given two transitions ?i ? ?q ?a?~f?i q?, with i ? {1, 2}, a context (?, ?) for them, and the ?-approximants A = ((val??)? )SX and A? = ((val??)??)SY , with ? ? N, the following holds: 1. If Effect(f1) = Effect(f2) holds on all valuations ? ? A ? A?, then out(? ?1 ?) = out(? ?2 ?). 2. If ?1, ?2 have the same flow, out(? ?1 ?) = out(? ?2 ?), and ? ? c, where c is the capacity of T , then Effect(f1) = Effect(f2) holds on all valuations ? ? A ? A?. Recall that in an SST that admits ?-approximants, states are of the form (q, AX , AY ), and we have ((val??)??)SX = AX (resp. ((val??)??)SY = AY ) for every initial run ? that ends in (q, AX , AY ) (resp. for every final run ? that starts in (q, AX , AY )). By pairing this with Lemma 5, we immediately obtain the following corollary: I Corollary 6. Let T be a trimmed flow-normalized SST. One can decide in polynomial time whether two given transitions ?1, ?2 of T with the same flow are equivalent. Moreover, if T has capacity c and admits ?-approximants for some ? ? c, then only two cases can happen: 1. either out(? ?1 ?) = out(? ?2 ?) for every context (?, ?) (so ?1, ?2 are equivalent), 2. or out(? ?1 ?) ? out(? ?2 ?) for every context (?, ?) (so ?1, ?2 are not equivalent). Another important consequence is the following theorem, that normalizes finite-valued SST in order to bound the maximum number of transitions linking the same pair of states and consuming the same input letter. This number is called edge ambiguity for short. I Theorem 7. Let T be a k-valued, flow-normalized SST that has m registers, capacity c, and that admits ?-approximants, for some ? ? c. One can construct an equivalent SST T ?, with the same states and the same registers as T , that has edge ambiguity at most k ? 2m. Proof. By Corollary 6, T has at most k pairwise non-equivalent transitions with the same input letter, the same source and target states, and the same flow. Moreover equivalence of such transitions can be decided. We can then normalize T by removing in each equivalence class all but one transitions with the same flow. Since T has at most 2m flows, the normalization results in an equivalent SST T ? with edge ambiguity at most k ? 2m. J The next section is devoted to prove a very similar result as above, but for all SST that can be obtained by shortcutting runs into single transitions, and that thus have arbitrary large capacity. This will be the main technical ingredient for establishing the decidability of the equivalence problem for k-valued SST. 4 Shortcut construction Here we focus on a transformation of relations that absorbs the first input letter when this is equal to a specific element, say a ? ? ? {?}. Such a transformation maps any relation R to the relation Ra = {(u, v) ? (au, v) ? R}. Observe that R = R? ? ?a?? Ra, where R? = R ? ({?} ? ??). It is easy to see that the class of relations realized by SST is effectively closed under the transformation R ? Ra. To prove this closure property, it is convenient to restrict, without loss of generality, to SST with transient initial states, namely, SST where no transition reaches an initial state. Under this assumption, the closure property also preserves the state space (though some states may become useless), the set of registers, the property of being k-valued, as well as the ?-approximants, if they are admitted by the original SST. However, the transformation does not preserves the capacity, which may increase. I Lemma 8. Given a flow-normalized SST T with transient initial states, and given a letter a ? ? ? {?}, one can construct an SST Ta with transient initial states such that Dom(Ta) = {u ? ?? ? au ? Dom(T )}, Ta on input u produces the same outputs as T on input au. Moreover, Ta has the same states and the same registers as T ; if T has capacity c, then Ta has capacity 2c; if T admits ?-approximants, then so does Ta (via the same annotation). The above construction can be applied inductively to compute an SST for any left quotient Ru = {(v, w) ? (u v, w) ? R} of R. For u = a1 . . . an ? (??{?})?, we let Tu = (. . . (Ta1 )a2 . . . )an . The last and most technical step consists in proving that edge ambiguity can be uniformly bounded in every SST Tu, provided that the initial SST T is finite-valued, flow-normalized, and admits ?-approximants for a large enough ?. More precisely, we aim at establishing that, for ? much larger than the capacity of T , the notion of ?-approximant, besides satisfying Equation (1), also satisfies the following property: ?? ? ? ?? ? ? ?? ? ?? ? (val??)? (val??)? ? ? = (val??)? = (val??)??. ? (2) In this case we say that the ?-approximants are tight. Intuitively, the above property can be explained as follows. When considering an SST with states annotated with ?-approximants, it may happen that for some larger parameter ? some initial runs induce register valuations at a state (q, AX , AY ) whose ?-approximants are strictly included in AX (e.g. possibly entailing new periodicities). These runs should be thought of as exceptional cases, and there is a way of pumping them so as to restore the equality between AX and the induced ?-approximant. Let us first see how tight approximants are used. The theorem below assumes that there is an SST T ? with tight approximants (later we will show how to compute such an SST), and bounds the edge ambiguity of the SST Tu? that realizes a left quotient of T ?. I Theorem 9. Let T ? be a k-valued, flow-normalized SST realizing R, with transient initial states and tight ?-approximants. For every u ? ??, one can construct an SST Tu? realizing Ru, with the same states and the same registers as T ?, and with edge ambiguity at most k ? 2m. Proof. The crux is to show that the SST Tu? obtained from Lemma 8 has at most k pairwise non-equivalent transitions with the same flow (for any given source/target state and label). Once this is proven, one can proceed as in the proof of Theorem 7, by removing all but one transition with the same flow in each equivalence class. By way of contradiction, assume that Tu? has k + 1 pairwise non-equivalent transitions ?1, . . . , ?k+1 with the same flow. Since T ? admits tight ?-approximants, by Lemma 8 we know that the source and target state, respectively, of the previous transitions are annotated with tight ?-approximants, say (AX , AY ) and (A?X , A?Y ), respectively. We begin by applying the first claim of Lemma 5, implying that the equation Effect(fi) = Effect(fj) is violated for some valuation ? ? AX ? A?Y . Then, we let ? = max(?, SuSc) a?nd use Equations (1) and (2) to get a context (?, ?) such that (val??)?? = AX and (val??)? = A?Y . Finally, knowing that ? is at least the capacity of Tu? , we apply the second claim of Lemma 5 to get out(? ?i ?) ? out(? ?j ?), thus witnessing non-equivalence of all pairs of transitions ?i, ?j at the same time. This contradicts the assumption that Tu? (and hence T ?) is k-valued. J Now, let T be a flow-normalized SST with m registers, capacity c, and trimmed state space Q. Below, we show how to compute, with the help of Proposition 4, an SST T ? equivalent to T that admits tight approximants. For simplicity, we will mostly focus on register valuations induced by initial runs, even though similar results can be also stated for gap valuations induced by final runs. We begin by giving a few technical results based on pumping arguments. We say that register x is productive along ? if the update induced by ? maps x to a word that contains at least one letter from ?. We also recall that a loop of a run needs to induce a flow-idempotent update. I Lemma 10. If ? = ?1 ? ?2 is an initial run of T , with ? loop, then for every n > 0 the pumped run ?(n) = ?1 ?n ?2 induces valuations val?(n)? mapping any register x to a word of the form u0 v1n?1 u1 . . . v2nm?1 u2m, where u0, . . . , u2m, v1, . . . , v2m ? ?? depend on ? and x, but not on n. Moreover, we have vi ? ? for some i if there is a register x? that is productive along ? and that flows into x along ?2. Given a tuple of pairwise disjoint loops ? = ?1, . . . , ?` in a run ?, we write ?? ?? ? when ?? is obtained from ? by simultaneously pumping n times every loop ?i, for some n > 0. When using this notation, we often omit the subscript ?; in this case we tacitly assume that ? is uniquely determined from ?. In this way, when writing, for instance, ??, ??? ? ?, we will know that ??, ??? are obtained by pumping the same loops of ?. We also say that a property on runs holds for all but finitely many ?? ? ? if it holds on runs ?? that are obtained from ? by pumping n times the loops in a fixed tuple ?, for all n > n0 and for a sufficiently large n0. I Lemma 11. Let ? be an initial run and x a register. If val??(x) has length (resp. period) larger than ? = m c SQS 23?2m , then for every ? ? ? and for all but finitely many ?? ? ?, val???(x) has length (resp. period) larger than ?. Using the previous lemmas and the fact that the type of quantification ?for all but finitely many runs? commutes with conjunctions (e.g. those used to enforce properties on each register x ? X), we obtain that ?-approximants are tight for sufficiently large ?: I Proposition 12. Let T ? be the SST admitting ?-approximants that is obtained from T using Proposition 4, for any ? ? m c SQS 23?2m , where Q is the set of states of T . The ?-approximants of T ? are tight. 5 Equivalence algorithm The equivalence algorithm for k-valued SST follows a classical approach of Culik and Karhum?ki [10] that is based on so-called test sets. A test set for two SST T1, T2 over input alphabet ? is a set F ? ?? such that T1, T2 are equivalent if and only if they are equivalent over F . The main contribution of [10] is to show that finite test sets exist and be computed effectively for k-valued one-way transducers.The key ingredient of their proof is to show the existence of a test set that works for all transducers with fixed number of states. An essential observation is that for k-valued one-way, or even two-way, transducers one can assume that the edge ambiguity is at most k. The reason for this is simply that the output is generated sequentially. For SST the situation is far more complex because the output is generated piecewise. The purpose of the normalizations performed in Section 3 was precisely to restore the property of bounded edge ambiguity. In a nutshell, the existence of a test set for transducers is a consequence of Ehrenfeucht?s conjecture, whereas the effectiveness is based on the resolution of word equations due to Makanin (see e.g. the survey [11]). Ehrenfeucht?s conjecture was originally stated as a conjecture about formal languages: for every language L ? ??, there is a finite subset F ? L such that for all morphisms f, g ? ?? ? ??, f (w) = g(w) for every w ? L if and only if f (w) = g(w) for every w ? F . Such a set F is called a test set for L. There is an equivalent formulation of Ehrenfeucht?s conjecture in terms of a compactness property of word equations [20]. Let ? and ? be two alphabets, where the elements in ? are called unknowns. A word equation is a pair (u, v) ? ?? ? ??, and a solution is a morphism ? ? ?? ? ?? such that ?(u) = ?(v). Ehrenfeucht?s conjecture is equivalent to saying that any system of equations over a finite set ? of unknowns has a finite, equivalent subsystem, where equivalence means that the solution sets are the same. The latter compactness property was proved in [1, 15] by encoding words by polynomials and using Hilbert?s basis theorem. In view of Propositions 2, 4, and 12, we can restrict without loss of generality to SST that are flow-normalized and that admit tight approximants. Hereafter, we shall tacitly assume that all transducers are of this form. Given some integers k, n, m, and e, let Ck(n, m, e) be the class of k-valued SST with at most n states, m registers, and edge-ambiguity at most e. Note that if T is k-valued, then by Theorem 7 it belongs to Ck(n, m, e), where n, m are the number of states and registers of T and e = k ? 2m. Similarly, by Lemma 8 and Theorem 9, every left quotient Tu also belongs to Ck(n, m, e). Now, let us fix k, n, m, e and consider an arbitrary SST T from Ck(n, m, e). Following [10] we first build an abstraction of T by replacing each maximal factor from ?? occurring in some update function of T , by a distinct unknown from ?. The SST ?(T ) obtained in this way is called a schema; its outputs are words over ?. Note that the assumption of bounded edge ambiguity is essential here to get a uniform bound on the number of unknowns required for a schema. Clearly, there are only finitely many schemas of SST in Ck(n, m, e). We denote by ?T ? ? ? ?? the partial mapping (concretization) that associates with each unknown the corresponding word from ?? as specified by the updates of T . We can rephrase the equivalence T1 ? T2 of two arbitrary SST from Ck(n, m, e) as an infinite ?system? of word equations1 S = ?u??? ?? S? over set of unknowns ? ? ??. The unknowns from ? are used for the schema ?(T1), whereas those from ?? are used for ?(T2); in particular, ?T1 ? ? ? ?? and ?T2 ? ?? ? ??. The disjunctions in S are finite, with ? ranging over the possible schemas ?1, ?2 (for T1 and T2, respectively) and the possible partitions of the set of runs of ?1 and ?2 over the input u, into at most k groups (one for each possible output). Finally, S? is a (finite) system of word equations, stating the equality of the words from ?? ? ??? that belong to the same group according to ?. The following lemma was stated in [10] for k-valued one-way transducers, but it holds as well for two-way transducers and for SST (even copyful, with a proper definition for Ck(n, m, e)): I Lemma 13. Given two SST T1, T2 from Ck(n, m, e), the system S = ?u??? ?? S? has ?T1 ? ?T2 as solution if and only if T1 ? T2. As shown in [10], the Ehrenfeucht conjecture can be used to show that any infinite system S as in Lemma 13 is equivalent to some finite sub-system SN = ?u???N ?? S?. This gives: I Lemma 14. Given n, m, e ? N, there is N ? N such that ??N is a test set for every pair of SST T1, T2 from Ck(n, m, e). Using Theorem 7 and Lemma 14 we can derive immediately the existence of a finite test set for any two k-valued SST. The last question is how to compute such a test set effectively. For this we will use the shortcut construction provided in Section 4. 1 Formally, S depends on n and k, but for simplicity we leave out the indices. I Lemma 15. Assume that the formulas SN and SN+1 are equivalent, i.e., they have the same solutions. Then ??N is a test set for any pair of SST from Ck(n, m, e). Proof. Let T1 ?r T2 denote equivalence of T1 and T2 relativized to ??r. The goal is to prove that ??N is a test set, namely, for all r > N and all T1, T2 ? Ck(n, m, e), T1 ?r T2 holds if and only if T1 ?N T2. Clearly, for any r ? 0, T1 ?r+1 T2 is equivalent to T1,a ?r T2,a for every a ? ?, and T1 ?0 T2 (the latter being abbreviated as (?) below). Moreover, by Theorem 9, we have T1,a, T2,a ? Ck(n, m, e). This enables the following proof by induction on r: T1 ?r+1 T2 ? T1,a ?r T2,a (?a ? ?) and (?) ? (ind. hyp.) T1 ?N T2. ? T1,a ?N T2,a (?a ? ?) and (?) ? T1 ?N+1 T2 J Using Makanin?s algorithm for solving word equations (and even for deciding the existential theory of word equations, see e.g. [11] for a modern presentation) we obtain: I Proposition 16. Given n, m, e ? N, there is N ? N such that ??N is a test set for every pair of SST from Ck(n, m, e), and such an N can be effectively computed. Proof. By Lemma 14 we know that N exists, and Makanin?s algorithm allows to determine whether SN , SN+1 are equivalent, so to determine N by Lemma 15. J We finally obtain the main result: I Theorem 1. The equivalence problem for finite-valued SST is decidable. Of course, Theorem 1 does not come with any complexity upper bound, mainly because of the Ehrenfeucht conjecture. The only known lower bound is Pspace-hardness, which holds even for single-valued SST over unary output alphabets, and follows from a simple reduction from universality of NFA. Quite surprisingly, the exact complexity of equivalence is not known even for deterministic SST, where the problem is known to be between Nlogspace and Pspace [3]. We also recall that equivalence of deterministic SST with unary output can be checked in Ptime using invariants [4]. Finally, we recall that the currently best upper bound for solving word equations is Pspace [21] (with even linear space requirement, as shown in [19]). 6 Conclusions Our paper answers to a question left open in [5], showing that the equivalence problem for finite-valued SST is decidable. We followed a proof for one-way transducers due to Culik and Karhum?ki [10], that is based on the Ehrenfeucht conjecture. The main contribution of the paper is to provide the technical development that allows to follow the proof scheme of [10]. We believe that this development will also allow to obtain stronger results. We conjecture that finite-valued SST can be effectively decomposed into finite unions of unambiguous SST. This would entail that in the finite-valued setting, two-way transducers and SST have the same expressive power, as is it the case for single-valued transducers. If this holds with elementary complexity, then the equivalence of single-valued SST (or two-way transducers) could also be solved with elementary complexity. We believe that the complexity is indeed elementary, and leave this for future work. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 24 M.H. Albert and J. Lawrence . A proof of Ehrenfeucht's conjecture . Theor. Comput. Sci. , 41 ( 1 ): 121 - 123 , 1985 . Rajeev Alur and Pavel Cern? . Expressiveness of streaming string transducer . In IARCS Annual Conference on Foundation of Software Technology and Theoretical Computer Science (FSTTCS'10) , volume 8 of LIPIcs, pages 1 - 12 . Schloss Dagstuhl - Leibniz-Zentrum f?r Informatik, 2010 . Rajeev Alur and Pavol Cern? . Streaming Transducers for Algorithmic Verification of Single-pass List-processing Programs . In POPL'11. ACM , 2011 . Regular functions and cost register automata . In Proc. of Annual ACM/IEEE Symposium on Logic in Computer Science (LICS 2013 ), pages 13 - 22 . IEEE, 2013 . Rajeev Alur and Jyotirmoy Deshmukh . Nondeterministic streaming string transducers . In International Colloquium on Automata, Languages and Programming (ICALP'11) , volume 6756 of LNCS . Springer, 2011 . Michael Benedikt , Timothy Duff, Aditya Sharad, and James Worrell . Polynomial automata: Zeroness and applications . In Annual ACM/IEEE Symposium on Logic in Computer Science (LICS'17) , pages 1 - 12 . IEEE, 2017 . Adrien Boiret , Radoslaw Pi?rkowski, and Janusz Schmude . Reducing Transducer Equivalence to Register Automata Problems Solved by "Hilbert Method" . In IARCS Annual Conference on Foundations of Software Technology and Theoretical Computer Science (FSTTCS'18) , volume 122 of LIPIcs , pages 48 : 1 - 48 : 16 . Schloss Dagstuhl - Leibniz-Zentrum f?r Informatik, 2018 . Mikolaj Boja?czyk . The Hilbert Method for Transducer Equivalence . ACM SIGLOG News , January 2019 . Bruno Courcelle and Joost Engelfriet . Graph Structure and Monadic Second-Order Logic - A Language-Theoretic Approach , volume 138 of Encyclopedia of mathematics and its applications. Cambridge University Press, 2012 . Karel Culik II and Juhani Karhum?ki . The equivalence of finite valued transducers (on HDT0L languages) is decidable . Theor. Comput. Sci. , 47 : 71 - 84 , 1986 . Volker Diekert . Makanin's Algorithm . In M. Lothaire, editor, Algebraic combinatorics on words, volume 90 of Encyclopedia of mathematics and its applications , chapter 12 , pages 387 - 442 . Cambridge University Press, 2002 . Joost Engelfriet and Hendrik Jan Hoogeboom . MSO definable string transductions and two-way finite-state transducers . ACM Trans. Comput. Log. , 2 ( 2 ): 216 - 254 , 2001 . Comput . and System Sci ., 2 : 88 - 101 , 1968 . Paul Gallot , Anca Muscholl, Gabriele Puppis, and Sylvain Salvati . On the Decomposition of Finite-Valued Streaming String Transducers . In Annual Symposium on Theoretical Aspects of Computer Science (STACS'17) , volume 66 of LIPIcs , pages 34 : 1 - 34 : 14 . Schloss Dagstuhl - Leibniz-Zentrum f?r Informatik, 2017 . Victor S. Guba . Equivalence of infinite systems of equations in free groups and semigroups to finite subsystems . Mat. Zametki , 40 ( 3 ): 688 -- 690 , 1986 . Eitan M. Gurari . The equivalence problem for deterministic two-way sequential transducers is decidable . SIAM Journal of Computing , 448 - 452 , 1982 . Eitan M. Gurari and Oscar H. Ibarra . A note on finite-valued and finitely ambiguous transducers . Math. Syst. Theory , 16 ( 1 ): 61 - 66 , 1983 . Oscar H. Ibarra . The unsolvability of the equivalence problem for e-free NGSM's with unary input (output) alphabet and applications . SIAM J. of Comput. , 7 ( 4 ): 524 - 532 , 1978 . Artur Jez . Word Equations in Nondeterministic Linear Space . In Proc. International Colloquium on Automata, Languages, and Programming (ICALP'17) , volume 80 of LIPIcs , pages 95 : 1 - 95 : 13 . Schloss Dagstuhl-Leibniz-Zentrum f?r Informatik, 2017 . Juhani Karhum?ki . The Ehrenfeucht conjecture: a compactness claim for finitely generated free monoids . Theor. Comput. Sci. , 29 : 285 - 308 , 1984 . Wojciech Plandowski . Satisfiability of word equations with constants is in PSPACE . JACM, 51 ( 3 ): 483 - 496 , 2004 . In Annual Symposium on Theoretical Aspects of Computer Science (STACS'08) , volume 1 of LIPIcs, pages 621 - 632 . Schloss Dagstuhl - Leibniz-Zentrum f?r Informatik, 2008 . Jacques Sakarovitch and Rodrigo de Souza. Lexicographic decomposition of K-valued transducers . Theory Comput. Sci. , 47 : 758 - 785 , 2010 . Andreas Weber . Decomposing A k-Valued Transducer into k Unambiguous Ones . RAIRO-ITA , 30 ( 5 ): 379 - 413 , 1996 .


This is a preview of a remote PDF: http://drops.dagstuhl.de/opus/volltexte/2019/10698/pdf/LIPIcs-ICALP-2019-122.pdf

Anca Muscholl, Gabriele Puppis. Equivalence of Finite-Valued Streaming String Transducers Is Decidable, LIPICS - Leibniz International Proceedings in Informatics, 2019, 122:1-122:15, DOI: 10.4230/LIPIcs.ICALP.2019.122