#### Equivalence of Finite-Valued Streaming String Transducers Is Decidable

I C A L P
Equivalence of Finite-Valued Streaming String Transducers Is Decidable
Anca Muscholl 0 1
Gabriele Puppis CNRS 0 1
LaBRI 0 1
Bordeaux 0 1
France 0 1
0 LaBRI, University of Bordeaux , France
1 Category Track B: Automata , Logic, Semantics, and Theory of Programming
In this paper we provide a positive answer to a question left open by Alur and and Deshmukh in 2011 by showing that equivalence of finite-valued copyless streaming string transducers is decidable. 2012 ACM Subject Classification Theory of computation ? Transducers Related Version A full version of the paper is available at https://arxiv.org/abs/1902.06973. Finite transducers are simple devices that allow to reason about data transformations in an effective, and even efficient way. In their most basic form they transform strings using finite control. Unlike automata, their power heavily depends on various parameters, like non-determinism, the capability of scanning the input several times, or the kind of storage they may use. The oldest transducer model, known as generalized sequential machine, extends finite automata by outputs. Inspired by an approach that applies to arbitrary relational structures [9], logic-based transformations (also called transductions) were considered by Engelfriet and Hoogeboom [12]. They showed that two-way transducers and monadic-second order (MSO) definable transductions are equivalent in the deterministic case (and even if the transduction is single-valued, which is more general than determinism). This equivalence supports thus the notion of ?regular? functions, in the spirit of classical results on regular word languages from automata theory and logics due to B?chi, Elgot, Trakhtenbrot, Rabin, and others. A one-way transducer model that uses write-only registers as additional storage was proposed a few years ago by Alur and Cern? [2], and called streaming string transducer (SST). SST were shown equivalent to two-way transducers and MSO definable transductions in the deterministic setting, and again, even in the single-valued case. In the relational case the picture is less satisfactory, as expressive equivalence is only preserved for SST and non-deterministic MSO transductions [5], which extend the original MSO transductions by existentially quantified monadic parameters. On the other hand, two-way transducers and SST are incomparable in the relational case. Between functions and relations there is however one class of transductions that exhibits a better behavior, and this is the class of finite-valued transductions. Being finite-valued means that there exists some constant k such that every input belonging to the domain has at most k outputs. Finite-valued transductions were intensively studied in the setting of one-way and twoway transducers. For one-way transducers, k-valuedness can be checked in Ptime [17]. In addition, every k-valued one-way transducer can be effectively decomposed into a union of k
and phrases String transducers; equivalence; Ehrenfeucht conjecture
Introduction
122:2
unambiguous one-way transducers of exponential size [24, 23]. For both two-way transducers
and SST, checking k-valuedness is in Pspace.
Besides expressiveness, another fundamental question concerning transducers is the
equivalence problem, that is, the problem of deciding whether two transducers define the same
relation (or the same partial function if we consider the single-valued case). The equivalence
problem turns out to be Pspace-complete for deterministic two-way transducers [16],
singlevalued two-way transducers, as well as for single-valued SST [5]. For deterministic SST,
equivalence is in Pspace [3], but it is open whether this complexity upper bound is optimal.
For arbitrary SST, and in fact even for non-deterministic one-way transducers over a unary
output alphabet, equivalence is undecidable [13, 18]. The equivalence problem for k-valued
one-way transducers was shown to be decidable by Culik and Karhum?ki using an elegant
argument based on Ehrenfeucht?s conjecture [10], and the authors noted that the same proof
goes through for two-way transducers as well. The decidability status for the equivalence
problem for k-valued SST was first stated as an open problem in [5]. Another open problem
is whether SST and two-way transducers are equivalent in the finite-valued case, like in
the single-valued case. It is worth noting, however, that in the full relational case SST and
two-way transducers are incomparable. Concerning this last open question, a partial positive
answer was given in [14], by decomposing any finite-valued SST with only one register into a
finite union of unambiguous SST. This decomposition result also entails the decidability of
the equivalence problem for the considered class.
The main result of this paper is a positive answer to the first question left open in [5]:
I Theorem 1. The equivalence problem for finite-valued SST is decidable.
We show the above result with a proof idea due to Culik and Karhum?ki [10], based on
the Ehrenfeucht conjecture. Our proof is much more involved, because SST produce their
outputs piece-wise, in contrast to one-way and two-way transducers, that produce output
linearly while reading the input. We manage to overcome this obstacle using some (mild)
word combinatorics and word equations, by introducing a suitable normalization procedure
for SST. We believe that our technique will also allow to solve the second problem left open
in [5], which is the expressive equivalence between finite-valued SST and two-way transducers.
Related work
The equivalence problem for transducers has recently raised interest for more complex types
of transducers in the single-valued case: Filiot and Reynier showed that equivalence of
copyful, deterministic SST is decidable by showing them equivalent to HDT0L systems and
applying [10], which contains the above-mentioned result as a special case. Subsequently,
Benedikt et al. showed that equivalence of copyful, deterministic SST has Ackerman complexity,
with a proof based on polynomial automata and ultimately on Hilbert?s basis theorem [6].
Interestingly, the use of Hilbert?s basis theorem goes back to the proof of Ehrenfeucht?s
conjecture [1, 15]. A similar approach was used by Boiret et al. in [7] to show that bottom-up
register automata over unordered forests have a decidable equivalence problem, see also the
nice survey [8].
Overview
Section 2 introduces the transducer model, then Section 3 sets up the technical machinery
that allows to normalize finite-valued SST. Section 4 shows the major normalization result,
which holds for left quotients of SST. Finally Section 5 recalls the Ehrenfeucht-based proof for
equivalence and the application to finite-valued SST. A full version of the paper is available
at https://arxiv.org/abs/1902.06973.
2
Streaming string transducers
A streaming string transducer (SST ) is a tuple T = (?, ?, X, Q, U, I, E, F, xout), where ?
and ? are finite input and output alphabets, X is a finite set of registers (usually denoted
x, x?, x1, x2, etc.), Q is a finite set of states, U is a finite set of register updates, that is,
functions from X to (X ? ?)?, I, F ? Q are subsets of states, defining the initial and final
states, E ? Q ? ? ? U ? Q is a transition relation, describing, for each state and input symbol,
the possible register updates and target states, and finally xout ? X is a register for the
output. Note that, compared to the original definition from [2], here we forbid for simplicity
the use of final production rules, that perform ad additional register update after the end of
the input. This simplification is immaterial with respect to the decidability of the equivalence
problem. For example, it can be enforced, without loss of generality, by assuming that all
well-formed inputs are terminated by a special marker, say ?, on which the transducer can
apply a specific transition. We assume here that all inputs of a transducer are non-empty
and of the form u ?, with ? not occurring in u.
Copyless restriction and capacity
An SST as above is copyless if for all register updates f ? U , every register x ? X appears at
most once in the word f (x1) . . . f (xm), where X = {x1, . . . , xm}. For a copyless SST, every
output has length at most linear in the length of the input. More precisely, every output
associated with an input u has length at most cSuS, where c = maxf?U ?x?X Sf (x)S? is the
maximum number of letters that the SST can add to its registers along a single transition
(this number c is called capacity of the SST).
Hereafter, we assume that all SST are copyless.
Register updates and flows
Every register update, and in general every function f ? X ? (X ? ?)? is naturally extended
to a morphism on (X ? ?)?, by defining it as identity over ?. When reasoning with register
updates, it is sometimes possible to abstract away the specific words over ?, and only consider
how the contents of the registers flows into other registers. Formally, the flow of an update
f ? X ? (X ? ?)? is the bipartite graph that consists of two ordered sequences of nodes, one
on the left and one on the right, with each node in a sequence corresponding to a specific
register, and arrows that go from the node corresponding to register x to a right node
corresponding to register x whenever x occurs in f (x). For example, the flow of the update
f defined by f (x1) = a x1 aa x3, f (x2) = b a, and f (x3) = x2 b is the second bipartite graph in
the figure on page 5.
Note that there are finitely many flows on a fixed number of registers. Moreover, flows
can be equipped with a natural composition operation: given two flows F1 and F2, F1 ? F2 is
the bipartite graph obtained by glueing the right nodes of F1 with the left nodes of F2, and
by shortcutting pairs of consecutive arrows. We call flow monoid of an SST T the monoid of
flows generated by the updates of T , with the composition operation as associative product.
Transitions, runs, and loops
A transition (q, a, f, q?) of an SST T is conveniently denoted by the arrow q ?a?Tf? q?, and
~
the subscript T is often omitted when clear from the context. A run on w = a1 . . . an is a
sequence of transitions of the form q0 ??a1?~?f1? q1 ??a2?~?f2? . . . ?a?n?~?fn? qn. Sometimes, a run as
above is equally denoted by q0 ?w?~?f qn, so as to highlight the underlying input w and the
induced register update f = f1 ? ? ? ? ? fn. A run is initial (resp. final) if it begins with an
initial (resp. final) state; it is successful if it is both initial and final.
Given two registers x, x? and a run ? ? q ?w?~?f q?, we say that x flows into x? along ? if x
occurs in f (x?). Note that this property depends only on the flow of the induced update f .
An SST is said to be trimmed is every state occurs in at least one successful run, so every
state is reachable from the initial states and co-reachable from the final states. This property
can be easily enforced with a polynomial-time preprocessing.
When reasoning with automata, it is common practice to use pumping arguments.
Pumping will also be used here, but the notion of loop needs to be refined as to take into
account the effect of register updates. Formally, a loop of a run ? of an SST is any non-empty
factor of ? of the form ? ? q ?w?~?f q, that starts and ends in the same state q, and induces a
flow-idempotent update, namely, an update f such that f and f ? f have the same flow.
Outputs and finite-valuedness
The output of a successful run ? ? q0 ?w?~?f qn is defined as out(?) = (f0 ? f )(xout), where
f0(x) = ? for all x ? X. Sometimes, we write out(f ) in place of out(?). The relation realized
by an SST is the set of pairs (u, v) ? ?? ? ??, where u is a well-formed input (namely,
terminating with ?) and v is the output associated with some successful run on u. An SST
is k-valued if for every input u, there are at most k different outputs associated with u. It
is single-valued (resp. finite-valued) if it is k-valued for k = 1 (resp. for some k ? N). The
domain an SST T , denoted Dom(T ), is the set of input words that have some successful run
in T . Two SST T1, T2 are equivalent, denoted as T1 ? T2, if they realize the same relation
over ?? ? ??.
Register valuations
A register valuation is a function from X to ??. Given a successful run ? ? q0 ?a?1~?f1 q1 ?a?2~?f2
. . . a?n?~?fn qn and a position i ? {0, . . . , n} in it, the register valuation at position i in ? is the
function val?,i that is defined inductively on i as follows: val?,0(x) = ?, for all x ? X, and
val?,i+1 = val?,i ? fi. Note that val?,n(xout) coincides with the final output out(f1 ? ? ? ? ? fn)
produced by ?.
3
Normalizations
A major stumbling block in deciding equivalence of SST, as well as other crucial problems, lies
in the fact that the same output can be produced by very different runs. This phenomenon
already appears with much simpler transducers, e.g. with one-way transducers, where runs
may produce the same output, but at different speeds. However, the phenomenon is more
subtle for SST, as the output is produced piece-wise, and not sequentially: runs with same
output may appear to be different in many ways, e.g. in terms of the flows of the register
updates, or in terms of shifts of portions of the output. The goal of this section is to provide
suitable normalization steps that remove, one at a time, the above mentioned degrees of
freedom in producing the same output.
Another issue that we will be concerned with is the compatibility of the normalization
steps with constructions on transducers that shortcut arbitrary long runs into a single
transition. Essentially, we aim at having an effective notion of equivalence w.r.t. final outputs
that works not only for transitions but also for runs.
Normalization of flows
In this section, m will always denote the number of registers of an SST and X = {x1, . . . , xm}
the set of registers. It is convenient to equip X with a total order, say x1 < ? ? ? < xm.
Accordingly, we let ? = x1 . . . xm be the juxtaposition of all register names, and f (?) =
f (x1) . . . f (xm) for every register update f .
We say that a register update f is non-erasing if for every register x, f (?) contains at
least an occurrence of x (in fact, exactly one, since T is copyless). This can be rephrased as
a property of the flow of f , where every node on the left must have an outgoing arrow. In a
similar way, we say that f is non-permuting if registers appear in f ?
( ) with their natural
order and without jumps, that is, f (?) ? ??x1?? . . . ??xk??, for some k ? m. As before, this
can be rephrased by saying that the arrows in the flow of f must not be crossing, and the
target nodes to the right must form a prefix of ?. Below are some examples of updates with
their flows: the first update f is erasing, the second update g is non-erasing but permuting,
and the third update h is non-erasing and non-permuting.
f? xxx123 ??? ___ x1 _ x3 _
g? xxx123 ??? ___ xx12 __ x3 _
h? xxx123 ??? ___ xx13 __ x2 _
We say that T is flow-normalized if all its register updates are non-erasing and non-permuting.
Note that a flow-normalized SST with m registers can have at most 2m different flows.
I Proposition 2. One can transform any SST into an equivalent flow-normalized one.
Recall that a register valuation is a function from X to ??. With a flow-normalized SST,
one can also define a dual notion of valuation, representing ?gaps? between registers that
shrink along the run. For this we introduce m + 1 fresh variables y0, y1, . . . , ym, called gaps.
Hereafter, Y = {y0, y1, . . . , ym} will always denote the set of gaps. We use the term valuation
to generically denote a register/gap valuation, that is, a function from X ? Y to ??.
The idea is that a gap yj represents a word that is inserted between register xj (if
j > 0) and register xj+1 (if j < n) so as to form the final output. Formally, given a word
w ? ??x1?? . . . ??xk??, with k ? m, and given two registers xi, xj, with i < j, we denote by
w?xi, xj? the maximal factor of w strictly between the unique occurrence of xi and the unique
occurrence of xj, using the following conventions for the degenerate cases: if i = 0, then
w?xi, xj? is a maximal prefix of w; if i > 0 but there is no occurrence of xi, then w?xi, xj? = ?;
finally, if there is an occurrence of xi but no occurrence of xj in w, then w?xi, xj? is a maximal
suffix. Given a run ? ? q0 ?a?1~?f1 q1 ?a?2~?f2 . . . a?n?~?fn qn and a position i in it, the valuation at
position i of ? is the function val?,i ? X ? Y ? ?? such that
val?,i restricted to X is the register valuation at position i of ?,
val?,i maps every gap yj to the word (fi+1 ? ? ? ? ? fn)(?)?xj, xj+1?.
By definition, the image of the word ? = y0 x1 y1 . . . xm ym via the valuation val?,i is
always equal to the final output out(?), for all positions i. In this sense, the sequence of
valuations val?,0, val?,1, . . . , val?,n can be identified with a sequence of factorizations of out(?).
For example, below are the factorizations of the output before and after a transition with
register update f such that f (x1) = s x1 u x2 t and f (x2) = v, for s, u, t, v ? ??:
y?
0
y0
s
x1
x2
y1
u
This also suggests the principle that gaps, like registers, are updated along transitions via
suitable morphisms, but in a symmetric way, that is, from right to left. For instance, in the
above picture, the gaps y0, y1, y2 are updated by the function f ? such that f ?(y0) = y0 s,
f ?(y1) = u, and f ?(y2) = t y1 v y2. In general, the function f ?, called gap update, is uniquely
determined by the register update f , and vice versa, f is uniquely determined by the gap
update f ?. Another perhaps interesting phenomenon is that the gap update f ? is also
non-erasing and non-permuting (the notion of non-permuting gap assignment is defined
w.r.t. the reverse order ym < ? ? ? < y0).
Normalization of states
The next normalization step splits the states of an SST in such a way that it becomes possible
to associate with each state an over-approximation of the possible register/gap valuations
witnessed when the state is visited along a successful run. These over-approximations are
very simple languages over the output alphabet ?, e.g. singleton languages like {aba} and
periodic languages like {ab}?{a} (often denoted (ab)?a to improve readability). Basically
our over-approximations refer to length and period constraints. The period of a word w is
the least number 0 < p ? SwS such that w is a prefix of (w[1, p])?. For example, the period of
w = abcab is 3.
For a given parameter ? ? N we define the family L? that contains:
the empty language ?,
the singleton languages {u}, with u ? ?? and SuS ? ?,
the periodic languages u?v, with u ? ?+ primitive (i.e. u = wk only if k = 1),
SuS ? ?, and v ? ?? strict prefix of u,
the universal language ??.
The languages in L?, partially ordered by containment, form a finite meet semi-lattice, where
the meet is the intersection ?. We depict here part of the lattice L? for a parameter ? ? 3:
??
a?
?
(ab)?
ab
?
(ab)?a
a
aba
. . .
The semi-lattice structure allows to derive a best over-approximation in L? of any language
oLn?si?n?g,letthoant lias:ngLu?a?g=es?.{FLor? ?exLa?m?pLle?, ?foLr}?. =W3e, wwiellhmavoest{lyabuas}e??th=e {aapbpar}o,xi{mabaatiboan}?o?p=er(aatob)r??a?,
and {abbb}?? = ??. Note also that if SwS ? ? then w?? = {w}. A useful property is the
compatibility of ?? with concatenation, which immediately extends to compatibility with
word morphisms:
I Lemma 3. (L1 ? L2)?? = (L1?? ? L2??)?? for every ? ? N and L1, L2 ? ??.
Recall that X, Y denote, respectively, the sets of registers and gaps of a flow-normalized
SST. Given a valuation ? ? X ? Y ? ??, its ?-approximant is the function ??? ? X ? Y ? L?
that maps any z ? X ? Y to the language {?(z)}??. The set of ?-approximants is denoted
L?X?Y , and consists of all maps from X ? Y to L?. Further let
Valq = {val?,i ? ? successful run visiting q at any position i}
be the set of possible valuations induced by an arbitrary successful run when visiting state q.
A first desirable property is that all valuations in Valq have the same ?-approximant,
which is thus determined by the state q. Formally, given a flow-normalized SST T with
trimmed state space Q, we say that T admits ?-approximants if every state q ? Q can be
effectively annotated with an ?-approximant Aq ? L?X?Y in such a way that
?? ? Valq ?
??? = Aq.
(1)
This condition is best understood as an invariant on lengths and periods that can be enforced
on valuations of registers and gaps when visiting a particular state. For instance, when
the approximant Aq guarantees a certain period, then this period will be the same for all
valuations occurring at q, independently of the specific initial run that may lead to q (for
registers), and of the run that may lead from q to an accepting state (for gaps).
The proposition below shows that it is always possible to refine any SST T so as to admit
?-approximants, for any parameter ?. The proof for ?-approximants that concern only
registers could be understood as unfolding the SST T , and merging nodes corresponding to
any two inputs u and v, with u prefix of v, whenever the induced ?-approximants at u and v
are the same for every register x. In general, the resulting SST can be seen a covering of the
original SST T , in the sense formalized by Sakarovitch and de Souza in [22]: T ? is a covering
of T if the states of T ? can be mapped homomorphically to states of T , while preserving
transitions and the distinction into initial and final states, and, moreover, the outgoing
transitions of every state of T ? map one-to-one to outgoing transitions of a corresponding
state of T . This implies that the successful runs of T and those of T ? are in one-to-one
correspondence.
I Proposition 4. Let T be a flow-normalized SST, and let ? ? N. One can construct an
equivalent flow-normalized SST T ? that admits ?-approximants and that is a covering of T .
Notation. Whenever an SST admits ?-approximants Aq as above, it is convenient to denote
its states by triples of the form (q, AX , AY ), where AX (resp. AY ) is the restriction of the
?-approximant Aq of state q to registers (resp. gaps).
Note that the smaller the parameter ?, the weaker is the property required for
?approximants (in particular, for ? = 0 the lattice L? collapses to ? and ??). Choosing
? to be at least the capacity of the SST is already a reasonable choice, as it gives a nice
characterization of equivalence of transitions w.r.t. the produced outputs (cf. Lemma 5
below). However, we will see that it is desirable to have even finer approximants, in such a
way that our results will be compatible with left quotients of SST, that shortcut arbitrary
long runs into single transitions. We postpone the technical details to Section 4, and only
provide a rough intuition underlying the choice of the appropriate parameter ?. We will
choose ? much larger than the capacity of the SST, so that, by pumping arguments, one
can show that, for every state q and every parameter ? ? ?, the ?-approximant cannot be
strictly smaller than the ?-approximant on all valuations from Valq.
Normalization of transitions
We finally turn to studying a notion of equivalence on transitions that is similar to the
twosided Myhill-Nerode equivalence on words. We will only compare transitions that consume
the same input letter and link the same pair of states. Instead of using words as two-sided
contexts, we will use initial and final runs that can be attached to the considered transitions
in order to form successful runs, and instead of comparing membership in a language, we
will compare the effect on the produced outputs.
Consider two transitions ?1 ? q ?a?~f?1 q? and ?2 ? q ?a?~f?2 q?. We say that ?1 and ?2 are
equivalent if for every initial run ? leading to q and every final run ? starting in q?, the
outputs out(? ?1 ?) and out(? ?2 ?) are equal. We often refer to (?, ?) as a context for ?1, ?2.
In general, two transitions of an SST having the same source, target and ?-label might turn
out to be non-equivalent, and still produce the same output within specific contexts. However,
Lemma 5 below shows that this is not the case with ?-approximants at hand, provided that
? is at least the capacity of the SST. More precisely, we will show that the equivalence of two
a~f1 a~f2
transitions ?1 ? (q, AX , AY ) ??? (q?, A?X , A?Y ) and ?2 ? (q, AX , AY ) ??? (q?, A?X , A?Y ), where
f1, f2 have the same flow, only depends on the ?-approximants AX , A?Y that annotate the
source and target states. This will imply that ?1, ?2 either always produce the same output
or always produce different outputs, independently of the surrounding contexts. To prove the
statement, we have to consider register valuations induced by initial runs, and symmetrically
gap valuations induced by final runs. It helps to introduce the following:
Notation. Given an initial run ?, val?? is the register valuation induced at the end of ?;
symmetrically, val?? is the gap valuation induced at the beginning of a final run ?.
We also recall a consequence of the flow normalization: the effect on the final output
of an update f that occurs in a successful run can be described by a word Effect(f ) over
the alphabet X ? Y ? ?, defined as Effect(f ) = y0 f (x1) y1 . . . f (xm) ym+1. Note that in
Effect(f ) each register (resp. gap) occurs exactly once, according to the order x1 < ? ? ? < xm
(resp. y0 < ? ? ? < ym) ? the occurrences of registers and gaps, however, may not be strictly
interleaved. In Effect(f ), an occurrence of xi ? X represents an abstract valuation for
register xi before applying the update f , while an occurrence of yj ? Y represents an abstract
valuation for the gap yj after applying f . In particular, note that the x?s and the y?s refer
to valuations induced at different positions of a run. The maximal factors of Effect(f ) that
are entirely over ? represent the words that need to be added in order to get the register
valuation after f , or equally the gap valuation before f . For instance, by reusing the example
update f from page 5, where f (x1) = s x1 u x2 t and f (x2) = v, the effect of f is described by
the word Effect(f ) = y0 s x1 u x2 t y1 v y2, suggestively depicted as
Effect(f ) =
y0
s
x1
u
x2
t
y1
v
y2
(here the lengths of the blocks labeled with variables are immaterial). Note that the yj above
are in fact the yj? from the picture at page 5. In the above figure we have also highlighted
with dotted rectangles the factors that represent gap valuations before the update (e.g. y0 s),
and register valuations after the update (e.g. s x1 u x2 t).
Given a valuation ? and two approximants A ? L?X and A? ? LY? , one for register
valuations and the other for gap valuations, we write ? ? A ? A? to mean that ?(x) ? A(x)
and ?(y) ? A?(y) for all x ? X and y ? Y .
I Lemma 5. Let T be a trimmed flow-normalized SST. Given two transitions ?i ? ?q ?a?~f?i q?,
with i ? {1, 2}, a context (?, ?) for them, and the ?-approximants A = ((val??)? )SX and
A? = ((val??)??)SY , with ? ? N, the following holds:
1. If Effect(f1) = Effect(f2) holds on all valuations ? ? A ? A?, then out(? ?1 ?) = out(? ?2 ?).
2. If ?1, ?2 have the same flow, out(? ?1 ?) = out(? ?2 ?), and ? ? c, where c is the capacity
of T , then Effect(f1) = Effect(f2) holds on all valuations ? ? A ? A?.
Recall that in an SST that admits ?-approximants, states are of the form (q, AX , AY ),
and we have ((val??)??)SX = AX (resp. ((val??)??)SY = AY ) for every initial run ? that ends
in (q, AX , AY ) (resp. for every final run ? that starts in (q, AX , AY )). By pairing this with
Lemma 5, we immediately obtain the following corollary:
I Corollary 6. Let T be a trimmed flow-normalized SST. One can decide in polynomial time
whether two given transitions ?1, ?2 of T with the same flow are equivalent. Moreover, if T
has capacity c and admits ?-approximants for some ? ? c, then only two cases can happen:
1. either out(? ?1 ?) = out(? ?2 ?) for every context (?, ?) (so ?1, ?2 are equivalent),
2. or out(? ?1 ?) ? out(? ?2 ?) for every context (?, ?) (so ?1, ?2 are not equivalent).
Another important consequence is the following theorem, that normalizes finite-valued
SST in order to bound the maximum number of transitions linking the same pair of states
and consuming the same input letter. This number is called edge ambiguity for short.
I Theorem 7. Let T be a k-valued, flow-normalized SST that has m registers, capacity c,
and that admits ?-approximants, for some ? ? c. One can construct an equivalent SST T ?,
with the same states and the same registers as T , that has edge ambiguity at most k ? 2m.
Proof. By Corollary 6, T has at most k pairwise non-equivalent transitions with the same
input letter, the same source and target states, and the same flow. Moreover equivalence of
such transitions can be decided. We can then normalize T by removing in each equivalence
class all but one transitions with the same flow. Since T has at most 2m flows, the
normalization results in an equivalent SST T ? with edge ambiguity at most k ? 2m. J
The next section is devoted to prove a very similar result as above, but for all SST that
can be obtained by shortcutting runs into single transitions, and that thus have arbitrary
large capacity. This will be the main technical ingredient for establishing the decidability of
the equivalence problem for k-valued SST.
4
Shortcut construction
Here we focus on a transformation of relations that absorbs the first input letter when this
is equal to a specific element, say a ? ? ? {?}. Such a transformation maps any relation
R to the relation Ra = {(u, v) ? (au, v) ? R}. Observe that R = R? ? ?a?? Ra, where
R? = R ? ({?} ? ??).
It is easy to see that the class of relations realized by SST is effectively closed under the
transformation R ? Ra. To prove this closure property, it is convenient to restrict, without
loss of generality, to SST with transient initial states, namely, SST where no transition
reaches an initial state. Under this assumption, the closure property also preserves the state
space (though some states may become useless), the set of registers, the property of being
k-valued, as well as the ?-approximants, if they are admitted by the original SST. However,
the transformation does not preserves the capacity, which may increase.
I Lemma 8. Given a flow-normalized SST T with transient initial states, and given a letter
a ? ? ? {?}, one can construct an SST Ta with transient initial states such that
Dom(Ta) = {u ? ?? ? au ? Dom(T )},
Ta on input u produces the same outputs as T on input au.
Moreover, Ta has the same states and the same registers as T ; if T has capacity c, then Ta
has capacity 2c; if T admits ?-approximants, then so does Ta (via the same annotation).
The above construction can be applied inductively to compute an SST for any left quotient
Ru = {(v, w) ? (u v, w) ? R} of R. For u = a1 . . . an ? (??{?})?, we let Tu = (. . . (Ta1 )a2 . . . )an .
The last and most technical step consists in proving that edge ambiguity can be uniformly
bounded in every SST Tu, provided that the initial SST T is finite-valued, flow-normalized,
and admits ?-approximants for a large enough ?. More precisely, we aim at establishing that,
for ? much larger than the capacity of T , the notion of ?-approximant, besides satisfying
Equation (1), also satisfies the following property:
?? ? ?
?? ? ?
?? ?
?? ?
(val??)?
(val??)?
?
?
= (val??)?
= (val??)??.
?
(2)
In this case we say that the ?-approximants are tight.
Intuitively, the above property can be explained as follows. When considering an SST
with states annotated with ?-approximants, it may happen that for some larger parameter ?
some initial runs induce register valuations at a state (q, AX , AY ) whose ?-approximants
are strictly included in AX (e.g. possibly entailing new periodicities). These runs should be
thought of as exceptional cases, and there is a way of pumping them so as to restore the
equality between AX and the induced ?-approximant.
Let us first see how tight approximants are used. The theorem below assumes that there
is an SST T ? with tight approximants (later we will show how to compute such an SST),
and bounds the edge ambiguity of the SST Tu? that realizes a left quotient of T ?.
I Theorem 9. Let T ? be a k-valued, flow-normalized SST realizing R, with transient initial
states and tight ?-approximants. For every u ? ??, one can construct an SST Tu? realizing Ru,
with the same states and the same registers as T ?, and with edge ambiguity at most k ? 2m.
Proof. The crux is to show that the SST Tu? obtained from Lemma 8 has at most k pairwise
non-equivalent transitions with the same flow (for any given source/target state and label).
Once this is proven, one can proceed as in the proof of Theorem 7, by removing all but
one transition with the same flow in each equivalence class. By way of contradiction,
assume that Tu? has k + 1 pairwise non-equivalent transitions ?1, . . . , ?k+1 with the same flow.
Since T ? admits tight ?-approximants, by Lemma 8 we know that the source and target
state, respectively, of the previous transitions are annotated with tight ?-approximants, say
(AX , AY ) and (A?X , A?Y ), respectively.
We begin by applying the first claim of Lemma 5, implying that the equation Effect(fi) =
Effect(fj) is violated for some valuation ? ? AX ? A?Y . Then, we let ? = max(?, SuSc) a?nd use
Equations (1) and (2) to get a context (?, ?) such that (val??)?? = AX and (val??)? = A?Y .
Finally, knowing that ? is at least the capacity of Tu? , we apply the second claim of Lemma 5
to get out(? ?i ?) ? out(? ?j ?), thus witnessing non-equivalence of all pairs of transitions ?i, ?j
at the same time. This contradicts the assumption that Tu? (and hence T ?) is k-valued. J
Now, let T be a flow-normalized SST with m registers, capacity c, and trimmed state
space Q. Below, we show how to compute, with the help of Proposition 4, an SST T ?
equivalent to T that admits tight approximants. For simplicity, we will mostly focus on
register valuations induced by initial runs, even though similar results can be also stated for
gap valuations induced by final runs. We begin by giving a few technical results based on
pumping arguments. We say that register x is productive along ? if the update induced by ?
maps x to a word that contains at least one letter from ?. We also recall that a loop of a
run needs to induce a flow-idempotent update.
I Lemma 10. If ? = ?1 ? ?2 is an initial run of T , with ? loop, then for every n > 0 the
pumped run ?(n) = ?1 ?n ?2 induces valuations val?(n)? mapping any register x to a word of
the form u0 v1n?1 u1 . . . v2nm?1 u2m, where u0, . . . , u2m, v1, . . . , v2m ? ?? depend on ? and x, but
not on n. Moreover, we have vi ? ? for some i if there is a register x? that is productive along
? and that flows into x along ?2.
Given a tuple of pairwise disjoint loops ? = ?1, . . . , ?` in a run ?, we write ?? ?? ? when ??
is obtained from ? by simultaneously pumping n times every loop ?i, for some n > 0. When
using this notation, we often omit the subscript ?; in this case we tacitly assume that ? is
uniquely determined from ?. In this way, when writing, for instance, ??, ??? ? ?, we will know
that ??, ??? are obtained by pumping the same loops of ?. We also say that a property on
runs holds for all but finitely many ?? ? ? if it holds on runs ?? that are obtained from ? by
pumping n times the loops in a fixed tuple ?, for all n > n0 and for a sufficiently large n0.
I Lemma 11. Let ? be an initial run and x a register. If val??(x) has length (resp. period)
larger than ? = m c SQS 23?2m , then for every ? ? ? and for all but finitely many ?? ? ?,
val???(x) has length (resp. period) larger than ?.
Using the previous lemmas and the fact that the type of quantification ?for all but finitely
many runs? commutes with conjunctions (e.g. those used to enforce properties on each
register x ? X), we obtain that ?-approximants are tight for sufficiently large ?:
I Proposition 12. Let T ? be the SST admitting ?-approximants that is obtained from T using
Proposition 4, for any ? ? m c SQS 23?2m , where Q is the set of states of T . The ?-approximants
of T ? are tight.
5
Equivalence algorithm
The equivalence algorithm for k-valued SST follows a classical approach of Culik and
Karhum?ki [10] that is based on so-called test sets. A test set for two SST T1, T2 over input
alphabet ? is a set F ? ?? such that T1, T2 are equivalent if and only if they are equivalent
over F . The main contribution of [10] is to show that finite test sets exist and be computed
effectively for k-valued one-way transducers.The key ingredient of their proof is to show the
existence of a test set that works for all transducers with fixed number of states. An essential
observation is that for k-valued one-way, or even two-way, transducers one can assume that
the edge ambiguity is at most k. The reason for this is simply that the output is generated
sequentially. For SST the situation is far more complex because the output is generated
piecewise. The purpose of the normalizations performed in Section 3 was precisely to restore
the property of bounded edge ambiguity.
In a nutshell, the existence of a test set for transducers is a consequence of Ehrenfeucht?s
conjecture, whereas the effectiveness is based on the resolution of word equations due to
Makanin (see e.g. the survey [11]).
Ehrenfeucht?s conjecture was originally stated as a conjecture about formal languages:
for every language L ? ??, there is a finite subset F ? L such that for all morphisms
f, g ? ?? ? ??, f (w) = g(w) for every w ? L if and only if f (w) = g(w) for every w ? F . Such
a set F is called a test set for L.
There is an equivalent formulation of Ehrenfeucht?s conjecture in terms of a compactness
property of word equations [20]. Let ? and ? be two alphabets, where the elements in ? are
called unknowns. A word equation is a pair (u, v) ? ?? ? ??, and a solution is a morphism
? ? ?? ? ?? such that ?(u) = ?(v). Ehrenfeucht?s conjecture is equivalent to saying that any
system of equations over a finite set ? of unknowns has a finite, equivalent subsystem, where
equivalence means that the solution sets are the same. The latter compactness property was
proved in [1, 15] by encoding words by polynomials and using Hilbert?s basis theorem.
In view of Propositions 2, 4, and 12, we can restrict without loss of generality to SST that
are flow-normalized and that admit tight approximants. Hereafter, we shall tacitly assume
that all transducers are of this form. Given some integers k, n, m, and e, let Ck(n, m, e) be
the class of k-valued SST with at most n states, m registers, and edge-ambiguity at most e.
Note that if T is k-valued, then by Theorem 7 it belongs to Ck(n, m, e), where n, m are the
number of states and registers of T and e = k ? 2m. Similarly, by Lemma 8 and Theorem 9,
every left quotient Tu also belongs to Ck(n, m, e).
Now, let us fix k, n, m, e and consider an arbitrary SST T from Ck(n, m, e). Following
[10] we first build an abstraction of T by replacing each maximal factor from ?? occurring in
some update function of T , by a distinct unknown from ?. The SST ?(T ) obtained in this
way is called a schema; its outputs are words over ?. Note that the assumption of bounded
edge ambiguity is essential here to get a uniform bound on the number of unknowns required
for a schema. Clearly, there are only finitely many schemas of SST in Ck(n, m, e). We denote
by ?T ? ? ? ?? the partial mapping (concretization) that associates with each unknown the
corresponding word from ?? as specified by the updates of T .
We can rephrase the equivalence T1 ? T2 of two arbitrary SST from Ck(n, m, e) as an
infinite ?system? of word equations1 S = ?u??? ?? S? over set of unknowns ? ? ??. The
unknowns from ? are used for the schema ?(T1), whereas those from ?? are used for ?(T2);
in particular, ?T1 ? ? ? ?? and ?T2 ? ?? ? ??. The disjunctions in S are finite, with ? ranging
over the possible schemas ?1, ?2 (for T1 and T2, respectively) and the possible partitions of
the set of runs of ?1 and ?2 over the input u, into at most k groups (one for each possible
output). Finally, S? is a (finite) system of word equations, stating the equality of the words
from ?? ? ??? that belong to the same group according to ?.
The following lemma was stated in [10] for k-valued one-way transducers, but it holds
as well for two-way transducers and for SST (even copyful, with a proper definition for
Ck(n, m, e)):
I Lemma 13. Given two SST T1, T2 from Ck(n, m, e), the system S = ?u??? ?? S? has
?T1 ? ?T2 as solution if and only if T1 ? T2.
As shown in [10], the Ehrenfeucht conjecture can be used to show that any infinite system
S as in Lemma 13 is equivalent to some finite sub-system SN = ?u???N ?? S?. This gives:
I Lemma 14. Given n, m, e ? N, there is N ? N such that ??N is a test set for every pair of
SST T1, T2 from Ck(n, m, e).
Using Theorem 7 and Lemma 14 we can derive immediately the existence of a finite test
set for any two k-valued SST. The last question is how to compute such a test set effectively.
For this we will use the shortcut construction provided in Section 4.
1 Formally, S depends on n and k, but for simplicity we leave out the indices.
I Lemma 15. Assume that the formulas SN and SN+1 are equivalent, i.e., they have the
same solutions. Then ??N is a test set for any pair of SST from Ck(n, m, e).
Proof. Let T1 ?r T2 denote equivalence of T1 and T2 relativized to ??r. The goal is to prove
that ??N is a test set, namely, for all r > N and all T1, T2 ? Ck(n, m, e), T1 ?r T2 holds if
and only if T1 ?N T2. Clearly, for any r ? 0, T1 ?r+1 T2 is equivalent to T1,a ?r T2,a for every
a ? ?, and T1 ?0 T2 (the latter being abbreviated as (?) below). Moreover, by Theorem 9, we
have T1,a, T2,a ? Ck(n, m, e). This enables the following proof by induction on r:
T1 ?r+1 T2
?
T1,a ?r T2,a (?a ? ?) and (?)
? (ind. hyp.)
T1 ?N T2.
?
T1,a ?N T2,a (?a ? ?) and (?)
?
T1 ?N+1 T2
J
Using Makanin?s algorithm for solving word equations (and even for deciding the existential
theory of word equations, see e.g. [11] for a modern presentation) we obtain:
I Proposition 16. Given n, m, e ? N, there is N ? N such that ??N is a test set for every
pair of SST from Ck(n, m, e), and such an N can be effectively computed.
Proof. By Lemma 14 we know that N exists, and Makanin?s algorithm allows to determine
whether SN , SN+1 are equivalent, so to determine N by Lemma 15. J
We finally obtain the main result:
I Theorem 1. The equivalence problem for finite-valued SST is decidable.
Of course, Theorem 1 does not come with any complexity upper bound, mainly because
of the Ehrenfeucht conjecture. The only known lower bound is Pspace-hardness, which
holds even for single-valued SST over unary output alphabets, and follows from a simple
reduction from universality of NFA.
Quite surprisingly, the exact complexity of equivalence is not known even for deterministic
SST, where the problem is known to be between Nlogspace and Pspace [3]. We also
recall that equivalence of deterministic SST with unary output can be checked in Ptime
using invariants [4]. Finally, we recall that the currently best upper bound for solving word
equations is Pspace [21] (with even linear space requirement, as shown in [19]).
6
Conclusions
Our paper answers to a question left open in [5], showing that the equivalence problem for
finite-valued SST is decidable. We followed a proof for one-way transducers due to Culik and
Karhum?ki [10], that is based on the Ehrenfeucht conjecture. The main contribution of the
paper is to provide the technical development that allows to follow the proof scheme of [10].
We believe that this development will also allow to obtain stronger results. We conjecture
that finite-valued SST can be effectively decomposed into finite unions of unambiguous SST.
This would entail that in the finite-valued setting, two-way transducers and SST have the
same expressive power, as is it the case for single-valued transducers. If this holds with
elementary complexity, then the equivalence of single-valued SST (or two-way transducers)
could also be solved with elementary complexity. We believe that the complexity is indeed
elementary, and leave this for future work.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
24
M.H. Albert and J. Lawrence . A proof of Ehrenfeucht's conjecture . Theor. Comput. Sci. , 41 ( 1 ): 121 - 123 , 1985 .
Rajeev Alur and Pavel Cern? . Expressiveness of streaming string transducer . In IARCS Annual Conference on Foundation of Software Technology and Theoretical Computer Science (FSTTCS'10) , volume 8 of LIPIcs, pages 1 - 12 . Schloss Dagstuhl - Leibniz-Zentrum f?r Informatik, 2010 .
Rajeev Alur and Pavol Cern? . Streaming Transducers for Algorithmic Verification of Single-pass List-processing Programs . In POPL'11. ACM , 2011 .
Regular functions and cost register automata . In Proc. of Annual ACM/IEEE Symposium on Logic in Computer Science (LICS 2013 ), pages 13 - 22 . IEEE, 2013 .
Rajeev Alur and Jyotirmoy Deshmukh . Nondeterministic streaming string transducers . In International Colloquium on Automata, Languages and Programming (ICALP'11) , volume 6756 of LNCS . Springer, 2011 .
Michael Benedikt , Timothy Duff, Aditya Sharad, and James Worrell . Polynomial automata: Zeroness and applications . In Annual ACM/IEEE Symposium on Logic in Computer Science (LICS'17) , pages 1 - 12 . IEEE, 2017 .
Adrien Boiret , Radoslaw Pi?rkowski, and Janusz Schmude . Reducing Transducer Equivalence to Register Automata Problems Solved by "Hilbert Method" . In IARCS Annual Conference on Foundations of Software Technology and Theoretical Computer Science (FSTTCS'18) , volume 122 of LIPIcs , pages 48 : 1 - 48 : 16 . Schloss Dagstuhl - Leibniz-Zentrum f?r Informatik, 2018 .
Mikolaj Boja?czyk . The Hilbert Method for Transducer Equivalence . ACM SIGLOG News , January 2019 .
Bruno Courcelle and Joost Engelfriet . Graph Structure and Monadic Second-Order Logic - A Language-Theoretic Approach , volume 138 of Encyclopedia of mathematics and its applications.
Cambridge University Press, 2012 .
Karel Culik II and Juhani Karhum?ki . The equivalence of finite valued transducers (on HDT0L languages) is decidable . Theor. Comput. Sci. , 47 : 71 - 84 , 1986 .
Volker Diekert . Makanin's Algorithm . In M. Lothaire, editor, Algebraic combinatorics on words, volume 90 of Encyclopedia of mathematics and its applications , chapter 12 , pages 387 - 442 . Cambridge University Press, 2002 .
Joost Engelfriet and Hendrik Jan Hoogeboom . MSO definable string transductions and two-way finite-state transducers . ACM Trans. Comput. Log. , 2 ( 2 ): 216 - 254 , 2001 .
Comput . and System Sci ., 2 : 88 - 101 , 1968 .
Paul Gallot , Anca Muscholl, Gabriele Puppis, and Sylvain Salvati . On the Decomposition of Finite-Valued Streaming String Transducers . In Annual Symposium on Theoretical Aspects of Computer Science (STACS'17) , volume 66 of LIPIcs , pages 34 : 1 - 34 : 14 . Schloss Dagstuhl - Leibniz-Zentrum f?r Informatik, 2017 .
Victor S. Guba . Equivalence of infinite systems of equations in free groups and semigroups to finite subsystems . Mat. Zametki , 40 ( 3 ): 688 -- 690 , 1986 .
Eitan M. Gurari . The equivalence problem for deterministic two-way sequential transducers is decidable . SIAM Journal of Computing , 448 - 452 , 1982 .
Eitan M. Gurari and Oscar H. Ibarra . A note on finite-valued and finitely ambiguous transducers . Math. Syst. Theory , 16 ( 1 ): 61 - 66 , 1983 .
Oscar H. Ibarra . The unsolvability of the equivalence problem for e-free NGSM's with unary input (output) alphabet and applications . SIAM J. of Comput. , 7 ( 4 ): 524 - 532 , 1978 .
Artur Jez . Word Equations in Nondeterministic Linear Space . In Proc. International Colloquium on Automata, Languages, and Programming (ICALP'17) , volume 80 of LIPIcs , pages 95 : 1 - 95 : 13 . Schloss Dagstuhl-Leibniz-Zentrum f?r Informatik, 2017 .
Juhani Karhum?ki . The Ehrenfeucht conjecture: a compactness claim for finitely generated free monoids . Theor. Comput. Sci. , 29 : 285 - 308 , 1984 .
Wojciech Plandowski . Satisfiability of word equations with constants is in PSPACE . JACM, 51 ( 3 ): 483 - 496 , 2004 .
In Annual Symposium on Theoretical Aspects of Computer Science (STACS'08) , volume 1 of LIPIcs, pages 621 - 632 . Schloss Dagstuhl - Leibniz-Zentrum f?r Informatik, 2008 .
Jacques Sakarovitch and Rodrigo de Souza. Lexicographic decomposition of K-valued transducers . Theory Comput. Sci. , 47 : 758 - 785 , 2010 .
Andreas Weber . Decomposing A k-Valued Transducer into k Unambiguous Ones . RAIRO-ITA , 30 ( 5 ): 379 - 413 , 1996 .