Testing of hidopt

Testing of hidopt - hidden dynamic parameter estimator
HBR 180898

Testing the optimiser with a one-dimensional data and a linear mapping

In this simple test, one-dimensional synthetic data was generated. Four target values were defined, and a number of segments produced, each of which associated with one of the four target values. From this, a hidden dynamic parameter sequence could be produced. The observations in this case were a simply linearly dependent on the hidden dynamics (so they weren't so hidden after all!).

Figure 1. - The learnt mapping. hidopt was used to learn both the mapping (using an MLP architecture of 1:4:1) and the target values.
The learnt mapping is shown here, an approximation to the linear relationship actually used to build the data.

Figure 2. - The real and estimated hidden dynamics. hidopt changes the mapping and targets by gradient descent until outputs produced by the estimated hidden dynamics match the observation sequence.
Shown here is a scaled version of the estimated hidden dynamics, compared with the real hidden dynamics. The targets will not necessarily be identical to the original targets used to produce the data unless we apply extra constraints in the estimation.

Learning non-linear functions

The next test is more of a challenge for the optimiser. This time a really non-linear function is used with a many-to-one mapping: a quadratic.

Figure 1 shows the non-linear quadratic function used to generate the synthetic observation data. The x-axis here relates to the hidden dynamic domain, and the y-axis the observation domain. This is a particularly interesting function as there are only two unique target values in the observation domain, as two of the targets map to the same value.
Similar identical observation "targets" exist also for speech. Some identical "static" vowel sounds can be produced with different vocal tract configurations. The dynamic qualities of the speech however can reveal the true nature of the vowel (Carre et al. 1995). Figure 1. - The non-linear mapping with targets.

Figure 2 shows the same hidden dynamic sequence used in the first example, and the observation time sequence after passing through this non-linearity. Figure 2. - Underlying hidden dynamics and the
resulting observations.

hidopt is then used to learn the targets and non-linear mapping. It does this by adjusting the targets and mapping until its computed output sequence matches the observations.
Figure 3 shows this error function being search by hidopt using gradient descent (conjugate gradients). Each successive line search is shown by a green line, and the minimum by a red point. When the minimum is reached, the optimisation chooses a different direction in target/weight space, and a new line search is carried out. Here, all the changes on direction have been stretched out into one dimension: the x-axis of this plot.
Due to local minima in the error surface, hidopt is not always successful at finding a global minimum. In fact, several attempt were made from different starting positions before achieving this successful minimisation. Figure 3. - The error function being searched.

Figure 4 shows hidopt gradually learning the non-linear mapping with successive iterations. A quadratic bowl-shaped curve is eventually reached. The tailing-off at the extemities of the curve is not important as this is outside the range of the target values, and hence will not be used. Figure 4. - The estimated mapping with successive
iterations.

Learning multidimensional functions

Figure 1. - The 2d non-linear mapping. Synthetic data was generated from this challenging non-linear mapping. The curved plot shows how the reference grid is transformed.
Two-dimensional targets are used here which can be considered points on this two-dimensional plane.

Figure 2. - Targets and hidden dynamics. These points are shown here, and the hidden dynamics then become simple trajectories which approach, but do not reach these target points.

Figure 3. - Observed data. After the non-linear mapping, however, the trajectories are less straightforward.
This slide shows the synthetic output data which represents the observed speech, and (without the transformed target points that are shown here) is all the information that is available for learning both the target values and the non-linear mapping.

hidopt is then used to attempt to learn the targets and mapping from this observation data.

hidopt optimises its estimate of the targets and mapping by gradient descent until its computed output matches that of the observations.
Shown here is the best match to the observations that hidopt could achieve before the gradient descent stopped.
Figure 4. - Matching the observations.

Shown here are the estimated targets, and the hidden dynamics.
This will not necessarily be the same as Figure 2, as there are (at least) an infinite number of solutions that are linearly related.

Figure 5. - The estimated targets and dynamics.

A more satisfying view of the targets and hidden dynamics can be obtained by applying a (linear) whitening transform.
This looks a lot more like the original hidden dynamics in Figure 2 as the original targets were spread out evenly, which is what the whitening transform achieves.
The non-linear mapping can be easily adjusted to use these new targets with equivalent results. Figure 6. - Whitened version of Figure 5.

Figure 1 shows the non-linear quadratic function used to generate the synthetic observation data. The x-axis here relates to the hidden dynamic domain, and the y-axis the observation domain. This is a particularly interesting function as there are only two unique target values in the observation domain, as two of the targets map to the same value. Similar identical observation "targets" exist also for speech. Some identical "static" vowel sounds can be produced with different vocal tract configurations. The dynamic qualities of the speech however can reveal the true nature of the vowel (Carre et al. 1995).	Figure 1. - The non-linear mapping with targets.
Figure 2 shows the same hidden dynamic sequence used in the first example, and the observation time sequence after passing through this non-linearity.	Figure 2. - Underlying hidden dynamics and the resulting observations.
hidopt is then used to learn the targets and non-linear mapping. It does this by adjusting the targets and mapping until its computed output sequence matches the observations. Figure 3 shows this error function being search by hidopt using gradient descent (conjugate gradients). Each successive line search is shown by a green line, and the minimum by a red point. When the minimum is reached, the optimisation chooses a different direction in target/weight space, and a new line search is carried out. Here, all the changes on direction have been stretched out into one dimension: the x-axis of this plot. Due to local minima in the error surface, hidopt is not always successful at finding a global minimum. In fact, several attempt were made from different starting positions before achieving this successful minimisation.	Figure 3. - The error function being searched.
Figure 4 shows hidopt gradually learning the non-linear mapping with successive iterations. A quadratic bowl-shaped curve is eventually reached. The tailing-off at the extemities of the curve is not important as this is outside the range of the target values, and hence will not be used.	Figure 4. - The estimated mapping with successive iterations.

Figure 1. - The 2d non-linear mapping.	Synthetic data was generated from this challenging non-linear mapping. The curved plot shows how the reference grid is transformed. Two-dimensional targets are used here which can be considered points on this two-dimensional plane.
Figure 2. - Targets and hidden dynamics.	These points are shown here, and the hidden dynamics then become simple trajectories which approach, but do not reach these target points.
Figure 3. - Observed data.	After the non-linear mapping, however, the trajectories are less straightforward. This slide shows the synthetic output data which represents the observed speech, and (without the transformed target points that are shown here) is all the information that is available for learning both the target values and the non-linear mapping.

hidopt optimises its estimate of the targets and mapping by gradient descent until its computed output matches that of the observations. Shown here is the best match to the observations that hidopt could achieve before the gradient descent stopped.	Figure 4. - Matching the observations.
Shown here are the estimated targets, and the hidden dynamics. This will not necessarily be the same as Figure 2, as there are (at least) an infinite number of solutions that are linearly related.	Figure 5. - The estimated targets and dynamics.
A more satisfying view of the targets and hidden dynamics can be obtained by applying a (linear) whitening transform. This looks a lot more like the original hidden dynamics in Figure 2 as the original targets were spread out evenly, which is what the whitening transform achieves. The non-linear mapping can be easily adjusted to use these new targets with equivalent results.	Figure 6. - Whitened version of Figure 5.