this paper, semi-continuous HMMs (SCHMMs) (Bellagarda & Nahamoo 89; Huang & Jack 89) and continuous densities HMMs (CDHMMs) will be considered in conjunction with networks trained with the generalized delta rule (Rumelhart et al 86). It will be shown how to perform a joint global optimi ation of both the ANN and the HMM parameter estimation. In the proposed algorithm, the gradient of the optimization criterion with respect to the transformed observations is computed for the HMM system. The HMM can be trained with traditional methods (Rabiner 89) with which the gradient of an optimization criterion is computed. This gradient is sent to the ANN for the estimation of the weight associated to each connection of the network. No assumption need to be made or constraints imposed on the network outputs, except that the network output distribution should be modeled by a mixture of multivariate gaussians. Since training of HMMs is usually much faster than ANN training, we consider how to initialize the ANN in order to start from parameter values that are not too far from those obtained after training. Multiple ANNs are combined and an incremental design method is described in which specialized networks are integrated to the recognition system in order to improve its performance. Relate or Interesting papers have been published recently, describing attempts at com-