On the Estimation of Linear Softmax Parametrized Markov Chains

In reinforcement learning and deep learning, softmax parameterization is commonly used to represent discrete probability distributions.In this work, we study three possible softmax parametrizations of the transition matrix of the Markov chain. Through theoretical and empirical lenses, we provide several insights into the effect of such parametrizations on estimating the Markov transition matrix.