Minimal Working Examples

Here, I sample from / fit a sparse gaussian process using results from Titsias’ and Hensman et al. (2013).

### R code

N_p = 5000  # num data for fit & pred
N_u = 15    # num pseudo data

N_IS_SMALL = (N_p <= 5000)

x_p = seq(-5, 5, length.out = N_p)
x_u = seq(-3, 3, length.out = N_u)

# K = ( Kpp Kpu )
# K = ( Kup Kuu )

if(N_IS_SMALL)
    Kpp = exp(-((matrix(rep(x_p, N_p), N_p, N_p) -
               t(matrix(rep(x_p, N_p), N_p, N_p)))/4)^2)

Kpu = exp(-((matrix(rep(x_p, N_u), N_p, N_u) -
             matrix(rep(x_u, each = N_p), N_p, N_u))/4)^2)

Kuu = exp(-((matrix(rep(x_u, N_u), N_u, N_u) -
           t(matrix(rep(x_u, N_u), N_u, N_u)))/4)^2)

Kuui = solve(Kuu + diag(N_u)*1e-10)

# Prior Sample

mu_param = t(t(rnorm(N_u)))
sigma_param = diag(N_u) * 1e-2

# Exact VFE

y = sin(x_p) + rnorm(N_p, 0, 0.25)
func_pres = 10
sigma_param = solve(func_pres * Kuui %*% t(Kpu) %*% Kpu %*% Kuui + Kuui)
mu_param = func_pres * sigma_param %*% Kuui %*% t(Kpu) %*% y

mu = Kpu %*% Kuui %*% mu_param
if(N_IS_SMALL)
    sigma = Kpp + Kpu %*% Kuui %*% (sigma_param - Kuu) %*% Kuui %*% t(Kpu)

plot(x_p, mu, type = "l")
if(N_IS_SMALL)
    arrows(x_p, mu - 2*sqrt(diag(sigma)),
           x_p, mu + 2*sqrt(diag(sigma)),
           length = 0.01, angle = 90,
           col = "#BEBEBE20")

# SVI VFE

batch_size = 50
l = 0.1

y = sin(x_p) + rnorm(N_p, 0, 0.25)
func_pres = 10

theta_1 = solve(diag(N_u) * 1e-2) %*% t(t(rnorm(N_u)))
theta_2 = -0.5 * solve(diag(N_u) * 1e-2)

for(i in 1:10000) {
	indices = sample(1:N_p, batch_size)

	lambda = func_pres * Kuui %*% t(Kpu[indices, ]) %*% Kpu[indices, ] %*% Kuui + Kuui
	theta_2 = theta_2 - l*(0.5*lambda + theta_2)
	theta_1 = theta_1 + l*(func_pres * Kuui %*% t(Kpu[indices, ]) %*% y[indices] - theta_1)

	sigma_param = solve(-2 * theta_2)
	mu_param = sigma_param %*% theta_1

}

I might write this up at some point in C++.

2020

Efficient Gaussian Process Computation

I’ll try to give examples of efficient gaussian process computation here, like the vec trick (Kronecker product trick), efficient toeliptz and circulant matrix computations, RTS smoothing and Kalman filtering using state space representations, and so on.

4 min read

Gaussian Process Speech Synthesis (Draft)

Very untidy first working draft of the idea mentioned on the efficient computation page. Here, I fit a spectral mixture to some audio data to build a “generative model” for audio. I’ll implement efficient sampling later, and I’ll replace the arbitrary way this is trained with an LSTM-RNN to go straight from text/spectrograms to waveforms.

4 min read
Back to Top ↑

2019

Gaussian Process Middle C

First of my experiments on audio modelling using gaussian processes. Here, I construct a GP that, when sampled, plays middle c the way a grand piano would.

1 min read
Back to Top ↑

2018

Back to Top ↑