Uninterpretable fitted rbf variance

[jonwangio]

Hi there,

I was experimenting with how GPy captures the underlying process. I simulated many Gaussian Processes by using known rbf hyper-parameters, and tried to see if GPy model made meaningful inference about the hyper-parameters.

The starting point is very simple, with a noiseless simulation:

def arbKer(var, l):  # Arbitrary kernel with variance and lengthscale
    # RBF form
    kernel = GPy.kern.RBF(input_dim=1, variance=var, lengthscale=l)
    return kernel

def priorGPclean(X, var, l, nsamp, show=True):  # Simulate a bunch of noiseless/clean kernel
    # GP controlled noise drawn from a covariance matrix generated from kernel
    k = arbKer(var, l)
    mu = np.zeros((len(X))) # vector of the means
    C = k.K(X,X) # covariance matrix
    
    Y = np.random.multivariate_normal(mu,C,nsamp)
    Y = np.transpose(Y)
    return (Y)

X = np.random.rand(100)[:,None]; X=X*15-5
Y = priorGPclean(X, 4.5, 2., nsamp=1, show=False)

Now, clearly, according to the figure below, the very dense sampling from this simulated GP curve does not lead to an inference any closer to the known variance: var = 4.5, and how come such pattern led to an inference of var=8.5?

Could anyone please give me a hint?

Thank you very much!

[lionfish0]

I'm guessing that the log marginal likelihood just doesn't really change much between 4.5 and 8.5. See colab <https://colab.research.google.com/drive/1xfdLXrQyRGwiWBWFDVt_nmNjycK-xf9-?usp=sharing> . Black = the variance used to make the training data, the red line is the maximum in this grid search. [image: image.png] I think quite often the model is quite insensitive to the kernel variance, within one-order-of-magnitude. I was thinking in your case particularly that maybe this is because the likelihood noise variance is basically missing (so the kernel variance isn't being 'compared' to another scale, but adding unit normal noise to the training data doesn't make this much steeper. [image: image.png] I guess saying the variance is somewhere between 1-10, vs 0.01-0.1, is sufficient? I guess this might be a nice thing about GPs is that it isn't too hypersensitive to hyperparameters? You've also got a lot of data there, maybe that's helping? Mike PS It maybe depends on the kernel? The EQ kernel is maybe particularly forgiving? Maybe if you had two kernels added together the interplay of the two variances might matter more?

…

On Tue, 22 Feb 2022 at 11:57, JonWang ***@***.***> wrote: Hi there, I was experimenting with how GPy captures the underlying process. I simulated many Gaussian Processes by using known *rbf* hyper-parameters, and tried to see if GPy model made meaningful inference about the hyper-parameters. The starting point is very simple, with a noiseless simulation: def arbKer(var, l): # Arbitrary kernel with variance and lengthscale # RBF form kernel = GPy.kern.RBF(input_dim=1, variance=var, lengthscale=l) return kernel def priorGPclean(X, var, l, nsamp, show=True): # Simulate a bunch of noiseless/clean kernel # GP controlled noise drawn from a covariance matrix generated from kernel k = arbKer(var, l) mu = np.zeros((len(X))) # vector of the means C = k.K(X,X) # covariance matrix Y = np.random.multivariate_normal(mu,C,nsamp) Y = np.transpose(Y) return (Y) X = np.random.rand(100)[:,None]; X=X*15-5 Y = priorGPclean(X, 4.5, 2., nsamp=1, show=False) Now, clearly, according to the figure below, the very dense sampling from this simulated GP curve does not lead to an inference any closer to the known variance: var = 4.5, and how come such pattern led to an inference of var=8.5? [image: image] <https://user-images.githubusercontent.com/24724197/155125702-2d486565-b5a0-4576-80e4-5c88f96f2843.png> Could anyone please give me a hint? Thank you very much! — Reply to this email directly, view it on GitHub <#967>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB4MGQAX5IVXFRW5WBREW6DU4N24FANCNFSM5PBFLBSA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[jonwangio]

@lionfish0 Thanks for answering.

"I think quite often the model is quite insensitive to the kernel variance, within one-order-of-magnitude."
Very much agreed. Maybe should always consider variance together with lengthscale? At least, I guess fixing the lengthscale and sampling from GP with various variance within one-order-of-magnitude may result very similar sample GPs.

Also adding kernels makes sense. I will try to add another kernel to my experiment.

I consider this question has been answered. Thanks again.