ENH: Add gaussian process DWI signal representation notebook #202

jhlegarreta · 2024-06-04T23:34:22Z

Add gaussian process DWI signal representation notebook.

jhlegarreta · 2024-06-04T23:41:01Z

Depends on #188. @esavary If you did changes to the notebook, maybe you can do a PR to this branch with such changes, and I would merge it right away.

With the default kernels, I get this error plot, so the default kernels are not a good choice for the DWI signal:

Not sure if the signal not being normalized plays a role here.

esavary · 2024-06-05T13:44:23Z

I made a pull request against your branch with some suggestions. Essentially, it involves not fitting one voxel at a time. This ensures that the same hyperparameters are used for the kernel across all voxels. It also improves their estimation when done with more voxels (this is done inside the fit method). These changes result in a slight improvement. However, even with these changes, the optimization is still failing for the parameter k2__noise_level (from the WhiteNoise kernel). It hits the bounds, leading to a significant error, as shown in this plot (I used the default hyperparameters optimization from the fit method of sklearn not the custom one I coded):

Additionally, I tried fitting a larger cube, but it seems to be similar as with the SphericalKernel. See the image below showing the prediction:

An additional comment I have is that we might consider using the angles between the directions instead of bvecs directly, as suggested in Andersson (2015). However, I was unsure if that was only relevant when using the Spherical or Exponential kernel, so I did not include that change.

esavary · 2024-06-05T13:55:55Z

Depends on #188. @esavary If you did changes to the notebook, maybe you can do a PR to this branch with such changes, and I would merge it right away.

With the default kernels, I get this error plot, so the default kernels are not a good choice for the DWI signal:

Not sure if the signal not being normalized plays a role here.

Actually, I've just noticed something in your plot, I think the ground truth signal should not be zero (but not sure if it causes the problem). Could it be that the data you used for the test is empty except for the first gradient direction? ( I used this data for my plot: https://osf.io/download/6at98/)

jhlegarreta · 2024-06-08T17:57:15Z

Merged your PR into the branch.

Maybe we can now uncomment the orthoview calls and provide with the same values you used to generate the figures you pasted.

An additional comment I have is that we might consider using the angles between the directions instead of bvecs directly, as suggested in Andersson (2015). However, I was unsure if that was only relevant when using the Spherical or Exponential kernel, so I did not include that change.

In my understanding we want to predict on bvecs. We want to represent a given DWI volume using a GP, e.g. as if we were using another basis (e.g. SH) to represent it and be able to recreate it within some tolerance. The ISMRM paper/poster figure is helpful in this.

Actually, I've just noticed something in your plot, I think the ground truth signal should not be zero (but not sure if it causes the problem). Could it be that the data you used for the test is empty except for the first gradient direction? ( I used this data for my plot: https://osf.io/download/6at98/)

Maybe the volume I chose included some background voxels.

jhlegarreta · 2024-06-10T01:21:16Z

Maybe we should normalize the DWI signal (divide each voxel signal by the B0 signal in that location) prior to fitting the data. Not sure if this would have a large influence on the results, but maybe worthwhile trying.

jhlegarreta · 2024-06-15T17:52:49Z

Added a new notebook with a simulated signal using a multi-tensor model.

@arokem In an effort to get something like fig 4 in Andersson 2015 NIMG, I am getting the below figure for the signal I am simulating in docs/notebooks/dwi_simulated_gp.ipynb.

I would expects the diffusion signals to roughly describe the two crossing tensors, but what I get is a blob. Tensors are at 90 degrees, and the fODF I get from them shows so:

What am I missing? Thanks.

arokem · 2024-07-01T02:11:51Z

Sorry for the slowness. One quick thought is that Andersson et al. used a b=3,000 in their simulations and I see that you are using b=1,000. Another is that I suspect that the Andersson figures were generated with a transformation to diffusion distance, rather than in the original signal domain. Indeed, a crossing fiber signal would look like the linear combination of two "donuts" rather than the linear combination of two "cigars", because the signal looks like a donut (see e.g., figure 1 in my 2014 paper), and when you add these donuts together, especially in low b-values, you get something that more closely resembles a "blob". To get "cigars" you need to do something like this: https://github.com/arokem/osmosis/blob/72d848ddc172ce6a72baba4c0d4cf3675db1a381/osmosis/tensor.py#L233-L248

jhlegarreta · 2024-07-06T18:01:54Z

Removed the signal plot (#202 (comment)) for now so that it does not distract us as we progress with the kernels, etc. Will add it in a separate PR on top of this branch.

oesteban · 2024-07-10T08:03:31Z

@jhlegarreta can you commit with the cells' output so that we can use github to visualize the results? It's tough to look at the diff, but we workaround the problem of the HCP data.

jhlegarreta · 2024-07-10T23:51:46Z

@jhlegarreta can you commit with the cells' output so that we can use github to visualize the results? It's tough to look at the diff, but we workaround the problem of the HCP data.

Brought back the cell in the last push force:
https://github.com/nipreps/eddymotion/blob/40baac4d3225060bc83ab5daf95a026c10f115c9/docs/notebooks/dwi_simulated_gp.ipynb

Note that in PR #217 I had tried with the approach Ariel suggested but did not get the expected shape.

jhlegarreta · 2024-07-12T01:58:24Z

docs/notebooks/dwi_simulated_gp.ipynb

+    "# Create an interactive plot\n",
+    "%matplotlib notebook\n",
+    "\n",
+    "fig = plot_dwi_scatter(signal_repr)\n",


Shows two disks, so not quite right yet.

Installed ipympl to be able to interact with (rotate) the mpl figures.

jhlegarreta · 2024-07-12T01:58:53Z

docs/notebooks/dwi_simulated_gp.ipynb

+   "cell_type": "code",
+   "source": [
+    "# ToDo\n",
+    "# Plot as surface"


Have not succeeded in getting this right.

jhlegarreta · 2024-07-12T01:59:49Z

docs/notebooks/dwi_simulated_gp.ipynb

+    "ax.set_xlabel(\"bvec indices\")\n",
+    "ax.set_ylabel(\"signal\")\n",
+    "ax.legend()\n",
+    "plt.title(\"Gaussian process regression on dataset\")\n",


The plot looks perfect because I am predicting on the very same data that I used for fitting.

jhlegarreta · 2024-08-11T22:49:08Z

A few notes:

Have not investigated the optimization convergence error shown on the notebok (ABNORMAL_TERMINATION_IN_LNSRCH.): when training and predicting on all bvecs (except b0) the warning was being shown, but not the error.
I think I have figured out how to properly create the surface plots of the diffusion signal (which could be used for the prediction) and have it in a separate piece of code. When we take the time to discuss about how confident we are about the GP (e.g. the differences wrt the previous implementation, etc.), and other issues (example organisation, data and code to generate the empirical covariance plot) we can bring that part.

oesteban · 2024-08-27T19:05:02Z

Is there anything I can do here before the TechMon meeting?

jhlegarreta · 2024-08-27T22:02:11Z

@oesteban I apologise for not having worked on this in the last few weeks; I expect to be able to have another look around the tech meeting date. Sorry.

The single point prediction looks to be working, but I am not sure about the source of the error message mentioned in #202 (comment), so I need to investigate that. I would like to pick other random b-vecs to see if the prediction is OK or whether the convergence message persists.

Additionally, I would like to investigate what has changed wrt the implementation that was not working. I need to prepare
multi-fiber settings, and add the predicted point visualization part. So all this is on my roof; if you feel like, you can run the notebook and see if my impression that it is working seems OK (despite the convergence message).

Probably we can talk about all these experiments/design (esp. when we will add experiments with acquired data) in our eventual meeting.

Add gaussian process DWI signal representation notebooks: - One of the notebooks uses a simulated DWI signal. - The second notebook uses a real DWI signal.

Add singla plot cells.

Experiment using a single tensor in the GP notebook.

jhlegarreta · 2024-09-25T01:03:01Z

As things stand now, I would split the PR into 2 PRs: one with the notebook that uses a simulated fiber and another one the one that uses real data. The first one is self-contained and finished IMO: I would not complicate more the notebook.

oesteban · 2024-10-23T06:44:54Z

As things stand now, I would split the PR into 2 PRs: one with the notebook that uses a simulated fiber and another one the one that uses real data. The first one is self-contained and finished IMO: I would not complicate more the notebook.

I agree with this take. Should we close this one and go ahead with two fresh PRs? (feel free to reopen if I rushed closing).

jhlegarreta force-pushed the AddDWIGPNotebook branch from 7ad606e to c7d6897 Compare June 4, 2024 23:36

jhlegarreta mentioned this pull request Jun 8, 2024

ENH: Implement Gaussian Process #188

Merged

oesteban mentioned this pull request Jun 13, 2024

hyperparameters optimization jhlegarreta/eddymotion#2

Closed

jhlegarreta force-pushed the AddDWIGPNotebook branch 2 times, most recently from 7d89b3c to a8742d9 Compare June 15, 2024 17:36

jhlegarreta force-pushed the AddDWIGPNotebook branch from a8742d9 to f363635 Compare July 1, 2024 00:42

jhlegarreta force-pushed the AddDWIGPNotebook branch from f363635 to b1fb2bb Compare July 6, 2024 18:00

jhlegarreta mentioned this pull request Jul 6, 2024

ENH: Use the attenuation distance to plot the DWI signal #217

Draft

jhlegarreta force-pushed the AddDWIGPNotebook branch from b1fb2bb to 40baac4 Compare July 10, 2024 23:50

jhlegarreta force-pushed the AddDWIGPNotebook branch from 40baac4 to cc14404 Compare July 12, 2024 01:53

jhlegarreta commented Jul 12, 2024

View reviewed changes

jhlegarreta force-pushed the AddDWIGPNotebook branch 3 times, most recently from 5d17dca to bb98ca2 Compare August 11, 2024 22:39

ENH: Add gaussian process DWI signal representation notebooks

84ae26d

Add gaussian process DWI signal representation notebooks: - One of the notebooks uses a simulated DWI signal. - The second notebook uses a real DWI signal.

ENH: Add signal plot cells.

6943ca2

Add singla plot cells.

jhlegarreta force-pushed the AddDWIGPNotebook branch from bb98ca2 to edab295 Compare September 16, 2024 01:36

ENH: Experiment using a single tensor in GP notebook

8e6f199

Experiment using a single tensor in the GP notebook.

jhlegarreta force-pushed the AddDWIGPNotebook branch from edab295 to 8e6f199 Compare September 25, 2024 00:59

oesteban closed this Oct 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH: Add gaussian process DWI signal representation notebook #202

ENH: Add gaussian process DWI signal representation notebook #202

jhlegarreta commented Jun 4, 2024

jhlegarreta commented Jun 4, 2024

esavary commented Jun 5, 2024

esavary commented Jun 5, 2024 •

edited

Loading

jhlegarreta commented Jun 8, 2024

jhlegarreta commented Jun 10, 2024

jhlegarreta commented Jun 15, 2024

arokem commented Jul 1, 2024

jhlegarreta commented Jul 6, 2024

oesteban commented Jul 10, 2024

jhlegarreta commented Jul 10, 2024 •

edited

Loading

jhlegarreta Jul 12, 2024 •

edited

Loading

jhlegarreta Jul 12, 2024

jhlegarreta Jul 12, 2024

jhlegarreta commented Aug 11, 2024 •

edited

Loading

oesteban commented Aug 27, 2024

jhlegarreta commented Aug 27, 2024

jhlegarreta commented Sep 25, 2024

oesteban commented Oct 23, 2024

ENH: Add gaussian process DWI signal representation notebook #202

ENH: Add gaussian process DWI signal representation notebook #202

Conversation

jhlegarreta commented Jun 4, 2024

jhlegarreta commented Jun 4, 2024

esavary commented Jun 5, 2024

esavary commented Jun 5, 2024 • edited Loading

jhlegarreta commented Jun 8, 2024

jhlegarreta commented Jun 10, 2024

jhlegarreta commented Jun 15, 2024

arokem commented Jul 1, 2024

jhlegarreta commented Jul 6, 2024

oesteban commented Jul 10, 2024

jhlegarreta commented Jul 10, 2024 • edited Loading

jhlegarreta Jul 12, 2024 • edited Loading

Choose a reason for hiding this comment

jhlegarreta Jul 12, 2024

Choose a reason for hiding this comment

jhlegarreta Jul 12, 2024

Choose a reason for hiding this comment

jhlegarreta commented Aug 11, 2024 • edited Loading

oesteban commented Aug 27, 2024

jhlegarreta commented Aug 27, 2024

jhlegarreta commented Sep 25, 2024

oesteban commented Oct 23, 2024

esavary commented Jun 5, 2024 •

edited

Loading

jhlegarreta commented Jul 10, 2024 •

edited

Loading

jhlegarreta Jul 12, 2024 •

edited

Loading

jhlegarreta commented Aug 11, 2024 •

edited

Loading