Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ale analysis without model #9

Open
snvv opened this issue Dec 24, 2024 · 1 comment
Open

Ale analysis without model #9

snvv opened this issue Dec 24, 2024 · 1 comment
Labels
question Further information is requested

Comments

@snvv
Copy link

snvv commented Dec 24, 2024

Is it possible to create plots and statistical tables without the model itself? This might be achievable by providing the ale function with the following inputs: the feature matrix, the response vector, and the predictions for the response vector. This approach would particularly facilitate multi-period analysis. Furthermore, there is no need to specify a separate prediction function for models with sparse matrices or complex structures (e.g., mixed-effects models). Is this feasible?

This is not an issue (The program works correctly under various scenarios) but a request for a feature.

Best regards

@tripartio
Copy link
Owner

tripartio commented Dec 25, 2024

Yes and no.

No: ALE by definition describes a model. It does not describe a dataset directly. So, it is meaningless to talk about ALE without a model input. In fact, what you are describing, "the predictions for the response vector", necessarily implies a model--the predictions came from a model. They did not drop down from the sky,

Yes: Sure, you could set up a dummy "model" that simply returns predictions that came from somewhere, one prediction per line of data. But the crucial part of the ALE algorithm involves changing the original data in specific ways; that is how it determines the effects of the input variables. If you have a dummy "model" that returns the same prediction no matter what you do to the line of data, then the ALE algorithm will calculate an ALE effect of exactly zero for every single variable and the ALE statistics would all be exactly zero, too. So, even if you could work around ALE's requirement of an input model, the result of zero ALE would be useless.

I don't understand why you might want this. You mentioned, "This approach would particularly facilitate multi-period analysis." Could you please be more explicit about exactly what you are hoping you might get from ALE without an input model?

Concerning, "Furthermore, there is no need to specify a separate prediction function for models with sparse matrices or complex structures (e.g., mixed-effects models). Is this feasible?", I simply don't see any way around this requirement for a model-agnostic procedure like ALE--this is the only way that an ALE implementation can support any model that is provided to it. The alternative would be that it could only support some model types but not others, which is definitely not what we want.

Note that the ale implementation supports the default stats::predict() method by default without requiring a custom predict function. It is only for R packages that do not implement this standard R approach for the predict method that require a custom predict function. So, the approach of the ale package is:

  • If a package uses the standard R approach that implements stats::predict(), the ale package works smoothly with no custom predict function.
  • If a package does things its own way, then the ale package will still support its models but then users need to explain the custom way that the package's model predictions work by creating a custom predict function.

I think that's a very reasonable approach, but please let me know if you have ideas for making the package easier to use. I am actively developing it.

@tripartio tripartio added the question Further information is requested label Jan 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

2 participants