-
Notifications
You must be signed in to change notification settings - Fork 25
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add use_sparse_hessian
functionality to prevent a dense Hessian from being instantiated
#543
Conversation
Hey Alan, thanks for looking into this! Is this planned to be merged in the near future? I'm experiencing a similar problem with a large sparse matrix. |
Please reopen when it's ready for review. |
My data is not much wider than it is long -- it is roughly square (and large). My sense is that this PR, if completed and merged, would make glum work for me -- or should I expect that glum's solvers wouldn't work even if they could be run? Even if not, @tbenthompson said on #485 that
What is missing? I'd be happy to contribute something here, as I have yet to find a Python library that can handle this case without regularization. |
I started on this in #869, but I ran into this: Quantco/tabmat#17. I wonder if @ElizabethSantorellaQC might be able to offer some guidance. The text in that issue seems to mostly be focused on how to tell whether or not a sandwich product will be sparse, but even the option for the user to say "I know it will be, represent it as a sparse matrix" would be enough for my purposes. |
Thank you so much for looking into this feature! I agree, as a first step I would not worry about using heuristics to try to guess whether the hessian will be sparse, but just rely on the user telling glum whether it should be. (Btw I doubt Elizabeth will chime in on this as she hasn't worked on glum in a while.) |
When
use_sparse_hessian=True
,state.hessian
is stored as a sparse COO matrix. Partially addresses Issue #485 -- the dataset with 4M+ columns can now run without a memory error.