Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add use_sparse_hessian functionality to prevent a dense Hessian from being instantiated #543

Closed
wants to merge 9 commits into from

Conversation

AlanTuQC
Copy link

When use_sparse_hessian=True, state.hessian is stored as a sparse COO matrix. Partially addresses Issue #485 -- the dataset with 4M+ columns can now run without a memory error.

@abudis
Copy link

abudis commented Feb 8, 2023

Hey Alan, thanks for looking into this! Is this planned to be merged in the near future? I'm experiencing a similar problem with a large sparse matrix.

@lbittarello
Copy link
Member

Please reopen when it's ready for review.

@zmbc
Copy link

zmbc commented Oct 29, 2024

My data is not much wider than it is long -- it is roughly square (and large). My sense is that this PR, if completed and merged, would make glum work for me -- or should I expect that glum's solvers wouldn't work even if they could be run?

Even if not, @tbenthompson said on #485 that

the implementation of a solver more appropriate for [K >> N] would not be a huge undertaking and 75% of the pieces necessary already exist within glum

What is missing? I'd be happy to contribute something here, as I have yet to find a Python library that can handle this case without regularization.

zmbc added a commit to zmbc/glum that referenced this pull request Oct 29, 2024
@zmbc
Copy link

zmbc commented Oct 29, 2024

I started on this in #869, but I ran into this: Quantco/tabmat#17. I wonder if @ElizabethSantorellaQC might be able to offer some guidance. The text in that issue seems to mostly be focused on how to tell whether or not a sandwich product will be sparse, but even the option for the user to say "I know it will be, represent it as a sparse matrix" would be enough for my purposes.

@stanmart
Copy link
Collaborator

Thank you so much for looking into this feature! I agree, as a first step I would not worry about using heuristics to try to guess whether the hessian will be sparse, but just rely on the user telling glum whether it should be. (Btw I doubt Elizabeth will chime in on this as she hasn't worked on glum in a while.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants