Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Terrible performance on datasets which have a high percentage of rows containing PII #204

Open
olorin opened this issue Apr 26, 2017 · 2 comments
Open

Terrible performance on datasets which have a high percentage of rows containing PII #204

olorin opened this issue Apr 26, 2017 · 2 comments

Comments

@olorin
Copy link
Contributor

olorin commented Apr 26, 2017

Not sure which yet - something to do with unicode? Details TBA.
@olorin
Copy link
Contributor Author

olorin commented Apr 27, 2017

It's the PII observation state, after number of observations gets above some threshold (presumably governed by the amount of PII vs non-PII rows) performance goes out the window (not sure how much exactly yet, but definitely superlinear). #202 related

@olorin olorin changed the title Terrible performance on some datasets Terrible performance on datasets which have a high percentage of rows containing PII Apr 27, 2017
@olorin
Copy link
Contributor Author

olorin commented Apr 27, 2017

Got it, updatePIIObservations is accidentally O(n) when it should be constant-time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@olorin