Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using incremental materialisation of cost_per_query model over-allocates cost per query #165

Open
bisset-a opened this issue Oct 1, 2024 · 0 comments

Comments

@bisset-a
Copy link

bisset-a commented Oct 1, 2024

Hey there!

We noticed the cost_per_query model seems to over-allocate cost to some queries.

I think the issue comes from the incremental logic in this line. This line returns a timestamp. Later in the model, cost is attributed based on the hour the query falls into. When the timestamp generated in L34 is close to the hour (e.g. 03:57). In this case, all warehouse cost for the hour between 3:00 and 4:00 will be allocated to the queries that ended between 03:57 and 4:00 (which causes an over-allocation as there won't be many queries that occur in this space of 3mins).

Aggregated query cost per day using the incremental materialization we have been running for ~4 months gives quite different results from a table materialization.

Let me know if you would like me to investigate further and raise a PR. Thanks so much!!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant