Optimizer window pagination

[Kitefiko]

Describe the Bug

I believe I have found multiple issues with window pagination implementation.

• AttributeError is always thrown in ListConnectionWithTotalCount.resolve_connection_from_cache
• backup solution in case of AttributeError prevents pagination
• order and thus pagination itself is non-deterministic when explicit order is not used
• usage of last returns everything and bypasses max_results

System Information

• Operating system: linux
• Strawberry version (if applicable): 0.47.1
• Database version: 10.11.6-MariaDB

Additional Context

ListConnectionWithTotalCount.resolve_connection_from_cache

AttributeError is always thrown here:

strawberry-django/strawberry_django/relay.py

Line 195 in d6051db

has_next_page = result._strawberry_row_number < result if result else False

I think there should be something like this?

has_next_page = result[-1]._strawberry_row_number < result[-1]._strawberry_total_count if result else False

Backup solution in case of AttributeError prevents pagination

This might be intended behavior, but:
In case of AttributeError pagination falls back to resolve_connection but (probably due to unavailable overfetch?) pageInfo's hasNextPage is wrongly reporting as false

Order and thus pagination itself is non-deterministic when explicit order is not used

This is probably the biggest issue. Maybe fixable via pk ordering in case of empty order_by in apply_window_pagination ?

Usage of last returns everything and bypasses max_results

When last is used on it's own and execution ends up here:

strawberry-django/strawberry_django/optimizer.py

Lines 561 to 573 in d6051db

	slice_metadata = SliceMetadata.from_arguments(
	Info(_raw_info=info, _field=field),
	first=field_kwargs.get("first"),
	last=field_kwargs.get("last"),
	before=field_kwargs.get("before"),
	after=field_kwargs.get("after"),
	)
	qs = apply_window_pagination(
	qs,
	related_field_id=related_field_id,
	offset=slice_metadata.start,
	limit=slice_metadata.end - slice_metadata.start,
	)

slice_metadata.start is evaluated to 0
slice_metadata.end becomes sys.maxsize

and queryset ends up filtering out nothing

Upvote & Fund

• We're using Polar.sh so you can upvote and help fund this issue.
• We receive the funding once the issue is completed & confirmed by you.
• Thank you in advance for helping prioritize & fund our backlog.

[bellini666]

Hey @Kitefiko ,

Hrm, I'm not really I understood the issue. When and why does AttributeError gets raised for you? Could you try to provide an MRE for it?

[SupImDos]

Hi @bellini666

We've run into these issue as well, particularly with ListConnectionWithTotalCount.resolve_connection_from_cache here:

strawberry-django/strawberry_django/relay.py

Line 195 in d6051db

has_next_page = result._strawberry_row_number < result if result else False

Just so its clear, the result object here is actually the QuerySet._result_cache list, so it will never have the annotated _strawberry_row_number / _strawberry_total_count attributes. It's items will though, so as suggested in parent issue it should instead be something like:

has_next_page = (
    result[-1]._strawberry_row_number < result[-1]._strawberry_total_count
    if result
    else False
)

[SupImDos]

@bellini666 FYI #622 only fixes 1 of the 4 problems raised in this issue, so it may be worth keeping this open?

[bellini666]

@SupImDos good point, going to reopen this

[bellini666]

@SupImDos @Kitefiko could any of you help me with some MREs for the remaining issues here so I can try to tackle?

[SupImDos]

@SupImDos @Kitefiko could any of you help me with some MREs for the remaining issues here so I can try to tackle?

@bellini666 Yes, we've specifically run into the "Usage of last returns everything and bypasses max_results" issue, so I should be able to put together an MRE for that soon.

[SupImDos]

@bellini666
Sorry for the delay! I've created an MRE for the "Usage of last returns everything and bypasses max_results" issue:

• Branch: mre/pagination-with-only-last
• Commit: 922ec47

In short, doing:

query {
  milestoneConn(last: 3) {
    edges {
      node {
        id
      }
    }
  }
}

Results in:

SELECT "projects_milestone"."name",
    "projects_milestone"."id",
    "projects_milestone"."due_date",
    "projects_milestone"."project_id"
FROM "projects_milestone" LIMIT 9223372036854775807    # <- This is ``sys.maxsize``

[bellini666]

Hi @SupImDos ,

Thanks for the MRE! 😊

Here is what is happening: The relay limit/offset algorithm on strawberry is implemented using slices, and those are calculated here: https://github.com/strawberry-graphql/strawberry/blob/main/strawberry/relay/utils.py#L126

If you take a look at this line, you will see that if you only ask for last without anything else, it will do a slice like nodes[: sys.maxint] and then indeed get the last results in python.

A good workaround would be to make this do nodes[-last:], but when I tried that, django raised ValueError: Negative indexing is not supported.

I think the only options we have to solve this specific case (we only have last and nothing else) would be:

1. Reverse the sorting in the queryset and change last to first

In theory, this would work the same, but then we would need to re-reverse the results after.

2. Call .count() in the queryset before the slice so we can do nodes[count - last:]

This one would not require re-reversing, but is somewhat more complicated to implement, as it would require changing the strawberry API to allow hooking into it.

What do you think? Anything that I'm missing here?