Paginating with proxy routes

[mkleywegt]

Hi Aupajo,

It is indeed getting rather specific. I realized that what I am trying to paginate are not really pages per se (they are now they have a route assigned to them). The payoff to this, I guess, would be having flexible general purpose pagination module for Middleman.

I will add an example for this later today.

Below is the original post from the Middleman Forum:

Background:

• I list resources (patterns) that live in a yaml file. For the purpose of paginating them I gave them a route with the proxy method ('single pattern'). The index files are also defined with a proxy route ('patterns').
• I have it set up so that the main route ^/patterns (which is the parent of the proxy routes 'patterns') transparently displays all its children's listings (using the same template). Navigating from the parent to a child thus narrows down the list of resources.

Here are my findings:

• Both frontmatter and matching the url result in the right index files being discovered by the extension ('Creating new resources for index'). However, the frontmatter method fails for me. It doesn't get to displaying a stack trace, simply showing 'Page not found' instead. Directory indexes are activated, but disabling makes no difference. So working from the url method, the pagination gets displayed.
• The pagination object is indeed passed on to the index. But in my case, it spits out the entire site map (<middleman::pagination::inpagecontext etc.>). It might not be able to work out the query, so it returns all the data. Subpages were not available as a result.

I have experimented with including a variable to the pageable directive, with no notable difference.

I also tried to activate pagination within the loop that defines the proxy. Middleman then reports it already activated pagination (as it tries to do it a couple of times), also without a notable difference.

[mkleywegt]

So, here is the relevant part of my config.rb

data.projects.details.each do |project|
  proxy "/#{project[:slug]}", "/localizable/project.html", :locals => { :project => project }, locale: :en, :ignore => false

  proxy "/#{project[:slug]}/patterns", "/localizable/patterns.html", :locals => { :project => project, :haspatterns => true }, locale: :en, :ignore => false

  project.patterns.each do |pattern|
    proxy "/#{project[:slug]}/patterns/#{pattern[:bundle]}", "/localizable/patterns.html", :locals => { :pattern => pattern, :project => project }, locale: :en, :ignore => false 
  end

  project.patterns.each do |pattern|
    p = data.patterns.details.find{ |x| x['bundle'] == pattern[:bundle]}

    p.details.each do |single|
      proxy "/#{project[:slug]}/patterns/#{pattern[:bundle]}/#{single[:slug]}", "/localizable/pattern_single.html", :locals => { :single => single, :pattern => pattern, :project => project }, locale: :en, :ignore => false
    end
  end
end
activate :pagination do
  pageable :so do |page|
  # Match any page whose URL includes "/patterns/"
  page.path.start_with?('patterns/')
  #page.data.so.present?
  end
end

As you can see, there are some variables involved that are used to pull data out of yaml files (in the templates as well).

In the template for patterns.html I have the pagination frontmatter:

---

pagination:
for: so
per_page: 2

---

Which is correctly read out by the extension. (Showing 2 per page)
The amount of pages currently in the set cannot be determined, as are the previous and back links. So, the status is 'Page 1 of 0', all the links point to the current page, and it is not paginating the patterns.

[Aupajo]

Thanks, @mkleywegt, this is great. I think I can help you out.

I'll reproduce this first moment I get a chance and diagnose it from there.

Can you post an example of your data/projects.yml?

[mkleywegt]

Hi @Aupajo, thanks again for looking at it.

These lists get unwieldy, so I've abridged the contents.

Here's projects.yaml

---
details:
  - id: "1"
    name: "First"
    slug: "first"
    patterns:
      - id: "1"
        bundle: "main"
      - id: "2"
        bundle: "dashboard"
      - id: "3"
        bundle: "users"
  - id: "2"
    name: "Second"
    slug: "second"
    patterns:
      - id: "1"
        bundle: "main"
      - id: "2"
        bundle: "additional"
      - id: "3"
        bundle: "users"

And patterns.yaml

---
details:
  - bundle: "main"
    name: "Main"
    details:
      - id: "1"
        name: "First Button"
        slug: "first-button"
      - id: "5"
        name: "Fifth Button"
        slug: "fifth-button"
      - id: "9"
        name: "Nineth Button"
        slug: "nineth-button"
      - id: "10"
        name: "Tenth Button"
        slug: "tenth-button"
  - bundle: "additional"
    name: "Additional"
    details:
      - id: "2"
        name: "Second Button"
        slug: "second-button"
      - id: "6"
        name: "Sixth Button"
        slug: "sixth-button"
  - bundle: "dashboard"
    name: "Dashboard"
    details:
      - id: "3"
        name: "Third Button"
        slug: "third-button"
      - id: "7"
        name: "Seventh Button"
        slug: "seventh-button" 
  - bundle: "users"
    name: "Users"
    details:
      - id: "4"
        name: "Fourth Button"
        slug: "fourth-button"
      - id: "8"
        name: "Eighth Button"
        slug: "eighth-button"

Let me know if you need anything else.

[Aupajo]

Hey, sorry it's taken me a little while to get round to this issue.

I've successfully reproduced the issue with the configuration you've supplied, which is good news as it means it'll be easier for me to diagnose the problem.

As a side note, I found the YAML/proxy configuration set-up very hard to follow. If you're specifying slugs and making cross-references, you might find it easier to store the data in separate templates rather than in the YAML.

For instance, you could have a set up that looks like:

source/
   - layouts/
     - layout.erb
     - project.erb
     - pattern.erb
   - projects/
      - first.html.erb
      - second.html.erb
   - patterns/
      - main.html.erb
      - additional.html.erb

projects/first.html.erb can look like this:

---
name: First
patterns: [main, additional]
---

You might find it cleaner.

You might find this technique for extracting objects in Middleman of use, too.

Stay tuned for the fix with the existing set-up, though.

[Aupajo]

Okay, there are two steps to getting this working.

The first is with the current configuration. The solution is to have a separate pageable resource for each proxy set. Basically, the trick is to keep track of the different sets in a variable outside the data.projects.each block and share it with the activate :pagination block. You'd also need to pass in a unique name for each pageable resource and modify the pagination metadata in each proxied resource. This is a really complicated set-up.

The second is the fact you're using directory-index-style paths in your proxies (e.g. the route is /my-project-name rather than /my-project-name.html). That will prevent Middleman Pagination from being able to correctly generate the secondary index pages, but I consider it a common enough (mis)use of proxy to consider it a bug with Middleman Pagination, so I'll introduce a fix for it.

Once that fix is in place, I'll try to paste a working code example for you of how you can solve the first problem.

[mkleywegt]

Thank you for your suggestions. My inclination towards the YAML-heavy setup is easy of maintenance (hmm) by preventing duplicating templates (there would not have to be any difference between e.g. first.html.erb and second.html.erb). I would also be able to add and remove paginated items without having to touch the source folder.

Having said that, I agree with your sentiment that it is hard to follow and certainly not clean enough (needing slugs et cetera). I will look into refactoring that. Extracting objects will help there too. I'll wait for your fix.

[Aupajo]

@mkleywegt Thanks. You wouldn't have to have any duplication with the approach above (you can use layouts in place of proxied files).

[Aupajo]

@mkleywegt Thanks for waiting! It took sitting down for a couple of hours to hash it out :D

I've got a basic working example for you at https://github.com/Aupajo/middleman-example-for-mkleywegt

It's fairly procedural, and it might be missing a layer of pagination indexes, but hopefully it's enough for you to go on.

Some subtle things to note:

• This is using the middleman-pagination 1.0.6 gem (updated now the 1.0.7 gem)
• Proxy paths are not preceded with slashes
• Proxy paths end in .html
• Pagination can now be specified in the metadata hash that's passed to the proxy (see https://github.com/Aupajo/middleman-example-for-mkleywegt/blob/master/config.rb#L37)

Hope that helps!

[mkleywegt]

@Aupajo - that is great work. Very much appreciated.

Re: proxy paths ending in .html — while the .html is included in the proxy route in the config.rb (as per your example), visiting the proxy paths only works without appending .html .

With this addition, I have the setup pretty much working. I posted a specific problem over in the other repository, see Memory leak (unless you'd like that here).

[mkleywegt]

And I noticed that paginated subpages use the lay-out assigned to the first page, but they don't have access to local variables. Where would I pass those?