Package proposal - Custom Report Asset

(John gill) #1

tl;dr - A report asset that nests a listing_engine asset to define the contents of the report, decoupling generation from serving.

I semi-frequently find myself thinking “I’d love to provide that asset listing / search page to the users … but it’s too heavy to have them running on demand”. The Report asset types in Matrix (report_links, report_asset_statuses) already have a “generate” concept that decouples viewing the report from generating the report. This would be a useful feature to be able to leverage in custom reports.

I see the 5-minutes-consideration version looking something like a hollowed out copy of one of the built-in reports which accepts a link to another asset to define the actual data in the report.

  • A new asset type, with attributes to allow users to provide
    • A listing asset (Asset Listing or Search Page) which the user has already set up to produce the desired output
    • All the same options as a Nest Content Container to allow passing of parameters to the “nested” listing asset
    • Some kind of options to specify how frequently the report can be re-generated
  • On the admin side, the details screen would be much like an existing report providing
    • information on when the report was last generated
    • a button to allow regeneration
    • Ideally, some restriction to allow only “publishing” users (simple_edit_users?) to generate the report once per time window specified on the asset
  • At generation time, the report should be able to “walk” the asset listing to get the whole thing
    • e.g. I set up an Asset Listing over an area of 10,000 assets with 500 per page. The report should (optionally) fetch all 20 pages when generating, so it can serve the complete version to users. This would likely require a small new HIPO :hippopotamus:
  • On the frontend, the report would serve the data that it got from the listing asset at the previous generation
    • would need to store content-type header to make sure you could use this asset to spit out HTML/JSON/XML (determined by the design set up of the listing asset, or an option on the report asset itself)

This would really change the scope of what we are comfortable providing to our users, allowing more complex insights by leveraging partly-existing functionality and fitting in well with the existing report concept.

#2

Exactly what I need. I often use an assets listing that output JSON data and use it to drive a page as data source. All is good except for the server resources required to build the list because for this purpose I can’t paginate the list. What I do in the time being is to load the assets listing when I know the list has changed, download it as a JSON file and then upload it to Matrix as JSON file and use that instead of the listing.

Of course due to the manual process, this will not suit for listing that changes very often. So what is proposed here will fill that need perfectly.

(David Schoen) #3

I’d just prefer this handled via general HTTP semantics and ESIs like stale-while-revalidate (never waiting for the upstream to regenerate to show it the user) and TTLs (max-age/s-maxage).

I think as we transition to having ESI everywhere, it may be more practical to expose those controls in the UI more elegantly. They’re technically there now on most installs, but difficult to get at and understand.

(John gill) #4

I almost mentioned HTTP caching in the original post, I considered that approach but I don’t think it’s an answer for the kinds of use cases I’m thinking about

  • Primarily non-public reports - these would be utilities used by publishers and content workers.
    • Caching is challenging to reason about (as you noted) even for public URLs. Add in authentication and it’s a nightmare.
  • Publishers are aware of _nocache and _recache and not afraid to use them.
  • Cache Manager is restricted to System Administrators, so many “site admins” (backend users in a shared instance with admin perms over one site) do not have access to it
  • This might be what you were referring to with “expose these controls … more elegantly”.

I suspect that if I took care I could set up something pretty close to what I want with ESI and cache headers, but I wouldn’t be comfortable relying on it for an asset listing that takes 15 seconds per page. The existing report “generate” concept is clear and robust, and much more accessible as a result.

(Keith Brown) #5

I have used this script before to do something similar… not identical but does allow you to then include the outputted pre-compiled back in.

https://matrix.squiz.net/manuals/server-administrator/chapters/publish-static-content-script

K