JS: Accessing all assets located within sub folders from parent

#1

Matrix Version: 5.5

Hi,
I have been using the following to create javascript variables for listing assets and their metadata fields, for use as part of a bigger JSON file.

%globals_asset_children:111111^as_asset:asset_name,asset_assetid,asset_url,asset_type_icon,asset_data,asset_created_short,asset_data_metadata%;

(where 111111 indicates folder number)

This is working fine for listing the assets within the folder. However, if the folder contains another sub folder, which has all of the final assets, the above code will only output the sub folder name.

Is there a way to access the assets located within sub folders, without creating new variables for each sub folder?

IE: Accessing the asset called policy.doc from within the below structure

Parent Folder (#111111)
-Sub Folder (#111222)
–policy.doc (#222222)

The following will work - but is referencing the sub folder:
%globals_asset_children:111222^as_asset:asset_name,asset_assetid,asset_url,asset_type_icon,asset_data,asset_created_short,asset_data_metadata%;

How can I achieve the same, but from the the parent folder?

Thank you

(John gill) #2

image

I was about to answer “no” but it turns out the answer is “kinda”.

I hadn’t previously found a solution to “list multiple levels of assets with only keywords”, I always resorted to SSJS solutions like Hierarchical asset list with server side javascript (SSJ)

Sample asset setup

image

Normally you get stuck at the point where you can get attributes of the children (like your example)

%asset_children^as_asset:asset_name% = ["Page Contents","A","B","C"]

or you can get the assetids of the grandchildren

%asset_children^as_asset:asset_children% = [["25098"],["24743","24744","24745"],["24746","24747","24748"],["24749","24750","24751"]]

but you can’t get the attributes of the grandchildren, because by that point you’ve an array of arrays of assetids and you can’t use ^as_asset on such a structure.

What I hadn’t considered before is the possibility of operating on this array structure as a string. It turns out that the ^as_csv modifier will turn the above “array of arrays” into something like

%asset_children^as_asset:asset_children^as_csv%
"0"
"25098"
"24743","24744","24745"
"24746","24747","24748"
"24749","24750","24751"

if you could remove the newlines, add some commas, and strip the quotes, you’d be left with something that you might be able to turn back into an array. I don’t think this can be done with the basic ^replace modifier so you’ll need to create a Regular Expression asset and use the ^preg_replace modifier

I created #25100 with the following set up

image
image.png787x251 1.91 KB

so new lines become commas and double quotes get removed. That leaves you with

%asset_children^as_asset:asset_children^as_csv^preg_replace:25100%
0,25098,24743,24744,24745,24746,24747,24748,24749,24750,24751

which you can turn back into an array with ^explode. Once it’s an array again, you can do another round of ^as_asset

%asset_children^as_asset:asset_children^as_csv^preg_replace:25100^explode:,^as_asset:asset_name%
["Content Container","Apple","Acacia","Aardvark","Banana","Balsa","Baboon","Cherry","Cypress","Camel"]

Leaving you with a final flattened array of attributes of the grandchild assets. This isn’t quite what you’re after because it’s only the grandchildren, not the children. You might be able to tweak it to include the child level as well.

image

(John gill) #3

Got it. A better version that has the children and grandchildren in a single mega-keyword.

%asset_children^as_asset:asset_assetid,asset_children^as_json^preg_replace:25101^explode:,^as_asset:asset_name%

["Page Contents","Content Container","A","Apple","Acacia","Aardvark","B","Banana","Balsa","Baboon","C","Cherry","Cypress","Camel"]

Instead of using ^to_csv this uses ^to_json and a different Regular Expression asset to strip it back to a valid array of assetids.

Two regexes in the Regular Expression asset, both replacing with nothing

  • /"[a-z_]*":/
  • /[\[\]\{\}\"]/

image

(Nic Hubbard) #4

@JohnGill My question is, how do you have time to write these detailed answers? :slight_smile:

#5

Thank you @JohnGill for working on some solutions.
Have managed to get the first solution working to a point, but not the second. Second is throwing an error in the console. Unexpected token ; which I am trying to work backwards from to see at which point this occurs.
Thank you again.

(John gill) #6

The trick is to be so bad at keeping notes that you rely on the forum to be a knowledge base of Matrix stuff you used to know. The user I’m helping out is future-me.

I’m actually pretty stoked about this one, I’ve been wanting a two-level listing keyword for a while. Should be handy for some SSJS stuff.

Unexpected token sounds like a V8 error rather than a Matrix error, so I assume you’re using the keyword in SSJS? The keyword is likely evaluating to blank and you’re ending up with var blah = ;. Double check the assetid of the Regular Expression asset?

My usual approach for debugging keywords is to spam out all the intermediate stages, e.g.

<pre>
%asset_children^as_asset:asset_assetid,asset_children^as_json%
%asset_children^as_asset:asset_assetid,asset_children^as_json^preg_replace:25101%
%asset_children^as_asset:asset_assetid,asset_children^as_json^preg_replace:25101^explode:,%
%asset_children^as_asset:asset_assetid,asset_children^as_json^preg_replace:25101^explode:,^as_asset:asset_name%
</pre>
#7

Useful, thanks.

Have to ask though, what is the benefit of trying to get all of this into a single complex keyword rather than piggy-backing on a generic asset listing, accepting a root node via GET.

Regardless of whether you bake the desired attributes into the Type Format:

%globals_asset_contents_raw:1234^with_get:root=5678

or have the asset listing produce an array of asset IDs and provide attributes via the keyword:

%globals_asset_contents_raw:1234^with_get:root=5678^json_decode^as_asset:asset_name%

seems easier to write and later comprehend (without having to return here :wink: )
If it is down to performance, or simply a thought exercise to prove that you can, I understand.

(John gill) #8

Excellent question. That is indeed the approach I would take in the absence of a frankenkeyword, and have in the past.

The advantages of the keyword approach are:

Order of execution

Anytime you need to use %globals_asset_contents_raw you complicate things because that keyword is not necessarily processed at the same time as %asset_ keywords.

  • This leads to the very common issue of trying to use %globals_asset_contents_raw:1234^with_get:root={asset_metadata_field}%, which does not work.
  • The beauty of the keyword modifier approach is it will be processed whenever the parent value you are using would have been processed anyway.

Fewer caches (well, cache entries) involved

  • The asset listing #1234 gets its own Matrix Cache entry.
  • I hate caches, I’ve been bitten before by nesting the same asset multiple times on a page with different parameters.
    • I believe this issue was patched, but once bitten thrice shy. The fewer caches the better.

Simpler “global” infrastructure

  • 1 regular expression asset vs 13 assets that make up an Asset Listing Page
  • Regular Expression feels safer to share globally, you’re never going to be tempted to change it and there are fewer knobs and switches than an Asset Listing Page.
  • Probably just OCD, but I prefer it.

Performance might be an advantage, but if it is I doubt it’s significant. I haven’t tested it but I’d bet the keyword is very slightly faster or at least no slower.

Disadvantages of the keyword approach

The main disadvantage I can think of is if there are many assets to be listed.

The keyword knows nothing about paging so if you throw it a folder of 100 assets each of which contains 100 assets, it will probably just hang until PHP kills the process. The Asset Listing Page would use the Page Size setting.

(Although you could probably implement an equivalent limit in the regular expression with some fancy comma counting)

(John gill) #9

The performance comparison for which nobody asked

I got curious about the performance angle so I set up a test area of 110 assets (10 folders, each of which contains 10 folders). The Asset Listing Page version that @bkelly described I have set up with “Asset Map” sort order to make sure it spits out the same JSON as the keyword version.

I did two comparisons, one where the keywords were spitting out only the asset_assetid for each asset, and one spitting out all the fields that @pixelmedia had in their original query asset_name,asset_assetid,asset_url,asset_type_icon,asset_data,asset_created_short,asset_data_metadata

I did 8 tests of each on an otherwise bare page using /_performance

110 assets x just asset_id

ALP

Total Time 0.22s,  System 0.05s (22%), Queries: 0.13s (583)
Total Time 0.22s,  System 0.05s (22%), Queries: 0.13s (583)
Total Time 0.22s,  System 0.05s (24%), Queries: 0.14s (583)
Total Time 0.23s,  System 0.05s (21%), Queries: 0.15s (583)
Total Time 0.22s,  System 0.05s (22%), Queries: 0.14s (583)
Total Time 0.21s,  System 0.05s (22%), Queries: 0.13s (583)
Total Time 0.23s,  System 0.05s (22%), Queries: 0.14s (583)
Total Time 0.22s,  System 0.05s (23%), Queries: 0.13s (583)

Keyword

Total Time 0.15s,  System 0.02s (12%), Queries: 0.11s (544)
Total Time 0.15s,  System 0.02s (13%), Queries: 0.11s (544)
Total Time 0.15s,  System 0.02s (12%), Queries: 0.11s (544)
Total Time 0.16s,  System 0.02s (12%), Queries: 0.12s (544)
Total Time 0.16s,  System 0.02s (12%), Queries: 0.12s (544)
Total Time 0.15s,  System 0.02s (12%), Queries: 0.12s (544)
Total Time 0.19s,  System 0.02s (12%), Queries: 0.15s (544)
Total Time 0.16s,  System 0.02s (12%), Queries: 0.11s (544)

110 assets x lots of fields

ALP

Total Time 1.65s,  System 1.39s (84%), Queries: 1.17s (2272)
Total Time 1.47s,  System 1.27s (86%), Queries: 1.08s (2272)
Total Time 1.32s,  System 1.14s (87%), Queries: 0.96s (2272)
Total Time 1.64s,  System 1.47s (89%), Queries: 1.17s (2272)
Total Time 1.36s,  System 1.18s (87%), Queries: 0.99s (2272)
Total Time 1.51s,  System 1.3s (86%), Queries: 1.08s (2272)
Total Time 1.32s,  System 1.13s (86%), Queries: 0.97s (2272)
Total Time 1.36s,  System 1.19s (88%), Queries: 0.99s (2272)

Keyword

Total Time 1.31s,  System 0.03s (2%), Queries: 1s (2343)
Total Time 1.29s,  System 0.03s (2%), Queries: 0.99s (2343)
Total Time 1.34s,  System 0.03s (2%), Queries: 1.02s (2343)
Total Time 1.44s,  System 0.03s (2%), Queries: 1.11s (2343)
Total Time 1.45s,  System 0.03s (2%), Queries: 1.09s (2343)
Total Time 1.3s,  System 0.03s (2%), Queries: 1s (2343)
Total Time 1.29s,  System 0.03s (2%), Queries: 0.99s (2343)
Total Time 1.29s,  System 0.03s (2%), Queries: 0.98s (2343)

When you’re not getting much data about each asset, the keyword method has a decent edge, but once you include useful fields in the output the gap really closes and the asset listing version ends up using fewer queries (although is still marginally slower).

A note on asset_children_link_type_1

While I was doing this test I noticed an issue in the regular expression set up. It had been using /"[a-z_]*":/ to pull out the "asset_children": from the json, but once I restricted the link type that stopped working because it didn’t support digits in the json property "asset_children_link_type_1":.

To make this method safe for type restricted keywords you need to change the regular expression to replace /"[a-z_12]*":/ instead.

#10

Thanks for running the performance report. The solution I am working on would be for potentially a large number of assets, so this is good to note.
Would be interesting to see how it would compare against an asset listing, which is able to feed the info to a rest api and output as json. I haven’t looked into this as a possibility, but @JohnGill has got me thinking…
Thoughts?