Hi All,
I just found such an important thing about Matrix that all large website managers should know. Maybe its in the doco, maybe it isnt, but we werent aware of it any maybe you arent as well.
I just set up an page that lists books on our site (a total of about 90 items). As I was testing I was very surprised with how fast the page was being rendered. I was using the _nocache suffix and it was taking about 3 seconds to render - nice.
I hadnt changed the type formats from the default at all so it was just using %asset_name_linked% for each row.
THEN I added a metadata field (the price of the book) to the row format so that each item in the asset listing had a price next to it.
The time taken to render went from 3 seconds to 1min 30 seconds! 30 times slower!
I couldnt believe it so I tested it again. This time I made a new listing to list even more items (over 400). Took 3 seconds again with just the asset name being listed and went to over 6 minutes when adding a metadata field to the type format.
So I tried with a smaller listing (28 items) and it went from less than 1 second to 14 seconds.
I would have expected a small increase in wait time for metadata fields but these sorts of comparisons show that what you put in your asset listings is crucial.
Now all this would be just an interesting observation if we werent using metadata in EVERY ASSET LISTING WE HAVE! Now its a tragedy. How much extra processing must we be putting on our server of over 250000 assets just because we have metadata fields in all of our asset listings.
I need a lie down.
Steve.
Are you sorting by a metadata field?
There are some other optimisations you can do to improve the performance of your asset listings in other ways.
Try some of these if you can:
- Turn off sorting if you can. If you need to list assets in the order they appear in the asset map, turn on "Direct links only" (if possible) and turn off sorting.
[*]Use "Direct links only - on" if you can. This speeds the listing up a lot.
[*]List specific asset types, and don't use "Inherit". Enabling "Inherit" means an extra query which has to enumerate asset types from the database.
[*]Use an IA to make your listings as specific as possible (for example, don't make it traverse an entire site just to find a few assets).
[*]Selecting asset statuses in the "Asset Status Selector" is not actually required. If you select no statuses, it will just list all of the ones that the current user has read access to (ie. Live, and public read access). This also makes the query slightly smaller.
Adding a metadata field should increase the time slightly, but not much more than adding another attribute keyword. Metadata does load files from the file system while attributes get their values from the DB. Maybe you have a specific problem there? Obviously impossible for me to say, but that is the difference at least.
If you are a client, I suggest contacting Squiz Support so they can look into your system and see why metadata is giving you such bad performance If not, you could always open a bug report so we can look into possible optimisations if we can achieve the same load times.
Thanks for the feedback Dan and Greg,
Dan - The list has no sorting applied, no grouping, no asset status restrictions and direct links only = yes. We do inherit in our datatypes to list otherwise every time you guys add a new asset type we'd have to go and update all our master asset listings.
Greg - I assume that the metadata files are loaded only if required - ie it only opens the metadata files if there are metadata keywords in the search, group or row format screens. This must be the case for the discrepencies to be so large.
I will call support and get them to look into our system. If they cant work it out we will have a lot of work ahead of us retrofitting this new-found change into our asset listings.
I assume that we are going to see the same performance issues with our search pages as well.
Steve
An update for anyone interested…
Squiz found a little bit of redundant code that they very quickly removed and patched our backup system with [impressive turnaround time of a couple of days]. We did see an improvement in times but not of the magnitude we were hoping…
eg
http://www.dpi.nsw.gov.au/agriculture/field/field-crops/oilseeds
Our Oilseeds page (with a _nocache suffix added) nests an asset listing that has “custom grouping” by parent asset applied. it has 37 items in the listing.
Load time before the patch: 54 seconds
Load time after the patch: 49 seconds
Load time once the metadata was removed from the row format and asset name used instead: 24 seconds
Load time with no metadata and grouping also removed: 1 second.
So in this case:
Adding grouping (by parent asset) to a listing adds 23 seconds to the render time
Adding metadata to a row format adds a further 25 seconds
Our system contains over 250000 assets.
Steve
Any techies able to elaborate on the bit of code removed?
Metadata keywords were being replaced in two places in our listing engine. We removed one. If you want a patch, please contact Squiz Support or you can wait for the next release in March, which will include this change.
Another update for anyone interested…
Had our system patched with some of the new speed improvements that are available in the 3.20 build (we are on 3.18.6).
This has made a HUGE difference.
eg this page
http://www.dpi.nsw.gov.au/research/staff/research-staff-branch?queries_branch_query=Research+Operations
with a _nocache suffix now takes 2 seconds to load when it was yesterday taking 42 seconds. Nice one.
Our pages that contain grouping however are still quite slow (but have improved). The oilseeds page mentioned in a previous post has come from 54 seconds to 36 seconds (although if I removed the grouping Im sure it would only be 1 or 2 seconds).
I hope at some stage Squiz will look into getting the same sort of productivity gains in listings with grouping as they did in listings with metadata.
Steve