JSON Datasource truncating Datasource Record Set data

(Nic Hubbard) #1

Matrix Version: 5.1.1.0

I am using a JSON Datasource asset which is pulling in a large source file using a REST Resource asset. In the JSON Datasource I have configured rows as the root asset, which is an array. This then correctly shows in the Asset Map all of the rows.

However, when viewing the details of one of the Datasource Record Set assets it appears that Matrix is truncating the data of one of our fields. This field is called cells and itself contains a JSON array. I was accessing the array items by using cells[1].displayValue, cells[2].displayValue etc. However because Matrix seems to truncate this data, I cannot access all of the data I need to.

Is there a threshold for how many characters a field of the Datasource Record Set can hold? When viewing the “View JSON Data” screen of the JSON Datasource asset we do not have the truncation issue.

Would there be another way to handle a child array that is inside of a Datasource Record Set?

(Nic Hubbard) #2

Has anyone seen this before or know why this would be happening?

(Bart Banda) #3

Not that I know of, after how many characters does it truncate?
Can you maybe try using a keyword modifier on it to see if that fixes it? ^json_decode ?

(Nic Hubbard) #4

Not sure how that would apply in this case.

I have found what is causing the issue and that is some of our JSON fields have HTML in them. This is causing Matrix to stop at that place in the JSON which is making the JSON Datasource asset not work for us.

This seems like a bug to me, the HTML is JSON escaped.

(Nic Hubbard) #6

Another major issue that I have found with the JSON Datasource is that it doesn’t seem to parse an array that is over 10 objects. Here is what I see in the “Parsed Data” area of the JSON Datasource asset:

As you can see the JSON array has 20 objects but Matrix is only parsing the first 10 as seen in the screenshot.

Here is the Raw Source Data:

Screen Shot 2017-02-01 at 1.02.25 AM.png598x876 38.7 KB 

{
  "rows": [
    {
      "cells": [
        {
          "displayValue": "1"
        },
        {
          "displayValue": "2"
        },
        {
          "displayValue": "3"
        },
        {
          "displayValue": "4"
        },
        {
          "displayValue": "5"
        },
        {
          "displayValue": "6"
        },
        {
          "displayValue": "7"
        },
        {
          "displayValue": "8"
        },
        {
          "displayValue": "9"
        },
        {
          "displayValue": "10"
        },
        {
          "displayValue": "11"
        },
        {
          "displayValue": "12"
        },
        {
          "displayValue": "13"
        },
        {
          "displayValue": "14"
        },
        {
          "displayValue": "15"
        },
        {
          "displayValue": "16"
        },
        {
          "displayValue": "17"
        },
        {
          "displayValue": "18"
        },
        {
          "displayValue": "19"
        },
        {
          "displayValue": "20"
        }
        
        
      ]
    }
    
    
    
    
  ]
}

From testing it looks like the problem is in data_source_json.inc around line 601.

Is this another bug?

(Bart Banda) #7

Can you provide a link to the JSON data you are trying to parse so that we can try and replicate? Or upload an attachment with the JSON data?

Not sure, would be worth submitting a squizmap bug to get it reviewed further.

Can you use the REST JS asset as a workaround and out put the list that way?

(Nic Hubbard) #8

Can you provide a link to the JSON data you are trying to parse so that we can try and replicate? Or upload an attachment with the JSON data?

I can’t as it is on a private system with non-public data.

Not sure, would be worth submitting a squizmap bug to get it reviewed further.

I provided the JSON above that I tested that it reproduces the issue. Can you test this as well?

Can you use the REST JS asset as a workaround and out put the list that way?

Probably not easily. We are need to list the shadow assets and then using an asset listing be able to display the full record using the asset position. It would take hours to rewrite everything. I have already wasted precious time trying to get past these bugs.

(Bart Banda) #9

Thanks for reporting the bugs Nic, we’ll get someone to review them and will post an update here once we know more.

(Nic Hubbard) #10

I can provide the full JSON production if needed. Please let me know.

(Bart Banda) #11

Bug has been fixed and will be out in the next release https://squizmap.squiz.net/matrix/10529

(Emily) #12

Hi, sorry to bring up an old topic. I am looking into this JSON data again (to extend to a new project), and I note the bug fix is listed in 5.4.1.3 (as well as 5.3.6.1). I am using v5.4.3.2, so it should have the fix included when it was upgraded. The setup is the one that Nic originally created for testing.

I am viewing the Asset Listing which lists the JSON DS, which are created based on a REST Resource.

The JSON DS asset “parsed data” shows all the fields (>10) - so the bug is fixed :+1: . But what I can’t seem to get working is to get those fields >10 to actually display their content in the Asset Listing.

e.g. %ds__cells_17.displayValue% does not display its content in the Asset Listing. I can view that it has content in the “Parsed Data” however.

I am hoping I am overlooking something simple here…?
0-9 displays in the Asset Listing…

Thanks for any thoughts / help / time. :slight_smile:
Emily.

(Emily) #13

Secondly - I forgot to add…

I can see tickets from 2010 asking about Grouping by datasource keyword in an Asset Listing, and I see that it was not supported at that time. I note that Sorting by datasource keyword does work now.

Has this changed at all in the past 7 years?
If not, is there any plans for this?

In my test I have not been able to successfully group by the %ds__cells_1.displayValue% keyword. Resulting output is same as per the 2010-2012 posts.

I note there was a suggested workaround of using the shadow asset “name” as the group name. This may work, but it is not ideal. I will explore that option however if there is still no support for Grouping.

Thanks :slight_smile:

(Bart Banda) #14

Hey Emily,
Any way you could share the link you are using for the data source records so that we can see if we can replicate?

Otherwise, it’ll be hard to help without having a look at the system and implementation directly, so you might need to go through Squiz support to get some further help.

Probably, but not sure. You could group using SSJS as well, that would also make the listing generate quicker.