Pulling calendar data into JSON file

coolwebs · 2015-03-04 02:09:55 UTC

I've taken over a project and there is very little information or notes to go from. Basically the dev before me has setup som JSON pages in Squiz Matrix to pull in data from the calendar events. I really don't know how to set this up myself - even after reading the Squiz Matrix manual and trying to figure out how it has been setup by analysing the structure in Squiz Matrix.

BTW, the reason is that there is a mobile app which receives this data to display events in the application.

The JSON file combines single and recurring events (not sure how) into two JSON files that are somehow combined to give this URL http://www.slq.qld.gov.au/_slqmedia/whats_on/asset-listings/single-recurring-event-data.json

The code is below:

1) Recurring calendar event JSON code

{"id":"%asset_assetid%-%instance_start_datetime_Y%%instance_start_datetime_m%%instance_start_datetime_j%%instance_start_datetime_g%%instance_start_datetime_i%","title":"%event_name^json_encode%","summary":"%asset_metadata_Summary^json_encode%","description":"%asset_attribute_description^json_encode%","start_date_and_time":"%instance_start_datetime_D% %instance_start_datetime_j% %instance_start_datetime_M% %instance_start_datetime_Y%, %instance_start_datetime_g%:%instance_start_datetime_i% %instance_start_datetime_a%","end_date_and_time":"%instance_end_datetime_D% %instance_end_datetime_j% %instance_end_datetime_M% %instance_end_datetime_Y%, %instance_end_datetime_g%:%instance_end_datetime_i% %instance_end_datetime_a%","thumbnail":"%asset_thumbnail_url^json_encode%","location":"%asset_metadata_Venue^json_encode%%asset_metadata_VenueOther%, %asset_metadata_Address^json_encode%, %asset_metadata_Location^json_encode%","paid_event":"%asset_metadata_FreeOrPaid%","status":"%asset_metadata_Status^titleize%","event_type":"%asset_metadata_TypeMain^replace:":'^titleize%","read_more_url":"%asset_url^json_encode%"},

2) Single Calendar event JSON code

{"id":"%asset_assetid%","title":"%event_name^json_encode%","summary":"%asset_metadata_Summary^json_encode%","description":"%asset_attribute_description^json_encode%","start_date_and_time":"%event_start_datetime_D% %event_start_datetime_j% %event_start_datetime_M% %event_start_datetime_Y%, %event_start_datetime_g%:%event_start_datetime_i% %event_start_datetime_a%","end_date_and_time":"%event_end_datetime_D% %event_end_datetime_j% %event_end_datetime_M% %event_end_datetime_Y%, %event_end_datetime_g%:%event_end_datetime_i% %event_end_datetime_a%","thumbnail":"%asset_thumbnail_url^json_encode%","location":"%asset_metadata_Venue^json_encode%%asset_metadata_VenueOther%, %asset_metadata_Address^json_encode%, %asset_metadata_Location^json_encode%","paid_event":"%asset_metadata_FreeOrPaid%","status":"%asset_metadata_Status^titleize%","event_type":"%asset_metadata_TypeMain^replace:":'^titleize%","read_more_url":"%asset_url^json_encode%"},

To be honest, when looking at the feed of data that the URL retrieves http://www.slq.qld.gov.au/_slqmedia/whats_on/asset-listings/single-recurring-event-data.json, it really looks like it follows the code from the latter.

Anyway, the issue is that it has not retrieved all of the events. They are all live, I just checked that the data had been entered in correctly, and everything checked out. Is there any good reason that single calendar event data will not show up in the feed? Some have and many haven't and I can't see any obvious reason why the JSON data dump is being so selective.

I've also attached a screenshot of how the setup in Squiz Matrix looks too. JSON-data-setup.jpg (87.9 KB)

coolwebs · 2015-03-04 02:23:33 UTC

I'm almost quite sure that there must be some errors thrown somewhere and that just stops the process of importing certain data (attached to event assets) but I don't know of any debug console I could use. Is there anything in browser or Squiz Matrix that would allow me to debug and have an error console?

NickyP · 2015-03-04 05:52:31 UTC

I checked your http://www.slq.qld.g...event-data.json file using json visualiser showing 253 entries but 254 is empty. So the data is there if these numbers are correct.

What asset/s are you using for reading the json file?

coolwebs · 2015-03-04 07:25:54 UTC

I was using the JSONView firefox addon to check the code as well (I got sick of looking at unstructured code) and I can see what you saw. Line 254 is empty as it is an empty field. This happens because Squiz Matrix wants to append a comma after all the values (for it to work in an array) - i think this is interpreted as an empty request.

The JSON file is being read by "tracks" in a Wordpress site which a mobile application reads the data from it. Right now, I only have access to reading the JSON file through a web browser. I have access to the Squiz Matrix and I know that there are more events in the system than are showing up in the feed - and I have no errors to even get an idea where to go looking.

I checked out the data in the events themselves but it all checks out.

I did a bit of diagnostics by archiving some selected events (that are showing up within the feed). When I turned them off, some of the events that weren't showing previously - started to show up in the JSON feed (bit like a domino effect). I analysed the events that stood out and could not see much wrong with them except for a bit of sloppy html coding in the description due to staff cuttting and pasting into WYSIWYG editor. Anything that would normally break JSON is taken care of my encoding as you can see in the code above. Besides, that kind of thing WOULD show some sort of an error message I believe.

Also, I would really like to know what the order is that the content is read and shown by JSON into the page. So far it does not follow any particular order, just random. I get the feeling it queries the SQL database and follows the unitque ID order and it has absolutely nothing to do with the Asset ID order listing.

Got me scratching my head. I have a sneaky suspicion that the blame could lie with our hosting server.....

nnhubbard · 2015-03-04 16:32:23 UTC

What kind of asset are you using to list and write the JSON?

NickyP · 2015-03-04 23:14:52 UTC

I had a similar issue not that long ago until I used the stripthtml keyword that fixed the issue.

Long shot, but can you try "description":"%asset_attribute_description^replace_keywords:striphtml^json_encode%"

This will leave you with raw data (removes sloppy html) then all you have to do is re-style it with whatever you're using to write the JSON as Nic has asked

coolwebs · 2015-03-04 23:56:33 UTC

@Nic, Upcoming Events List - that's what I am using to list and write the JSON, using bodycopies within them to write the JSON code (that you see at the beginning of this topic). Please see screenshot. Thanks for the suggestion @NickyP - I will give it a go.

coolwebs · 2015-03-05 04:20:25 UTC

Sadly, the code you gave me @NickyP did not make any difference to the actual data feed coming out. Yes you were right, no need for truncation.

PeterM · 2015-03-06 03:35:58 UTC

Hi

How many events should it return?

If it is more than 255 then the reason could be that the 'Number of events to show' field in the Upcoming Events Listing seems to have a max of 255 (at least in v4.18 which I am using)...

So maybe could try using a listing instead?

Thanks

Peter

coolwebs · 2015-03-09 00:27:05 UTC

Ohh wow. Yeah that could be it.

Could you please show me how to get to the Number of events to show field?

We am using version 4.18.4 as well.

coolwebs · 2015-03-09 04:00:48 UTC

Yep, I am pretty sure that is the problem. I am not familiar with "a listing"

How would that work? Is there any reference materials I could research up on? I've been using Squiz matrix only about a month so my head still feels like cotton wool.

coolwebs · 2015-03-10 00:07:55 UTC

@PeterM, sorry I have answered your question properly. It should be showing a lot more than 255 events, I know there is about a list of 20 missing right now. Events could also vary. Be good if there was no limit (or at least a very high limit) on the "Number of Events to Show" field.

coolwebs · 2015-03-10 00:34:00 UTC

I tried to do it with asset listing page. It didn't work out. json-asset-listing-error.jpg (108 KB)

PeterM · 2015-03-10 01:13:07 UTC

Hi

I tend to use a seperate design asset and layout for outputing to data formats to ensure there is no addittional html added.

Eg: a json design could be:

<MySource_PRINT id_name="__global__" var="content_type" content_type="application/json" />

<MySource_AREA id_name="page_body" design_area="body" />

And also a layout if needed (to stop it inheriting one) where the formats and page contents a set to raw html content types.

You probably could just use whatever design/layout that were already applied to the existing upcoming events list though.

Also check that the page content and formats for the asset listing are also set to raw html to ensure they aren't enclosed in divs (as they seem to be in the image).

Using a listing you could also use the position format feature to avoid having the empty item at the end of the array - eg

the default format could be: ,{...}

first position format could be: {...}

page contents: [%asset_listing%]

which for 5 assests would result in something like: [{...},{...},{...},{...},{...}]

The downside of the asset listing for events is that you would need to handle the 'upcoming events' logic manually somehow...

Thanks

Peter

coolwebs · 2015-03-10 03:17:14 UTC

Sounds tricky. I have applied the json design to the asset (it was exactly the same as you suggested).

Also the blank paint layout as well.

I had to recreate the bodycopies again and made sure that they were linked to the customised asset type formats in the "Display Formats" screen. I would like to get rid of the "no results" bodycopy, but it is there for a good reason (debugging).

Think i have solved the issue with the extra divs as the error message is now very simple HTML:

<p>No results were found</p>

</div>

Is this because the page is still "under construction blue?" or is this where the manual handling comes in?

PeterM · 2015-03-10 03:30:39 UTC

Hi

The listing should show events that are Under Construction if the it has been set to do so on the Details screen - maybe all the details settings are ok there...

What I had referred to as the manual part was related to showing only 'upcoming' events.

The manual site also suggests using a calendar search as an alternative to an upcoming events listing (this might be a better option for you):

http://manuals.matrix.squizsuite.net/calendar/chapters/upcoming-events-list

Thanks

Peter