Troubleshooting Remote Content problems

(Douglas (@finnatic at @waikato)) #1

5.4.1.2

How do people troubleshoot remote content problems? I think we have an https comms issue, but nothing seems to be being logged in the error or system log, and adding SQ_SHOW_ERRORS as a query string doesn’t result in any difference.

(Douglas (@finnatic at @waikato)) #2

And just to add more details - what Chrome is showing me is:

This page isn’t working
$server didn’t send any data.
ERR_EMPTY_RESPONSE

And the url I then try is:
http://$server/$site/$page?SQ_SHOW_ERRORS

but with no success.

The remote content is an https address, and is working on our dev system which is also running under https - but our production system at this time is running under http and making the change is not something I want to rush.

(Mahearnpad) #3

The first thing I would do is try the external resource using Postman - it’s a Chrome addin.

(Douglas (@finnatic at @waikato)) #4

Thanks mahearn

The steps that I ended up working through were:

  1. Check the external resource separate of the Matrix CMS (browser, Postman perhaps?)
  2. Check access to the external resource from the Matrix CMS server via the shell using curl/wget/lynx/etc.
  3. See if Matrix could provide any errors using SQ_SHOW_ERRORS or similar and review logs via Log Manager.
  4. Monitor / check Matrix logs on the Matrix CMS server at the shell level.
  5. Monitor / check Apache/NGINX/Webserver logs on the Matrix CMS server.

In the end, what I found was that Apache was logging an error to it’s error log - “symbol lookup error: /usr/lib64/libnsssysinit.so: undefined symbol: PR_GetEnvSecure” which led to several stack overflow discussions.

Most of those suggested an IT Crowd solution of turning Apache off and on again, which resulted in the problem clearing but with root cause not clearly identifiable. Neither are really an answer I like, but the fact that the fault was with something at the Apache level perhaps explains why Matrix wasn’t logging anything.

(Marcus Fong) #5

This sounds like the kind of error you get when you run a package update, which updates your service (in this case Apache) and its supporting libraries on disk. If you don’t restart your service, you then have a version mismatch between the running service (old) and its libraries (new). This produces errors when the (old) service loads a (new) library from disk, and the library tries to reference newly added variable or function names that don’t yet exist in the service.

Then when you next restart the service, it comes back up as the new version. So everything matches properly, all the variable and function names are present and it all works fine.