Strange "failed to load external entity" errors

nnhubbard · 2013-03-05 19:05:25 UTC

We just started getting tons of errors across our website. They appear when in the _admin and trying to view different screens. Strangely they appear sometimes, but then trying to go back to that screen they don't show and everything seems normal:

PHP WarningFile:[SYSTEM_ROOT]/core/include/asset_edit_interface.incLine:106Message:SimpleXMLElement::__construct(): I/O warning : failed to load external entity "[SYSTEM_ROOT]/core/assets/site/edit_interface_screens.xml"Backtrace:Show
Warning: count(): Node no longer exists in /opt/matrix/fudge/dev/dev.inc on line 51 Fatal error: Uncaught exception 'Exception' with message: 'Unable to parse screen list file "/opt/matrix/core/assets/site/edit_interface_screens.xml": String could not be parsed as XML' in /opt/matrix/core/include/asset_edit_interface.inc(108): Asset_Edit_Interface('site')
#1 /opt/matrix/core/include/asset.inc(5073): getEI()
#2 /opt/matrix/core/include/asset_manager_edit_fns.inc(224): paintBackend(Object(Backend_Outputter))
#3 /opt/matrix/core/include/asset_manager.inc(8961): paintBackend()
#4 /opt/matrix/core/include/backend.inc(1526): paintBackend(Object(Backend))
#5 /opt/matrix/core/include/backend.inc(194): _printMain()
#6 /opt/matrix/core/include/mysource.inc(571): paint()
#7 /opt/matrix/core/web/index.php(30): start()

I thought maybe it was a locals thing, so I tried to run compile_locals.php and I get more errors:

PHP Warning:  fopen(/opt/matrix/data/private/logs/error.log): failed to open stream: Permission denied in /opt/matrix/core/include/general.inc on line 1300
PHP Warning:  fwrite() expects parameter 1 to be resource, boolean given in /opt/matrix/core/include/general.inc on line 1301
PHP Warning:  Unable to Write to log file /opt/matrix/data/private/logs/error.log [SYS0051] in /opt/matrix/core/include/locale_manager.inc on line 547
+-----------------------------------------------------------------------------------------------+
| PHP Warning                                                                                   |
|-----------------------------------------------------------------------------------------------|
| File : [SYSTEM_ROOT]/fudge/general/file_system.inc                                            |
| Line : 137                                                                                    |
|-----------------------------------------------------------------------------------------------|
| rename(/tmp/string_to_filemwAj9t,[SYSTEM_ROOT]/data/private/conf/main.inc): Permission denied |
+-----------------------------------------------------------------------------------------------+

Checking the syslog show:

Mar  5 08:45:03 neo suhosin[22755]: ALERT - script tried to disable memory_limit by setting it to a negative value -1 bytes which is not allowed (attacker 'REMOTE_ADDR not set', file 'unknown')
Mar  5 08:45:12 neo puc.edu: SimpleXMLElement::__construct(): I/O warning : failed to load external entity "[SYSTEM_ROOT]/core/include/asset_edit/edit_interface_screens.xml" ([SYSTEM_ROOT]/core/include/asset_edit_interface.inc:106)

Anyone know what could have started causing this? Nothing has changed on our server. Really not sure what is going on here...

nnhubbard · 2013-03-06 22:32:44 UTC

Anyone? It mainly now happens when trying to create a new asset.

Also, it seems that using HTTPS fixes the issue.

mfong · 2013-03-07 08:35:32 UTC

Anyone? It mainly now happens when trying to create a new asset.

I’m not really sure, Nic; the errors look as if Matrix is having trouble accessing files on your server’s disk(s). But I don’t see how that would correlate with your other observation:

Also, it seems that using HTTPS fixes the issue.

I wouldn’t normally expect “permission denied” errors on filesystem access to be related to whether Apache is using SSL or not. Is there anything in the Apache logs? Does “dmesg” show any errors?

nnhubbard · 2013-03-07 16:50:00 UTC

I'm not really sure, Nic; the errors look as if Matrix is having trouble accessing files on your server's disk(s). But I don't see how that would correlate with your other observation:

I wouldn't normally expect "permission denied" errors on filesystem access to be related to whether Apache is using SSL or not. Is there anything in the Apache logs? Does "dmesg" show any errors?

Yeah, after more testing it seemed that there was no correlation to HTTPS. The problem just seems to come and go which led me to believe that could have fixed it, when it fact it didn't.

We had done some upgrades to apache yesterday, which is the only thing that has changed:

apache2 2.2.16-6+squeeze11
apache2.2-bin 2.2.16-6+squeeze11
apache2.2-common 2.2.16-6+squeeze11
apache2-mpm-prefork 2.2.16-6+squeeze11
apache2-utils 2.2.16-6+squeeze11

After doing those I noticed the issue, and we reverted back. This is why I wondered if there was a Suhosin update that was causing this error, which seemed to be in the logs at the same time as the other error:

ALERT - script tried to disable memory_limit by setting it to a negative value -1 bytes which is not allowed (attacker 'REMOTE_ADDR not set', file 'unknown')

Benjamin_Pearson · 2013-03-07 18:44:37 UTC

Not sure either Nic, from your original post it looks like 3 different errors to me: xml node gets removed while using it, wrong permission and suhosin being a bully.

I guess if Apache was upgraded, check if the update changed your vhost settings or php settings.

nnhubbard · 2013-03-08 00:20:33 UTC

Not sure either Nic, from your original post it looks like 3 different errors to me: xml node gets removed while using it, wrong permission and suhosin being a bully.

I guess if Apache was upgraded, check if the update changed your vhost settings or php settings.

It is an extremely strange problem that our server admin and I cannot figure out. It seems to just appear at random times then go away.

We are resorting to building a VM and moving everything over. This isn't all bad as it will allow us to upgrade to Debian Weezy and the newest version of Matrix.

jhewitt86 · 2013-03-10 06:56:10 UTC

Hi Nic,

Did you find anything more out about this issue? I am now experiencing the error on most screens in the admin interface in Matrix.

nnhubbard · 2013-03-10 19:16:34 UTC

Hi Nic,

Did you find anything more out about this issue? I am now experiencing the error on most screens in the admin interface in Matrix.

rc52s4.png1597x119

Yep, that is the same exact error.

We have not yet figured it out and it is still occurring. I still suspect it has something to do with the upgrades to did the other day:

apache2 2.2.16-6+squeeze11
apache2.2-bin 2.2.16-6+squeeze11
apache2.2-common 2.2.16-6+squeeze11
apache2-mpm-prefork 2.2.16-6+squeeze11
apache2-utils 2.2.16-6+squeeze11

Did you happen to do these same updates?

jhewitt86 · 2013-03-10 22:41:26 UTC

Hey mate, this was a really strange problem. The error would render 30% of the time, and then eventually 100% of the time, and then not at all for a few minutes.

I had only made one server modification; I updated PEAR. My aim was to get JSON http responses working but I never made it that far (I'm on PHP 5.3.3). I rolled back to the backup I created prior to the upgrade and presto, she's all good again. No apache updates.

nnhubbard · 2013-03-10 23:24:23 UTC

Hey mate, this was a really strange problem. The error would render 30% of the time, and then eventually 100% of the time, and then not at all for a few minutes.

I had only made one server modification; I updated PEAR. My aim was to get JSON http responses working but I never made it that far (I'm on PHP 5.3.3). I rolled back to the backup I created prior to the upgrade and presto, she's all good again. No apache updates.

Wow, exactly like we are experiencing.

So strange, wish someone at Squiz could reproduce this.

gsherwood · 2013-03-10 23:58:16 UTC

Any idea what PEAR packages were updated?

nnhubbard · 2013-03-12 23:05:44 UTC

We just finished up moving our entire website to a new VM, and this problem has showed up again. Really not sure what is going on...

It feels like a bug.

Would a backtrace help?

Edison_Wang · 2013-03-13 00:09:38 UTC

First 2 errors are file permission errors. The last one is about Sohusin being too fussy.

When Matrix loads the screen of site, it tries to load the file "core/assets/site/edit_interface_screens.xml", but got denied for read access.

When it tries to write to log file, it got denied for write access.

What's the file permissions applied to those files in your VM? Can you make sure apache user has read access to all Matrix files, and has read+write access to data and cache folders?

Is there anyway you can install a fresh Matrix on that problematic VM, and give me SSH access to have a look? Or let us download the VM image (i'm using vmware fusion)?

nnhubbard · 2013-03-13 00:24:17 UTC

Is there anyway you can install a fresh Matrix on that problematic VM, and give me SSH access to have a look? Or let us download the VM image (i'm using vmware fusion)?

I will PM you.

Edison_Wang · 2013-03-14 23:14:03 UTC

I had a look at Nic's server, but couldn't replicate the problem then(it happens intermittently).

This issue doesn't look like a code problem. Most likely something wrong with one of the server package update.

If anyone can test and find out which package and version they upgraded caused the problem, please let us know.

I'm also requesting our sysadmin Chris Smith to do some package upgrade test and see if we can replicate the problem and isolate the bad package.

nnhubbard · 2013-03-27 20:41:05 UTC

I had a look at Nic's server, but couldn't replicate the problem then(it happens intermittently).

This issue doesn't look like a code problem. Most likely something wrong with one of the server package update.

If anyone can test and find out which package and version they upgraded caused the problem, please let us know.

I'm also requesting our sysadmin Chris Smith to do some package upgrade test and see if we can replicate the problem and isolate the bad package.

Any luck on finding out what could be causing this issue?

Chris_Smith · 2013-04-15 04:35:29 UTC

Hi Nic,

I haven't been able to replicate it in a simple environment/vm using debian wheezy. Can you send me a list of:

- php modules

- pear packages

- pecl packages

you have installed?

nnhubbard · 2013-04-15 15:54:04 UTC

Hi Nic,

I haven't been able to replicate it in a simple environment/vm using debian wheezy. Can you send me a list of:

- php modules

- pear packages

- pecl packages

you have installed?

Hi Chris. This error started happening after we did some Debian suggested package updates. Here is what we currently have:

PEAR

Package                Version State
Archive_Tar            1.3.7   stable
Console_Getopt         1.2.3   stable
HTTP_Client            1.2.1   stable
HTTP_Request           1.4.4   stable
I18N_UnicodeNormalizer 1.0.0   stable
Image_Canvas           0.3.4   alpha
Image_Color            1.0.4   stable
Image_Graph            0.8.0   alpha
MDB2                   2.4.1   stable
MDB2_Driver_pgsql      1.4.1   stable
Mail                   1.2.0   stable
Mail_Mime              1.8.7   stable
Mail_Queue             1.2.7   stable
Mail_mimeDecode        1.5.5   stable
Math_BigInteger        1.0.0   stable
Net_Socket             1.0.10  stable
Net_URL                1.0.15  stable
Numbers_Roman          1.0.2   stable
Numbers_Words          0.16.4  beta
PEAR                   1.9.1   stable
Structures_Graph       1.0.3   stable
Text_Diff              1.1.1   stable
XML_HTMLSax            2.1.2   stable
XML_Parser             1.3.4   stable
XML_Util               1.2.1   stable

PHP Modules

I used dpkg -l | grep php to get these, let me know if you wanted something else.

libapache2-mod-php5                5.3.3-7+squeeze15
php-pear                           5.3.3-7+squeeze15
php5-cli                           5.3.3-7+squeeze15
php5-common                        5.3.3-7+squeeze15
php5-curl                          5.3.3-7+squeeze15
php5-gd                            5.3.3-7+squeeze15
php5-ldap                          5.3.3-7+squeeze15
php5-pgsql                         5.3.3-7+squeeze15
php5-suhosin                       0.9.32.1-1

PECL

How do I get a list of these?

Chris_Smith · 2013-04-15 21:02:59 UTC

Hi Nic,

Hi Chris. This error started happening after we did some Debian suggested package updates. Here is what we currently have:

PEAR

Package                Version State
Archive_Tar            1.3.7   stable
Console_Getopt         1.2.3   stable
HTTP_Client            1.2.1   stable
HTTP_Request           1.4.4   stable
I18N_UnicodeNormalizer 1.0.0   stable
Image_Canvas           0.3.4   alpha
Image_Color            1.0.4   stable
Image_Graph            0.8.0   alpha
MDB2                   2.4.1   stable
MDB2_Driver_pgsql      1.4.1   stable
Mail                   1.2.0   stable
Mail_Mime              1.8.7   stable
Mail_Queue             1.2.7   stable
Mail_mimeDecode        1.5.5   stable
Math_BigInteger        1.0.0   stable
Net_Socket             1.0.10  stable
Net_URL                1.0.15  stable
Numbers_Roman          1.0.2   stable
Numbers_Words          0.16.4  beta
PEAR                   1.9.1   stable
Structures_Graph       1.0.3   stable
Text_Diff              1.1.1   stable
XML_HTMLSax            2.1.2   stable
XML_Parser             1.3.4   stable
XML_Util               1.2.1   stable

PHP Modules

I used dpkg -l | grep php to get these, let me know if you wanted something else.

libapache2-mod-php5                5.3.3-7+squeeze15
php-pear                           5.3.3-7+squeeze15
php5-cli                           5.3.3-7+squeeze15
php5-common                        5.3.3-7+squeeze15
php5-curl                          5.3.3-7+squeeze15
php5-gd                            5.3.3-7+squeeze15
php5-ldap                          5.3.3-7+squeeze15
php5-pgsql                         5.3.3-7+squeeze15
php5-suhosin                       0.9.32.1-1

PECL

How do I get a list of these?

We've got the same apache:

$ dpkg -l | grep apache2
ii apache2                             2.2.16-6+squeeze11           Apache HTTP Server metapackage
ii apache2-mpm-prefork                 2.2.16-6+squeeze11           Apache HTTP Server - traditional non-threaded model
ii apache2-utils                       2.2.16-6+squeeze11           utility programs for webservers
ii apache2.2-bin                       2.2.16-6+squeeze11           Apache HTTP Server common binary files
ii apache2.2-common                    2.2.16-6+squeeze11           Apache HTTP Server common files
ii libapache2-mod-php5                 5.3.3-7+squeeze15            server-side, HTML-embedded scripting language (Apache 2 module)

I've got some extra php modules:

$ dpkg -l | grep php
ii libapache2-mod-php5                 5.3.3-7+squeeze15            server-side, HTML-embedded scripting language (Apache 2 module)
ii php-pear                            5.3.3-7+squeeze15            PEAR - PHP Extension and Application Repository
ii php5-cli                            5.3.3-7+squeeze15            command-line interpreter for the php5 scripting language
ii php5-common                         5.3.3-7+squeeze15            Common files for packages built from the php5 source
ii php5-curl                           5.3.3-7+squeeze15            CURL module for php5
ii php5-gd                             5.3.3-7+squeeze15            GD module for php5
ii php5-ldap                           5.3.3-7+squeeze15            LDAP module for php5
ii php5-pgsql                          5.3.3-7+squeeze15            PostgreSQL module for php5
ii php5-pspell                         5.3.3-7+squeeze15            pspell module for php5
ii php5-suhosin                        0.9.32.1-1                   advanced protection module for php5
ii php5-xmlrpc                         5.3.3-7+squeeze15            XML-RPC module for php5
ii php5-xsl                            5.3.3-7+squeeze15            XSL module for php5

I'll remove the extras and try again.

I've got *slightly* different versions in pear (and you've got a couple of extra packages), but the main ones involved (I would think) would be the xml related ones, they are the same.

$ pear list
Installed packages, channel pear.php.net:
=========================================
Package                Version State
Archive_Tar            1.3.7   stable
Console_Getopt         1.2.3   stable
HTTP_Client            1.2.1   stable
HTTP_Request           1.4.4   stable
I18N_UnicodeNormalizer 1.0.0   stable
Image_Canvas           0.3.5   alpha
Image_Color            1.0.4   stable
Image_Graph            0.8.0   alpha
Mail                   1.2.0   stable
Mail_Mime              1.8.7   stable
Mail_Queue             1.2.7   stable
Mail_mimeDecode        1.5.5   stable
Net_Socket             1.0.10 stable
Net_URL                1.0.15 stable
PEAR                   1.9.1   stable
Structures_Graph       1.0.3   stable
Text_Diff              1.1.1   stable
XML_HTMLSax            2.1.2   stable
XML_Parser             1.3.4   stable
XML_Util               1.2.1   stable

I don't have any pecl packages. You can get that list via 'pecl list'.

I was mainly looking for an accelerator (apc, eaccelerator, something like that), but I can't see anything.

Is it a particular type of asset this problem happens with (maybe it's always a site, or trigger, or ....) ?

nnhubbard · 2013-04-15 21:05:57 UTC

Is it a particular type of asset this problem happens with (maybe it's always a site, or trigger, or ....) ?

It happens everywhere since the error comes when it is trying to load one of the screens xml files. I see it a lot on file assets, but that is probably because I edit them often. But I have seen it on Page Assets too.