Large number of DB connections for Matrix user

[nnhubbard]

We sometimes, mostly in the middle of the night, have a huge number of connections happen from the Matrix user. It doesn't seem normal to us, but please correct me if this is in fact normal.

    datid |     datname     | procpid | usesysid |  usename   |          current_query          | waiting |          xact_start           |          query_start          |         backend_start         | client_addr | client_port 
    -------+-----------------+---------+----------+------------+---------------------------------+---------+-------------------------------+-------------------------------+-------------------------------+-------------+-------------
    16386 | mysource_matrix |   23463 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:23:53.925655-08 |             |          -1
    16386 | mysource_matrix |   23475 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:23:55.83262-08  |             |          -1
    16386 | mysource_matrix |   23445 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:23:52.411212-08 |             |          -1
    16386 | mysource_matrix |   23446 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:23:52.415525-08 |             |          -1
    16386 | mysource_matrix |   23447 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:23:52.439549-08 |             |          -1
    16386 | mysource_matrix |   23448 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:23:52.544027-08 |             |          -1
    16386 | mysource_matrix |   23449 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:23:52.551795-08 |             |          -1
    16386 | mysource_matrix |   23451 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:23:52.649528-08 |             |          -1
    16386 | mysource_matrix |   23452 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:23:52.65517-08  |             |          -1
    16386 | mysource_matrix |   23453 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:23:52.659388-08 |             |          -1
    16386 | mysource_matrix |   23450 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:23:52.639695-08 |             |          -1
    16386 | mysource_matrix |   23484 |    16385 | matrix_web |                           | f       |                               | 2012-02-28 13:24:03.289285-08 | 2012-02-28 13:23:55.995271-08 |             |          -1
    16386 | mysource_matrix |   23456 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:23:53.123698-08 |             |          -1
    16386 | mysource_matrix |   23457 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:23:53.19532-08  |             |          -1
    16386 | mysource_matrix |   23459 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:23:53.228542-08 |             |          -1
    16386 | mysource_matrix |   23458 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:23:53.227403-08 |             |          -1
    16386 | mysource_matrix |   23460 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:23:53.395291-08 |             |          -1
    16386 | mysource_matrix |   23465 |    16385 | matrix_web |                           | f       |                               | 2012-02-28 13:24:03.327128-08 | 2012-02-28 13:23:54.065852-08 |             |          -1
    16386 | mysource_matrix |   23466 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:23:54.090641-08 |             |          -1
    16386 | mysource_matrix |   23471 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:23:55.059551-08 |             |          -1
    16386 | mysource_matrix |   23472 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:23:55.06565-08  |             |          -1
    16386 | mysource_matrix |   23473 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:23:55.067592-08 |             |          -1
    16386 | mysource_matrix |   23474 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:23:55.071091-08 |             |          -1
    16386 | mysource_matrix |   23485 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:23:56.305773-08 |             |          -1
    16386 | mysource_matrix |   23488 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:23:56.315393-08 |             |          -1
    16386 | mysource_matrix |   23486 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:23:56.307492-08 |             |          -1
    16386 | mysource_matrix |   23487 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:23:56.309355-08 |             |          -1
    16386 | mysource_matrix |   23489 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:23:56.423312-08 |             |          -1
    16386 | mysource_matrix |   23634 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:24:02.967556-08 |             |          -1
    16386 | mysource_matrix |   23491 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:23:56.447198-08 |             |          -1
    16386 | mysource_matrix |   23492 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:23:56.451396-08 |             |          -1
    16386 | mysource_matrix |   23509 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:23:57.875253-08 |             |          -1
    16386 | mysource_matrix |   23510 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:23:57.88325-08  |             |          -1
    16386 | mysource_matrix |   23511 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:23:57.91941-08  |             |          -1
    16386 | mysource_matrix |   23512 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:23:57.955219-08 |             |          -1
    16386 | mysource_matrix |   23513 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:23:57.963256-08 |             |          -1
    16386 | mysource_matrix |   23514 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:23:57.980098-08 |             |          -1
    16386 | mysource_matrix |   23519 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:23:58.00341-08  |             |          -1
    16386 | mysource_matrix |   23515 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:23:57.984189-08 |             |          -1
    16386 | mysource_matrix |   23520 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:23:58.011356-08 |             |          -1
    16386 | mysource_matrix |   23521 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:23:58.023524-08 |             |          -1
    16386 | mysource_matrix |   23518 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:23:58.027317-08 |             |          -1
    16386 | mysource_matrix |   23517 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:23:58.031117-08 |             |          -1
    16386 | mysource_matrix |   23516 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:23:57.985413-08 |             |          -1
    16386 | mysource_matrix |   23522 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:23:58.12726-08  |             |          -1
    16386 | mysource_matrix |   23523 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:23:58.140741-08 |             |          -1
    16386 | mysource_matrix |   23524 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:23:58.151376-08 |             |          -1
    16386 | mysource_matrix |   23564 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:24:00.061999-08 |             |          -1
    16386 | mysource_matrix |   23566 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:24:00.12709-08  |             |          -1
    16386 | mysource_matrix |   23567 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:24:00.135268-08 |             |          -1
    16386 | mysource_matrix |   23565 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:24:00.131774-08 |             |          -1
    16386 | mysource_matrix |   23569 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:24:00.22783-08  |             |          -1
    16386 | mysource_matrix |   23572 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:24:00.271194-08 |             |          -1
    16386 | mysource_matrix |   23575 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:24:00.411232-08 |             |          -1
    16386 | mysource_matrix |   23579 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:24:00.468075-08 |             |          -1
    16386 | mysource_matrix |   23580 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:24:00.481807-08 |             |          -1
    16386 | mysource_matrix |   23583 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:24:00.527674-08 |             |          -1
    16386 | mysource_matrix |   23584 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:24:00.544716-08 |             |          -1
    16386 | mysource_matrix |   23585 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:24:00.551228-08 |             |          -1
    16386 | mysource_matrix |   23581 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:24:00.499374-08 |             |          -1
    16386 | mysource_matrix |   23587 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:24:00.572712-08 |             |          -1
    16386 | mysource_matrix |   23586 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:24:00.576139-08 |             |          -1
    16386 | mysource_matrix |   23591 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:24:00.767235-08 |             |          -1
    16386 | mysource_matrix |   23592 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:24:00.746804-08 |             |          -1
    16386 | mysource_matrix |   23595 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:24:00.799447-08 |             |          -1
    16386 | mysource_matrix |   23599 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:24:00.86683-08  |             |          -1
    16386 | mysource_matrix |   23596 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:24:00.867351-08 |             |          -1
    16386 | mysource_matrix |   23601 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:24:00.962198-08 |             |          -1
    16386 | mysource_matrix |   23606 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:24:00.955968-08 |             |          -1
    16386 | mysource_matrix |   23608 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:24:01.019577-08 |             |          -1
    16386 | mysource_matrix |   23600 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:24:00.938901-08 |             |          -1
    16386 | mysource_matrix |   23609 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:24:01.044241-08 |             |          -1
    16386 | mysource_matrix |   23607 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:24:00.987479-08 |             |          -1
    16386 | mysource_matrix |   23610 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:24:01.079252-08 |             |          -1
    16386 | mysource_matrix |   23615 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:24:01.267877-08 |             |          -1
    16386 | mysource_matrix |   23619 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:24:01.295726-08 |             |          -1
    16386 | mysource_matrix |   23613 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:24:01.227715-08 |             |          -1
    16386 | mysource_matrix |   23620 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:24:01.439997-08 |             |          -1
    16386 | mysource_matrix |   23621 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:24:01.527451-08 |             |          -1
    16386 | mysource_matrix |   23627 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:24:02.275261-08 |             |          -1
    16386 | mysource_matrix |   23628 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:24:02.330171-08 |             |          -1
    16386 | mysource_matrix |   23629 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:24:02.443309-08 |             |          -1
    16386 | mysource_matrix |   23630 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:24:02.479631-08 |             |          -1
    16386 | mysource_matrix |   23631 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:24:02.660226-08 |             |          -1
    16386 | mysource_matrix |   23632 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:24:02.83166-08  |             |          -1
    16386 | mysource_matrix |   23633 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:24:02.835-08    |             |          -1
    16386 | mysource_matrix |   23635 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:24:02.999551-08 |             |          -1
    16386 | mysource_matrix |   23636 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:24:02.999758-08 |             |          -1
    16386 | mysource_matrix |   23637 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:24:03.031283-08 |             |          -1
    16386 | mysource_matrix |   23638 |    16385 | matrix_web |                           | f       |                               |                               | 2012-02-28 13:24:03.068458-08 |             |          -1
    11564 | postgres        |   23639 |       10 | postgres   | SELECT * FROM pg_stat_activity; | f       | 2012-02-28 13:24:03.314627-08 | 2012-02-28 13:24:03.314627-08 | 2012-02-28 13:24:03.275899-08 |             |          -1

Has anyone seen this sort of thing before? What was the cause?

[Chris_Smith]

[quote]
We sometimes, mostly in the middle of the night, have a huge number of connections happen from the Matrix user. It doesn't seem normal to us, but please correct me if this is in fact normal.



Has anyone seen this sort of thing before? What was the cause?

[/quote]



No it's not normal. It's interesting they are all idle and aren't waiting. Check out pg_locks as well, see if they are all backed up behind something with a lock.



What about the other side of things - lots of php processes (lots of matrix cron jobs, or scripts scheduled for the same time) ?

[ryanhiebert]

OK, I've added the following to my script that checks every minute for >90 connections. It only runs these when it sees too many connections. Can you recommend anything else to run at the same time? The script runs as root.

    echo "\n------PROCESSES-------"
    ps
    echo "\n------LOCKS-------"
    sudo -u postgres psql -c 'SELECT * FROM pg_locks'

[Chris_Smith]

[quote]
OK, I've added the following to my script that checks every minute for >90 connections. It only runs these when it sees too many connections. Can you recommend anything else to run at the same time? The script runs as root.

    echo "\n------PROCESSES-------"
    ps
    echo "\n------LOCKS-------"
    sudo -u postgres psql -c 'SELECT * FROM pg_locks'

[/quote]



Not off the top of my head. The activity table had a lot of different process id's - we just need to work out what they are really and this should tell us everything we need to know.

[Dan_Simmons]

The most common cause of idle transactions is usually session locking on the webserver.


Nic, Do you use memcache for sessions?



If not, next time this happens log onto the web server, run the following command and paste the output here:

    
    sudo lsof -n |grep sess_

See if you get something that looks similar to this:

    
    httpd   16242    httpd    5u      REG                8,1      738    7143536 /var/www/matrix/cache/sess_ce7f2cbdd89b4a0fb6f156bc6bf91189
    httpd   16243    httpd    5u      REG                8,1      738    7143536 /var/www/matrix/cache/sess_ce7f2cbdd89b4a0fb6f156bc6bf91189
    httpd   16244    httpd    5u      REG                8,1      738    7143536 /var/www/matrix/cache/sess_ce7f2cbdd89b4a0fb6f156bc6bf91189
    httpd   16245    httpd    5u      REG                8,1      738    7143536 /var/www/matrix/cache/sess_ce7f2cbdd89b4a0fb6f156bc6bf91189
    httpd   16247    httpd    5u      REG                8,1      738    7143536 /var/www/matrix/cache/sess_ce7f2cbdd89b4a0fb6f156bc6bf91189
    httpd   16248    httpd    5u      REG                8,1      738    7143536 /var/www/matrix/cache/sess_ce7f2cbdd89b4a0fb6f156bc6bf91189
    httpd   16249    httpd    5uW     REG                8,1      738    7143536 /var/www/matrix/cache/sess_ce7f2cbdd89b4a0fb6f156bc6bf91189
    httpd   16250    httpd    5u      REG                8,1      738    7143536 /var/www/matrix/cache/sess_ce7f2cbdd89b4a0fb6f156bc6bf91189

Note above where the file that the httpd processes are waiting to open is all the same file, and one of them is blocking the others from reading it.

If you do a `ps auxf` you'll also see many httpd processes open probably doing nothing.

Further evidence is to strace one of the processes and see what it's doing:

    
    # strace -p 16244
    Process 16244 attached - interrupt to quit
    flock(5, LOCK_EX

If it it's in a flock() call then that means it's blocking waiting for access to that file descriptor.

Basically, if the transactions on the DB server are idle it means the process on the web server is doing something (or waiting on something) with a database connection open. If you find that it is in fact session locking, then the easiest solution is to enable memcache for sessions.

[nnhubbard]

[quote]
The most common cause of idle transactions is usually session locking on the webserver.

[/quote]



Thanks Dan. Ryan Hiebert is our server admin, so I will make sure he reads this and responds.

[ryanhiebert]

Thanks Chris. With those added to my script. I got this output:

    datid |     datname     | procpid | usesysid |  usename   |                                                                                                       current_query                                                                                                                                                     | waiting |          xact_start           |          query_start          |         backend_start         | client_addr | client_port 
    -------+-----------------+---------+----------+------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+-------------------------------+-------------------------------+-------------------------------+-------------+-------------
    16386 | mysource_matrix |   27578 |    16385 | matrix_web | SELECT                                                                                                | f       | 2012-02-29 05:22:14.654484-08 | 2012-02-29 05:22:14.654484-08 | 2012-02-29 05:21:28.443781-08 |             |          -1
                                                          :     sq_ast_typ_inhd.type_code                                                                        
                                                          : FROM                                                                                                 
                                                          :     sq_ast_typ_inhd                                                                                  
                                                          : WHERE                                                                                                
                                                          : (                                                                                                    
                                                          :     sq_ast_typ_inhd.inhd_type_code IN ($1)                                                           
                                                          : )                                                                                                    
                                                          :                                                                                                      
                                                          : ORDER BY sq_ast_typ_inhd.inhd_type_code, sq_ast_typ_inhd.type_code_lvl                               
    16386 | mysource_matrix |   27650 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:11.101007-08 | 2012-02-29 05:21:45.647533-08 |             |          -1
    16386 | mysource_matrix |   27463 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:14.646812-08 | 2012-02-29 05:21:01.807136-08 |             |          -1
    16386 | mysource_matrix |   27521 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:13.507651-08 | 2012-02-29 05:21:12.177954-08 |             |          -1
    16386 | mysource_matrix |   27638 |    16385 | matrix_web | SELECT l.url, l.assetid, a.name, a.short_name, a.type_code, a.status, CASE WHEN l.http = '1' THEN 'http' ELSE 'https' END as protocol, MIN(al.link_type) AS link_type                                                                                               | f       | 2012-02-29 05:22:11.927409-08 | 2012-02-29 05:22:11.927681-08 | 2012-02-29 05:21:44.780413-08 |             |          -1
                                                          :                                 FROM sq_ast_lookup l                                                 
                                                          :                                   INNER JOIN sq_ast a ON l.assetid = a.assetid                       
                                                          :                                   INNER JOIN sq_ast_lnk al ON a.assetid = al.minorid                 
                                                          :                                  WHERE  l.root_urlid = 13                                            
                                                          :                                   AND l.url IN (                                                     
                                                          :                                                                 SELECT * FROM sq_get_lineage_from_url($1)                                                                                                                                      
                                                          :                                                         )                                            
                                                          :                                 GROUP BY l.url, l.assetid, a.name, a.short_name, a.type_code, a.status, protocol                                                                                                                                               
                                                          :                                 ORDER BY l.url ASC                                                   
    16386 | mysource_matrix |   27447 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:14.045876-08 | 2012-02-29 05:20:58.051492-08 |             |          -1
    16386 | mysource_matrix |   27529 |    16385 | matrix_web | (SELECT DISTINCT(ll.majorid), ll.type_code, ll.treeid, length(ll.treeid)/$1 as depth FROM             | f       | 2012-02-29 05:22:14.609697-08 | 2012-02-29 05:22:14.609826-08 | 2012-02-29 05:21:13.379491-08 |             |          -1
                                                          :                                         (                                                            
                                                          :                                         SELECT a.assetid, l.majorid, t.treeid, a.type_code, a.name, a.short_name, a.version, a.status, a.languages, a.charset, a.force_secure, a.created, a.created_userid, a.updated, a.updated_userid, a.published, a.published_userid, a.status_changed, a.status_changed_userid FROM sq_ast_lnk_tree t                                                                             
                                                          :                                          INNER JOIN sq_ast_lnk l ON t.linkid = l.linkid              
                                                          :                                          INNER JOIN sq_ast a ON l.majorid = a.assetid WHERE   ((t.treeid IN ('00050003000C0011000A0008000F000E','00050003000C0011000A0008000F','00050003000C0011000A0008','00050003000C0011000A','00050003000C0011','00050003000C','00050003','0005','00020006000F0004000F000E','00020006000F0004000F','00020006000F0004','00020006000F','00020006','0002','00050005000P0004000I000E','00050005000P0004000I','00050005000P0004','00050005000P','00050005','00020006000F000E0001000M000E','00020006000F000E0001000M','00020006000F000E0001','00020006000F000E','00050005000H0008000M000E','0005000500                                                                                                                           
    16386 | mysource_matrix |   27561 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:14.142654-08 | 2012-02-29 05:21:22.052214-08 |             |          -1
    16386 | mysource_matrix |   27520 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:11.985714-08 | 2012-02-29 05:21:12.027195-08 |             |          -1
    16386 | mysource_matrix |   27726 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:13.342017-08 | 2012-02-29 05:22:12.320336-08 |             |          -1
    16386 | mysource_matrix |   27637 |    16385 | matrix_web | SELECT l.url, l.assetid, a.name, a.short_name, a.type_code, a.status, CASE WHEN l.http = '1' THEN 'http' ELSE 'https' END as protocol, MIN(al.link_type) AS link_type                                                                                               | f       | 2012-02-29 05:22:11.614566-08 | 2012-02-29 05:22:11.614705-08 | 2012-02-29 05:21:44.714571-08 |             |          -1
                                                          :                                 FROM sq_ast_lookup l                                                 
                                                          :                                   INNER JOIN sq_ast a ON l.assetid = a.assetid                       
                                                          :                                   INNER JOIN sq_ast_lnk al ON a.assetid = al.minorid                 
                                                          :                                  WHERE  l.root_urlid = 13                                            
                                                          :                                   AND l.url IN (                                                     
                                                          :                                                                 SELECT * FROM sq_get_lineage_from_url($1)                                                                                                                                      
                                                          :                                                         )                                            
                                                          :                                 GROUP BY l.url, l.assetid, a.name, a.short_name, a.type_code, a.status, protocol                                                                                                                                               
                                                          :                                 ORDER BY l.url ASC                                                   
    16386 | mysource_matrix |   27727 |    16385 | matrix_web |                                                                                                 | f       |                               |                               | 2012-02-29 05:22:12.562002-08 |             |          -1
    16386 | mysource_matrix |   27711 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:09.144435-08 | 2012-02-29 05:22:08.018651-08 |             |          -1
    16386 | mysource_matrix |   27491 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:14.652199-08 | 2012-02-29 05:21:06.431104-08 |             |          -1
    16386 | mysource_matrix |   27439 |    16385 | matrix_web | SELECT  l.minorid, l.majorid, l.linkid, l.value, l.link_type,                                         | f       | 2012-02-29 05:22:14.554222-08 | 2012-02-29 05:22:14.554222-08 | 2012-02-29 05:20:55.75934-08  |             |          -1
                                                          :                                                 l.type_code as major_type_code, l.sort_order, l.is_dependant, l.is_exclusive, l.locked FROM sq_vw_ast_lnk_major l  WHERE  l.minorid IN ($1)                                                                    
                                                          :                         AND (l.link_type & $2) > 0  AND l.is_dependant = $3   ORDER BY l.sort_order, l.type_code                                                                                                                                                      
    16386 | mysource_matrix |   27575 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:14.641482-08 | 2012-02-29 05:21:27.45-08     |             |          -1
    16386 | mysource_matrix |   27681 |    16385 | matrix_web | SELECT l.url, l.assetid, a.name, a.short_name, a.type_code, a.status, CASE WHEN l.http = '1' THEN 'http' ELSE 'https' END as protocol, MIN(al.link_type) AS link_type                                                                                               | f       | 2012-02-29 05:22:12.332038-08 | 2012-02-29 05:22:12.33282-08  | 2012-02-29 05:21:57.347359-08 |             |          -1
                                                          :                                 FROM sq_ast_lookup l                                                 
                                                          :                                   INNER JOIN sq_ast a ON l.assetid = a.assetid                       
                                                          :                                   INNER JOIN sq_ast_lnk al ON a.assetid = al.minorid                 
                                                          :                                  WHERE  l.root_urlid = 13                                            
                                                          :                                   AND l.url IN (                                                     
                                                          :                                                                 SELECT * FROM sq_get_lineage_from_url($1)                                                                                                                                      
                                                          :                                                         )                                            
                                                          :                                 GROUP BY l.url, l.assetid, a.name, a.short_name, a.type_code, a.status, protocol                                                                                                                                               
                                                          :                                 ORDER BY l.url ASC                                                   
    16386 | mysource_matrix |   27492 |    16385 | matrix_web | DEALLOCATE pdo_stmt_00000291                                                                          | f       |                               | 2012-02-29 05:22:14.584719-08 | 2012-02-29 05:21:06.476177-08 |             |          -1
    16386 | mysource_matrix |   27621 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:14.220845-08 | 2012-02-29 05:21:41.599393-08 |             |          -1
    16386 | mysource_matrix |   27725 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:12.342428-08 | 2012-02-29 05:22:10.970535-08 |             |          -1
    16386 | mysource_matrix |   27712 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:09.463973-08 | 2012-02-29 05:22:08.303968-08 |             |          -1
    16386 | mysource_matrix |   27636 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:13.208495-08 | 2012-02-29 05:21:43.975293-08 |             |          -1
    16386 | mysource_matrix |   27653 |    16385 | matrix_web | SELECT l.url, l.assetid, a.name, a.short_name, a.type_code, a.status, CASE WHEN l.http = '1' THEN 'http' ELSE 'https' END as protocol, MIN(al.link_type) AS link_type                                                                                               | f       | 2012-02-29 05:22:14.234352-08 | 2012-02-29 05:22:14.234541-08 | 2012-02-29 05:21:46.464439-08 |             |          -1
                                                          :                                 FROM sq_ast_lookup l                                                 
                                                          :                                   INNER JOIN sq_ast a ON l.assetid = a.assetid                       
                                                          :                                   INNER JOIN sq_ast_lnk al ON a.assetid = al.minorid                 
                                                          :                                  WHERE  l.root_urlid = 13                                            
                                                          :                                   AND l.url IN (                                                     
                                                          :                                                                 SELECT * FROM sq_get_lineage_from_url($1)                                                                                                                                      
                                                          :                                                         )                                            
                                                          :                                 GROUP BY l.url, l.assetid, a.name, a.short_name, a.type_code, a.status, protocol                                                                                                                                               
                                                          :                                 ORDER BY l.url ASC                                                   
    16386 | mysource_matrix |   27610 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:14.649113-08 | 2012-02-29 05:21:37.649046-08 |             |          -1
    16386 | mysource_matrix |   27516 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:14.64545-08  | 2012-02-29 05:21:11.463407-08 |             |          -1
    16386 | mysource_matrix |   27708 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:08.735666-08 | 2012-02-29 05:22:07.070311-08 |             |          -1
    16386 | mysource_matrix |   27569 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:12.52457-08  | 2012-02-29 05:21:25.524429-08 |             |          -1
    16386 | mysource_matrix |   27709 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:14.096508-08 | 2012-02-29 05:22:07.399365-08 |             |          -1
    16386 | mysource_matrix |   27723 |    16385 | matrix_web |                                                                                                 | f       |                               |                               | 2012-02-29 05:22:10.342483-08 |             |          -1
    16386 | mysource_matrix |   27489 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:14.616527-08 | 2012-02-29 05:21:05.846065-08 |             |          -1
    16386 | mysource_matrix |   27713 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:14.217419-08 | 2012-02-29 05:22:08.352848-08 |             |          -1
    16386 | mysource_matrix |   27501 |    16385 | matrix_web | SELECT  l.minorid, l.majorid, l.linkid, l.value, l.link_type,                                         | f       | 2012-02-29 05:22:14.581623-08 | 2012-02-29 05:22:14.581623-08 | 2012-02-29 05:21:06.782812-08 |             |          -1
                                                          :                                                 l.type_code as major_type_code, l.sort_order, l.is_dependant, l.is_exclusive, l.locked FROM sq_vw_ast_lnk_major l INNER JOIN sq_ast_perm p ON p.assetid = l.majorid LEFT JOIN sq_vw_ast_role r ON (p.userid = r.roleid AND r.assetid = l.minorid)  WHERE  l.minorid IN ($1)                                                                                                    
                                                          :                         AND (l.link_type & $2) > 0  AND (p.userid IN ($3,$4,$5) OR r.userid IN ($6,$7,$8))                                                                                                                                                      
                                                          :                                         AND (                                                        
                                                          :                                                 (p.permission = 1 AND (                              
                                                          :                                                                         p.userid <> $9               
                                                          :                                                                         OR r.userid <> $10           
                                                          :                                                                         OR (p.userid = $11 AND p.granted = '1')                                                                                                                                                 
                                                          :                                                                         OR (r.userid = $12 AND p.granted = '1')                                                                                                                                                 
                                                          :                                                                 )                                    
                                                          :                                                 ) OR (p.permission > $13 AND p.granted = '1')) AND l.type_code IN (                                                                                                                                                 
                                                          :                                                                                                 SELECT type_code                                                                                                                                                 
                                                          :                                                                                                 FROM sq_ast_typ_inhd                                                                                                                                                  
                                                          :                                                                                                 WHERE inhd_type_code IN ($14)                                                                                                                                          
                                                          :                                                                                                 ) GROUP BY l.linkid, l.minorid, l.majorid, l.value, l.link_type,                                                                                                   
                                                          :                                                                   l.type_code, l.sort_order, l.is_dependant, l.is_exclusive, l.locked, p.assetid HAVING MIN(p.granted) <> '0' ORDER BY l.sort_order, l.ty                                                     
    16386 | mysource_matrix |   27628 |    16385 | matrix_web | SELECT l.url, l.assetid, a.name, a.short_name, a.type_code, a.status, CASE WHEN l.http = '1' THEN 'http' ELSE 'https' END as protocol, MIN(al.link_type) AS link_type                                                                                               | f       | 2012-02-29 05:22:12.686659-08 | 2012-02-29 05:22:12.686945-08 | 2012-02-29 05:21:43.087766-08 |             |          -1
                                                          :                                 FROM sq_ast_lookup l                                                 
                                                          :                                   INNER JOIN sq_ast a ON l.assetid = a.assetid                       
                                                          :                                   INNER JOIN sq_ast_lnk al ON a.assetid = al.minorid                 
                                                          :                                  WHERE  l.root_urlid = 13                                            
                                                          :                                   AND l.url IN (                                                     
                                                          :                                                                 SELECT * FROM sq_get_lineage_from_url($1)                                                                                                                                      
                                                          :                                                         )                                            
                                                          :                                 GROUP BY l.url, l.assetid, a.name, a.short_name, a.type_code, a.status, protocol                                                                                                                                               
                                                          :                                 ORDER BY l.url ASC                                                   
    16386 | mysource_matrix |   27699 |    16385 | matrix_web | SELECT l.url, l.assetid, a.name, a.short_name, a.type_code, a.status, CASE WHEN l.http = '1' THEN 'http' ELSE 'https' END as protocol, MIN(al.link_type) AS link_type                                                                                               | f       | 2012-02-29 05:22:13.613739-08 | 2012-02-29 05:22:13.614071-08 | 2012-02-29 05:22:03.987491-08 |             |          -1
                                                          :                                 FROM sq_ast_lookup l                                                 
                                                          :                                   INNER JOIN sq_ast a ON l.assetid = a.assetid                       
                                                          :                                   INNER JOIN sq_ast_lnk al ON a.assetid = al.minorid                 
                                                          :                                  WHERE  l.root_urlid = 13                                            
                                                          :                                   AND l.url IN (                                                     
                                                          :                                                                 SELECT * FROM sq_get_lineage_from_url($1)                                                                                                                                      
                                                          :                                                         )                                            
                                                          :                                 GROUP BY l.url, l.assetid, a.name, a.short_name, a.type_code, a.status, protocol                                                                                                                                               
                                                          :                                 ORDER BY l.url ASC                                                   
    16386 | mysource_matrix |   27710 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:08.765236-08 | 2012-02-29 05:22:07.549149-08 |             |          -1
    16386 | mysource_matrix |   27565 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:14.569013-08 | 2012-02-29 05:21:23.265678-08 |             |          -1
    16386 | mysource_matrix |   27728 |    16385 | matrix_web |                                                                                                 | f       |                               |                               | 2012-02-29 05:22:12.63204-08  |             |          -1
    16386 | mysource_matrix |   27714 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:14.506968-08 | 2012-02-29 05:22:08.372593-08 |             |          -1
    16386 | mysource_matrix |   27680 |    16385 | matrix_web | SELECT l.url, l.assetid, a.name, a.short_name, a.type_code, a.status, CASE WHEN l.http = '1' THEN 'http' ELSE 'https' END as protocol, MIN(al.link_type) AS link_type                                                                                               | f       | 2012-02-29 05:22:11.481805-08 | 2012-02-29 05:22:11.482464-08 | 2012-02-29 05:21:56.815447-08 |             |          -1
                                                          :                                 FROM sq_ast_lookup l                                                 
                                                          :                                   INNER JOIN sq_ast a ON l.assetid = a.assetid                       
                                                          :                                   INNER JOIN sq_ast_lnk al ON a.assetid = al.minorid                 
                                                          :                                  WHERE  l.root_urlid = 13                                            
                                                          :                                   AND l.url IN (                                                     
                                                          :                                                                 SELECT * FROM sq_get_lineage_from_url($1)                                                                                                                                      
                                                          :                                                         )                                            
                                                          :                                 GROUP BY l.url, l.assetid, a.name, a.short_name, a.type_code, a.status, protocol                                                                                                                                               
                                                          :                                 ORDER BY l.url ASC                                                   
    16386 | mysource_matrix |   27674 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:14.540454-08 | 2012-02-29 05:21:53.651338-08 |             |          -1
    16386 | mysource_matrix |   27519 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:14.624831-08 | 2012-02-29 05:21:11.802905-08 |             |          -1
    16386 | mysource_matrix |   27522 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:11.278017-08 | 2012-02-29 05:21:12.201854-08 |             |          -1
    16386 | mysource_matrix |   27696 |    16385 | matrix_web | SELECT                                                                                                | f       |                               | 2012-02-29 05:22:14.651631-08 | 2012-02-29 05:22:03.84097-08  |             |          -1
                                                          :     sq_cache.expires, sq_cache.path                                                                  
                                                          : FROM                                                                                                 
                                                          :     sq_cache                                                                                         
                                                          : WHERE                                                                                                
                                                          : (                                                                                                    
                                                          :     sq_cache.cache_key = ($1)                                                                        
                                                          :     AND sq_cache.perm_key = ($2)                                                                     
                                                          :     AND sq_cache.url = ($3)                                                                          
                                                          : )                                                                                                    
                                                          :                                                                                                      
    16386 | mysource_matrix |   27715 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:09.816241-08 | 2012-02-29 05:22:08.651401-08 |             |          -1
    16386 | mysource_matrix |   27629 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:13.280323-08 | 2012-02-29 05:21:43.168765-08 |             |          -1
    16386 | mysource_matrix |   27570 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:11.47262-08  | 2012-02-29 05:21:25.715369-08 |             |          -1
    16386 | mysource_matrix |   27583 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:14.647497-08 | 2012-02-29 05:21:29.416055-08 |             |          -1
    16386 | mysource_matrix |   27716 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:14.06169-08  | 2012-02-29 05:22:08.911317-08 |             |          -1
    16386 | mysource_matrix |   27537 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:14.63377-08  | 2012-02-29 05:21:15.323747-08 |             |          -1
    16386 | mysource_matrix |   27538 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:14.562349-08 | 2012-02-29 05:21:15.46449-08  |             |          -1
    16386 | mysource_matrix |   27546 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:14.596611-08 | 2012-02-29 05:21:17.787425-08 |             |          -1
    16386 | mysource_matrix |   27545 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:14.653016-08 | 2012-02-29 05:21:17.74755-08  |             |          -1
    16386 | mysource_matrix |   27611 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:08.206541-08 | 2012-02-29 05:21:37.675313-08 |             |          -1
    16386 | mysource_matrix |   27675 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:14.518031-08 | 2012-02-29 05:21:53.879824-08 |             |          -1
    16386 | mysource_matrix |   27553 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:14.57785-08  | 2012-02-29 05:21:19.15138-08  |             |          -1
    16386 | mysource_matrix |   27648 |    16385 | matrix_web | SELECT l.url, l.assetid, a.name, a.short_name, a.type_code, a.status, CASE WHEN l.http = '1' THEN 'http' ELSE 'https' END as protocol, MIN(al.link_type) AS link_type                                                                                               | f       | 2012-02-29 05:22:13.640553-08 | 2012-02-29 05:22:13.640901-08 | 2012-02-29 05:21:45.107518-08 |             |          -1
                                                          :                                 FROM sq_ast_lookup l                                                 
                                                          :                                   INNER JOIN sq_ast a ON l.assetid = a.assetid                       
                                                          :                                   INNER JOIN sq_ast_lnk al ON a.assetid = al.minorid                 
                                                          :                                  WHERE  l.root_urlid = 13                                            
                                                          :                                   AND l.url IN (                                                     
                                                          :                                                                 SELECT * FROM sq_get_lineage_from_url($1)                                                                                                                                      
                                                          :                                                         )                                            
                                                          :                                 GROUP BY l.url, l.assetid, a.name, a.short_name, a.type_code, a.status, protocol                                                                                                                                               
                                                          :                                 ORDER BY l.url ASC                                                   
    16386 | mysource_matrix |   27717 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:14.573078-08 | 2012-02-29 05:22:08.969216-08 |             |          -1
    16386 | mysource_matrix |   27649 |    16385 | matrix_web | SELECT l.url, l.assetid, a.name, a.short_name, a.type_code, a.status, CASE WHEN l.http = '1' THEN 'http' ELSE 'https' END as protocol, MIN(al.link_type) AS link_type                                                                                               | f       | 2012-02-29 05:22:11.805074-08 | 2012-02-29 05:22:11.805813-08 | 2012-02-29 05:21:45.237933-08 |             |          -1
                                                          :                                 FROM sq_ast_lookup l                                                 
                                                          :                                   INNER JOIN sq_ast a ON l.assetid = a.assetid                       
                                                          :                                   INNER JOIN sq_ast_lnk al ON a.assetid = al.minorid                 
                                                          :                                  WHERE  l.root_urlid = 13                                            
                                                          :                                   AND l.url IN (                                                     
                                                          :                                                                 SELECT * FROM sq_get_lineage_from_url($1)                                                                                                                                      
                                                          :                                                         )                                            
                                                          :                                 GROUP BY l.url, l.assetid, a.name, a.short_name, a.type_code, a.status, protocol                                                                                                                                               
                                                          :                                 ORDER BY l.url ASC                                                   
    16386 | mysource_matrix |   27557 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:14.653993-08 | 2012-02-29 05:21:20.231838-08 |             |          -1
    16386 | mysource_matrix |   27558 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:10.738948-08 | 2012-02-29 05:21:21.271337-08 |             |          -1
    16386 | mysource_matrix |   27559 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:14.629992-08 | 2012-02-29 05:21:21.5018-08   |             |          -1
    11564 | postgres        |   27729 |       10 | postgres   | SELECT * FROM pg_stat_activity;                                                                       | f       | 2012-02-29 05:22:14.396493-08 | 2012-02-29 05:22:14.396493-08 | 2012-02-29 05:22:13.925439-08 |             |          -1
    16386 | mysource_matrix |   27566 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:13.411204-08 | 2012-02-29 05:21:23.303327-08 |             |          -1
    16386 | mysource_matrix |   27048 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:18:53.087581-08 | 2012-02-29 05:17:31.123626-08 |             |          -1
    16386 | mysource_matrix |   27571 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:14.646916-08 | 2012-02-29 05:21:25.923099-08 |             |          -1
    16386 | mysource_matrix |   27572 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:12.046363-08 | 2012-02-29 05:21:26.343887-08 |             |          -1
    16386 | mysource_matrix |   27574 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:14.371174-08 | 2012-02-29 05:21:26.859306-08 |             |          -1
    16386 | mysource_matrix |   27584 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:11.068697-08 | 2012-02-29 05:21:29.834053-08 |             |          -1
    16386 | mysource_matrix |   27585 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:14.245422-08 | 2012-02-29 05:21:30.571454-08 |             |          -1
    16386 | mysource_matrix |   27724 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:11.749047-08 | 2012-02-29 05:22:10.620102-08 |             |          -1
    16386 | mysource_matrix |   27587 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:14.61938-08  | 2012-02-29 05:21:31.023725-08 |             |          -1
    16386 | mysource_matrix |   27589 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:13.65201-08  | 2012-02-29 05:21:31.154654-08 |             |          -1
    16386 | mysource_matrix |   27590 |    16385 | matrix_web | SELECT DISTINCT a.assetid, l.majorid, a.type_code, a.status, a.name, a.short_name, pt.path, l.sort_order                                                                                                                                                      | f       |                               | 2012-02-29 05:22:14.59255-08  | 2012-02-29 05:21:31.367384-08 |             |          -1
                                                          :                            FROM sq_ast a                                                             
                                                          :                                  INNER JOIN sq_ast_lnk l ON a.assetid = l.minorid                    
                                                          :                                  INNER JOIN sq_ast_path pt ON a.assetid = pt.assetid                 
                                                          :                                  INNER JOIN sq_ast_perm p ON a.assetid = p.assetid                   
                                                          :                                  LEFT JOIN sq_vw_ast_role r ON p.userid = r.roleid AND p.assetid = r.assetid  WHERE    l.majorid IN ('58014')                                                                                                                       
                                                          :                                  AND (l.link_type & 1) > 0 AND (p.userid  IN ($1, $2, $3) OR r.userid IN ($4, $5, $6))  AND (                                                                                                                                          
                                                          :                                 (p.permission = $7 AND (                                             
                                                          :                                                 p.userid <> $8                                       
                                                          :                                                 OR r.userid <> $9                                    
                                                          :                                                 OR (p.userid = $10 AND p.granted = '1')              
                                                          :                                                 OR (r.userid = $11 AND p.granted = '1')              
                                                          :                                         )                                                            
                                                          :                                 )                                                                    
                                                          :                                 OR      (                                                            
                                                          :                                                 p.permission > $12 AND p.granted = '1'               
                                                          :                                         )                                                            
                                                          :                         )                                                                            
                                                          :                                                           GROUP BY a.assetid, l.majorid, a.type_code, a.status, a.name, a.short_name, pt.path, l.sort_order, p.assetid                                                                                                 
                                                          :                                                           HAVING MIN(p.granted) <> '0'               
                                                          :                            ORDER BY l.majorid, l.sort_order                                          
    16386 | mysource_matrix |   27677 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:08.313736-08 | 2012-02-29 05:21:55.126947-08 |             |          -1
    16386 | mysource_matrix |   27594 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:13.755029-08 | 2012-02-29 05:21:31.781899-08 |             |          -1
    16386 | mysource_matrix |   27596 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:13.919734-08 | 2012-02-29 05:21:32.413323-08 |             |          -1
    16386 | mysource_matrix |   27601 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:14.516277-08 | 2012-02-29 05:21:33.843016-08 |             |          -1
    16386 | mysource_matrix |   27656 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:13.657782-08 | 2012-02-29 05:21:47.142998-08 |             |          -1
    16386 | mysource_matrix |   27603 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:14.544729-08 | 2012-02-29 05:21:34.339694-08 |             |          -1
    16386 | mysource_matrix |   27604 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:06.063141-08 | 2012-02-29 05:21:34.359029-08 |             |          -1
    16386 | mysource_matrix |   27683 |    16385 | matrix_web | SELECT l.url, l.assetid, a.name, a.short_name, a.type_code, a.status, CASE WHEN l.http = '1' THEN 'http' ELSE 'https' END as protocol, MIN(al.link_type) AS link_type                                                                                               | f       | 2012-02-29 05:22:12.966789-08 | 2012-02-29 05:22:12.967026-08 | 2012-02-29 05:21:58.180403-08 |             |          -1
                                                          :                                 FROM sq_ast_lookup l                                                 
                                                          :                                   INNER JOIN sq_ast a ON l.assetid = a.assetid                       
                                                          :                                   INNER JOIN sq_ast_lnk al ON a.assetid = al.minorid                 
                                                          :                                  WHERE  l.root_urlid = 13                                            
                                                          :                                   AND l.url IN (                                                     
                                                          :                                                                 SELECT * FROM sq_get_lineage_from_url($1)                                                                                                                                      
                                                          :                                                         )                                            
                                                          :                                 GROUP BY l.url, l.assetid, a.name, a.short_name, a.type_code, a.status, protocol                                                                                                                                               
                                                          :                                 ORDER BY l.url ASC                                                   
    16386 | mysource_matrix |   27684 |    16385 | matrix_web | SELECT l.url, l.assetid, a.name, a.short_name, a.type_code, a.status, CASE WHEN l.http = '1' THEN 'http' ELSE 'https' END as protocol, MIN(al.link_type) AS link_type                                                                                               | f       | 2012-02-29 05:22:11.952112-08 | 2012-02-29 05:22:11.952661-08 | 2012-02-29 05:21:58.190629-08 |             |          -1
                                                          :                                 FROM sq_ast_lookup l                                                 
                                                          :                                   INNER JOIN sq_ast a ON l.assetid = a.assetid                       
                                                          :                                   INNER JOIN sq_ast_lnk al ON a.assetid = al.minorid                 
                                                          :                                  WHERE  l.root_urlid = 13                                            
                                                          :                                   AND l.url IN (                                                     
                                                          :                                                                 SELECT * FROM sq_get_lineage_from_url($1)                                                                                                                                      
                                                          :                                                         )                                            
                                                          :                                 GROUP BY l.url, l.assetid, a.name, a.short_name, a.type_code, a.status, protocol                                                                                                                                               
                                                          :                                 ORDER BY l.url ASC                                                   
    16386 | mysource_matrix |   27607 |    16385 | matrix_web | SELECT DISTINCT a.assetid, l.majorid, a.type_code, a.status, a.name, a.short_name, pt.path, l.sort_order                                                                                                                                                      | f       | 2012-02-29 05:22:14.519129-08 | 2012-02-29 05:22:14.519129-08 | 2012-02-29 05:21:35.277818-08 |             |          -1
                                                          :                            FROM sq_ast a                                                             
                                                          :                                  INNER JOIN sq_ast_lnk l ON a.assetid = l.minorid                    
                                                          :                                  INNER JOIN sq_ast_path pt ON a.assetid = pt.assetid                 
                                                          :                                  INNER JOIN sq_ast_perm p ON a.assetid = p.assetid                   
                                                          :                                  LEFT JOIN sq_vw_ast_role r ON p.userid = r.roleid AND p.assetid = r.assetid  WHERE    l.majorid IN ('46')                                                                                                                          
                                                          :                                  AND (l.link_type & 1) > 0 AND (p.userid  IN ($1, $2, $3) OR r.userid IN ($4, $5, $6))  AND (                                                                                                                                          
                                                          :                                 (p.permission = $7 AND (                                             
                                                          :                                                 p.userid <> $8                                       
                                                          :                                                 OR r.userid <> $9                                    
                                                          :                                                 OR (p.userid = $10 AND p.granted = '1')              
                                                          :                                                 OR (r.userid = $11 AND p.granted = '1')              
                                                          :                                         )                                                            
                                                          :                                 )                                                                    
                                                          :                                 OR      (                                                            
                                                          :                                                 p.permission > $12 AND p.granted = '1'               
                                                          :                                         )                                                            
                                                          :                         )                                                                            
                                                          :                                                           GROUP BY a.assetid, l.majorid, a.type_code, a.status, a.name, a.short_name, pt.path, l.sort_order, p.assetid                                                                                                 
                                                          :                                                           HAVING MIN(p.granted) <> '0'               
                                                          :                            ORDER BY l.majorid, l.sort_order                                          
    16386 | mysource_matrix |   27609 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:12.822155-08 | 2012-02-29 05:21:36.511386-08 |             |          -1
    16386 | mysource_matrix |   27698 |    16385 | matrix_web | SELECT l.url, l.assetid, a.name, a.short_name, a.type_code, a.status, CASE WHEN l.http = '1' THEN 'http' ELSE 'https' END as protocol, MIN(al.link_type) AS link_type                                                                                               | f       | 2012-02-29 05:22:14.06043-08  | 2012-02-29 05:22:14.060602-08 | 2012-02-29 05:22:03.947356-08 |             |          -1
                                                          :                                 FROM sq_ast_lookup l                                                 
                                                          :                                   INNER JOIN sq_ast a ON l.assetid = a.assetid                       
                                                          :                                   INNER JOIN sq_ast_lnk al ON a.assetid = al.minorid                 
                                                          :                                  WHERE  l.root_urlid = 13                                            
                                                          :                                   AND l.url IN (                                                     
                                                          :                                                                 SELECT * FROM sq_get_lineage_from_url($1)                                                                                                                                      
                                                          :                                                         )                                            
                                                          :                                 GROUP BY l.url, l.assetid, a.name, a.short_name, a.type_code, a.status, protocol                                                                                                                                               
                                                          :                                 ORDER BY l.url ASC                                                   
    16386 | mysource_matrix |   27630 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:13.805683-08 | 2012-02-29 05:21:43.271387-08 |             |          -1
    16386 | mysource_matrix |   27657 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:14.624329-08 | 2012-02-29 05:21:47.944135-08 |             |          -1
    16386 | mysource_matrix |   27658 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:14.228413-08 | 2012-02-29 05:21:48.595872-08 |             |          -1
    16386 | mysource_matrix |   27659 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:11.829464-08 | 2012-02-29 05:21:48.647435-08 |             |          -1
    16386 | mysource_matrix |   27660 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:13.625408-08 | 2012-02-29 05:21:49.103529-08 |             |          -1
    16386 | mysource_matrix |   27094 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:18:49.90279-08  | 2012-02-29 05:17:55.492714-08 |             |          -1
    16386 | mysource_matrix |   27661 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:12.460474-08 | 2012-02-29 05:21:49.64548-08  |             |          -1
    16386 | mysource_matrix |   27662 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:14.172023-08 | 2012-02-29 05:21:49.871414-08 |             |          -1
    16386 | mysource_matrix |   27663 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:14.587096-08 | 2012-02-29 05:21:50.040005-08 |             |          -1
    16386 | mysource_matrix |   27667 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:14.333256-08 | 2012-02-29 05:21:51.083658-08 |             |          -1
    16386 | mysource_matrix |   27668 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:14.334118-08 | 2012-02-29 05:21:51.768765-08 |             |          -1
    16386 | mysource_matrix |   27666 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:13.381337-08 | 2012-02-29 05:21:50.831367-08 |             |          -1
    16386 | mysource_matrix |   27672 |    16385 | matrix_web |                                                                                                 | f       |                               | 2012-02-29 05:22:14.12038-08  | 2012-02-29 05:21:52.527446-08 |             |          -1
    (98 rows)
    
    \n------PROCESSES-------
     PID TTY          TIME CMD
       1 ?        00:00:30 init
       2 ?        00:00:00 kthreadd
       3 ?        00:00:02 migration/0
       4 ?        00:00:03 ksoftirqd/0
       5 ?        00:00:00 watchdog/0
       6 ?        00:00:02 migration/1
       7 ?        00:00:01 ksoftirqd/1
       8 ?        00:00:00 watchdog/1
       9 ?        00:00:09 events/0
      10 ?        00:00:59 events/1
      11 ?        00:00:00 cpuset
      12 ?        00:00:00 khelper
      13 ?        00:00:00 netns
      14 ?        00:00:00 async/mgr
      15 ?        00:00:00 pm
      16 ?        00:00:03 sync_supers
      17 ?        00:00:02 bdi-default
      18 ?        00:00:00 kintegrityd/0
      19 ?        00:00:00 kintegrityd/1
      20 ?        00:02:33 kblockd/0
      21 ?        00:02:19 kblockd/1
      22 ?        00:00:00 kacpid
      23 ?        00:00:00 kacpi_notify
      24 ?        00:00:00 kacpi_hotplug
      25 ?        00:00:00 kseriod
      26 ?        00:00:00 kondemand/0
      27 ?        00:00:00 kondemand/1
      30 ?        00:00:00 khungtaskd
      31 ?        00:02:03 kswapd0
      32 ?        00:00:00 ksmd
      33 ?        00:00:00 aio/0
      34 ?        00:00:00 aio/1
      35 ?        00:00:00 crypto/0
      36 ?        00:00:00 crypto/1
     153 ?        00:00:00 ata/0
     154 ?        00:00:00 ata/1
     155 ?        00:00:00 ata_aux
     157 ?        00:00:00 scsi_eh_0
     159 ?        00:00:00 scsi_eh_1
     194 ?        00:02:30 flush-8:0
     195 ?        00:07:41 kjournald
     242 ?        00:00:00 udevd
     373 ?        00:00:00 kpsmoused
     713 ?        00:01:03 rsyslogd
     742 ?        00:00:02 acpid
     791 ?        00:00:29 cron
     982 ?        00:00:00 udevd
     983 ?        00:00:00 udevd
    1068 ?        00:00:00 sshd
    1166 ?        00:04:51 apache2
    1469 ?        00:00:00 squid
    26018 ?        00:00:00 sshd
    27685 ?        00:00:00 cron
    27695 ?        00:00:00 sh
    27697 ?        00:00:00 pgstats.sh
    27742 ?        00:00:00 ps
    31604 ?        00:00:20 python
    \n------LOCKS-------
     locktype  | database | relation | page | tuple | virtualxid | transactionid | classid | objid | objsubid | virtualtransaction |  pid  |       mode       | granted 
    ------------+----------+----------+------+-------+------------+---------------+---------+-------+----------+--------------------+-------+------------------+---------
    relation   |    16386 |    16994 |      |       |            |               |         |       |          | 49/3355            | 27537 | AccessShareLock  | t
    relation   |    16386 |    17055 |      |       |            |               |         |       |          | 72/5232            | 27587 | AccessShareLock  | t
    relation   |    16386 |    16405 |      |       |            |               |         |       |          | 12/14404           | 27727 | AccessShareLock  | t
    relation   |    16386 |    16452 |      |       |            |               |         |       |          | 15/0               | 27439 | AccessShareLock  | t
    relation   |    16386 |    17054 |      |       |            |               |         |       |          | 81/286             | 27749 | AccessShareLock  | t
    relation   |    16386 |    17067 |      |       |            |               |         |       |          | 59/4648            | 27557 | AccessShareLock  | t
    relation   |    16386 |    17056 |      |       |            |               |         |       |          | 15/0               | 27439 | AccessShareLock  | t
    relation   |    16386 |    16462 |      |       |            |               |         |       |          | 15/0               | 27439 | AccessShareLock  | t
    relation   |    16386 |    17077 |      |       |            |               |         |       |          | 20/4820            | 27725 | AccessShareLock  | t
    relation   |    16386 |    16405 |      |       |            |               |         |       |          | 55/0               | 27748 | AccessShareLock  | t
    relation   |    16386 |    16902 |      |       |            |               |         |       |          | 49/3355            | 27537 | AccessShareLock  | t
    relation   |    16386 |    16906 |      |       |            |               |         |       |          | 40/1870            | 27674 | AccessShareLock  | t
    virtualxid |          |          |      |       | 95/2961    |               |         |       |          | 95/2961            | 27663 | ExclusiveLock    | t
    relation   |    16386 |    16405 |      |       |            |               |         |       |          | 95/2961            | 27663 | AccessShareLock  | t
    relation   |    16386 |    16452 |      |       |            |               |         |       |          | 96/1390            | 27745 | AccessShareLock  | t
    relation   |    16386 |    17067 |      |       |            |               |         |       |          | 49/3355            | 27537 | AccessShareLock  | t
    virtualxid |          |          |      |       | 20/4820    |               |         |       |          | 20/4820            | 27725 | ExclusiveLock    | t
    virtualxid |          |          |      |       | 49/3355    |               |         |       |          | 49/3355            | 27537 | ExclusiveLock    | t
    relation   |    16386 |    17058 |      |       |            |               |         |       |          | 72/5232            | 27587 | AccessShareLock  | t
    relation   |    16386 |    17002 |      |       |            |               |         |       |          | 49/3355            | 27537 | AccessShareLock  | t
    relation   |    16386 |    17065 |      |       |            |               |         |       |          | 72/5232            | 27587 | AccessShareLock  | t
    relation   |    16386 |    17072 |      |       |            |               |         |       |          | 72/5232            | 27587 | AccessShareLock  | t
    relation   |    16386 |    16918 |      |       |            |               |         |       |          | 55/0               | 27748 | AccessShareLock  | t
    relation   |    16386 |    16452 |      |       |            |               |         |       |          | 29/6478            | 27723 | AccessShareLock  | t
    relation   |    16386 |    17063 |      |       |            |               |         |       |          | 81/286             | 27749 | AccessShareLock  | t
    relation   |    16386 |    17058 |      |       |            |               |         |       |          | 37/7678            | 27728 | AccessShareLock  | t
    relation   |    16386 |    17057 |      |       |            |               |         |       |          | 72/5232            | 27587 | AccessShareLock  | t
    relation   |    16386 |    16566 |      |       |            |               |         |       |          | 20/4820            | 27725 | AccessShareLock  | t
    relation   |    16386 |    16566 |      |       |            |               |         |       |          | 20/4820            | 27725 | RowExclusiveLock | t
    virtualxid |          |          |      |       | 18/7649    |               |         |       |          | 18/0               | 27758 | ExclusiveLock    | t
    relation   |    16386 |    16469 |      |       |            |               |         |       |          | 37/7678            | 27728 | AccessShareLock  | t
    relation   |    16386 |    16906 |      |       |            |               |         |       |          | 96/1390            | 27745 | AccessShareLock  | t
    virtualxid |          |          |      |       | 55/2234    |               |         |       |          | 55/0               | 27748 | ExclusiveLock    | t
    relation   |    16386 |    17002 |      |       |            |               |         |       |          | 15/0               | 27439 | AccessShareLock  | t
    relation   |    16386 |    17066 |      |       |            |               |         |       |          | 72/5232            | 27587 | AccessShareLock  | t
    relation   |    16386 |    16469 |      |       |            |               |         |       |          | 95/2961            | 27663 | AccessShareLock  | t
    relation   |    16386 |    17054 |      |       |            |               |         |       |          | 49/3355            | 27537 | AccessShareLock  | t
    relation   |    16386 |    16932 |      |       |            |               |         |       |          | 20/4820            | 27725 | AccessShareLock  | t
    relation   |    16386 |    16452 |      |       |            |               |         |       |          | 40/1870            | 27674 | AccessShareLock  | t
    relation   |    16386 |    17058 |      |       |            |               |         |       |          | 12/14404           | 27727 | AccessShareLock  | t
    relation   |    16386 |    16508 |      |       |            |               |         |       |          | 72/5232            | 27587 | AccessShareLock  | t
    relation   |    16386 |    17058 |      |       |            |               |         |       |          | 95/2961            | 27663 | AccessShareLock  | t
    relation   |    16386 |    17054 |      |       |            |               |         |       |          | 59/4648            | 27557 | AccessShareLock  | t
    relation   |    16386 |    16918 |      |       |            |               |         |       |          | 72/5232            | 27587 | AccessShareLock  | t
    relation   |    16386 |    17072 |      |       |            |               |         |       |          | 55/0               | 27748 | AccessShareLock  | t
    relation   |    16386 |    16469 |      |       |            |               |         |       |          | 12/14404           | 27727 | AccessShareLock  | t
    relation   |    16386 |    17063 |      |       |            |               |         |       |          | 49/3355            | 27537 | AccessShareLock  | t
    relation   |    16386 |    16405 |      |       |            |               |         |       |          | 72/5232            | 27587 | AccessShareLock  | t
    virtualxid |          |          |      |       | 15/7538    |               |         |       |          | 15/0               | 27439 | ExclusiveLock    | t
    relation   |    16386 |    17056 |      |       |            |               |         |       |          | 49/3355            | 27537 | AccessShareLock  | t
    relation   |    16386 |    16405 |      |       |            |               |         |       |          | 37/7678            | 27728 | AccessShareLock  | t
    relation   |    16386 |    16452 |      |       |            |               |         |       |          | 49/3355            | 27537 | AccessShareLock  | t
    relation   |    16386 |    16902 |      |       |            |               |         |       |          | 15/0               | 27439 | AccessShareLock  | t
    relation   |    16386 |     2603 |      |       |            |               |         |       |          | 50/3710            | 27759 | AccessShareLock  | t
    relation   |    16386 |    16415 |      |       |            |               |         |       |          | 1/3870204          | 27578 | AccessShareLock  | t
    relation   |    16386 |    17063 |      |       |            |               |         |       |          | 59/4648            | 27557 | AccessShareLock  | t
    virtualxid |          |          |      |       | 81/286     |               |         |       |          | 81/286             | 27749 | ExclusiveLock    | t
    relation   |    16386 |    16906 |      |       |            |               |         |       |          | 29/6478            | 27723 | AccessShareLock  | t
    relation   |    16386 |    16405 |      |       |            |               |         |       |          | 59/4648            | 27557 | AccessShareLock  | t
    relation   |    16386 |    16513 |      |       |            |               |         |       |          | 72/5232            | 27587 | AccessShareLock  | t
    relation   |    16386 |    16880 |      |       |            |               |         |       |          | 49/3355            | 27537 | AccessShareLock  | t
    relation   |    16386 |    16469 |      |       |            |               |         |       |          | 29/6478            | 27723 | AccessShareLock  | t
    virtualxid |          |          |      |       | 62/3877    |               |         |       |          | 62/3877            | 27741 | ExclusiveLock    | t
    virtualxid |          |          |      |       | 59/4648    |               |         |       |          | 59/4648            | 27557 | ExclusiveLock    | t
    relation   |    16386 |    17064 |      |       |            |               |         |       |          | 72/5232            | 27587 | AccessShareLock  | t
    relation   |    16386 |    16405 |      |       |            |               |         |       |          | 49/3355            | 27537 | AccessShareLock  | t
    relation   |    16386 |    16452 |      |       |            |               |         |       |          | 37/7678            | 27728 | AccessShareLock  | t
    relation   |    16386 |    17063 |      |       |            |               |         |       |          | 72/5232            | 27587 | AccessShareLock  | t
    relation   |    16386 |    17058 |      |       |            |               |         |       |          | 29/6478            | 27723 | AccessShareLock  | t
    virtualxid |          |          |      |       | 37/7678    |               |         |       |          | 37/7678            | 27728 | ExclusiveLock    | t
    relation   |    16386 |    17196 |      |       |            |               |         |       |          | 49/3355            | 27537 | AccessShareLock  | t
    relation   |    16386 |    17055 |      |       |            |               |         |       |          | 15/0               | 27439 | AccessShareLock  | t
    relation   |    16386 |    16452 |      |       |            |               |         |       |          | 72/5232            | 27587 | AccessShareLock  | t
    relation   |    16386 |    17067 |      |       |            |               |         |       |          | 55/0               | 27748 | AccessShareLock  | t
    relation   |    16386 |    17056 |      |       |            |               |         |       |          | 72/5232            | 27587 | AccessShareLock  | t
    relation   |    16386 |    17057 |      |       |            |               |         |       |          | 15/0               | 27439 | AccessShareLock  | t
    relation   |    16386 |    17058 |      |       |            |               |         |       |          | 96/1390            | 27745 | AccessShareLock  | t
    virtualxid |          |          |      |       | 29/6478    |               |         |       |          | 29/6478            | 27723 | ExclusiveLock    | t
    relation   |    16386 |    16566 |      |       |            |               |         |       |          | 50/3710            | 27759 | AccessShareLock  | t
    relation   |    16386 |    16906 |      |       |            |               |         |       |          | 12/14404           | 27727 | AccessShareLock  | t
    relation   |    16386 |    16918 |      |       |            |               |         |       |          | 49/3355            | 27537 | AccessShareLock  | t
    relation   |    16386 |    16906 |      |       |            |               |         |       |          | 95/2961            | 27663 | AccessShareLock  | t
    relation   |    16386 |    16405 |      |       |            |               |         |       |          | 40/1870            | 27674 | AccessShareLock  | t
    relation   |    16386 |    17054 |      |       |            |               |         |       |          | 72/5232            | 27587 | AccessShareLock  | t
    virtualxid |          |          |      |       | 44/2371    |               |         |       |          | 44/2371            | 27715 | ExclusiveLock    | t
    relation   |    16386 |    17058 |      |       |            |               |         |       |          | 15/0               | 27439 | AccessShareLock  | t
    relation   |    16386 |    16469 |      |       |            |               |         |       |          | 96/1390            | 27745 | AccessShareLock  | t
    relation   |    16386 |    17071 |      |       |            |               |         |       |          | 55/0               | 27748 | AccessShareLock  | t
    relation   |    16386 |    17049 |      |       |            |               |         |       |          | 1/3870204          | 27578 | AccessShareLock  | t
    relation   |    16386 |    16520 |      |       |            |               |         |       |          | 72/5232            | 27587 | AccessShareLock  | t
    relation   |    16386 |    16566 |      |       |            |               |         |       |          | 18/0               | 27758 | AccessShareLock  | t
    relation   |    16386 |    16914 |      |       |            |               |         |       |          | 72/5232            | 27587 | AccessShareLock  | t
    relation   |    16386 |    17059 |      |       |            |               |         |       |          | 15/0               | 27439 | AccessShareLock  | t
    relation   |    16386 |    16918 |      |       |            |               |         |       |          | 59/4648            | 27557 | AccessShareLock  | t
    virtualxid |          |          |      |       | 72/5232    |               |         |       |          | 72/5232            | 27587 | ExclusiveLock    | t
    relation   |    16386 |    17057 |      |       |            |               |         |       |          | 49/3355            | 27537 | AccessShareLock  | t
    virtualxid |          |          |      |       | 12/14404   |               |         |       |          | 12/14404           | 27727 | ExclusiveLock    | t
    relation   |    16386 |    16566 |      |       |            |               |         |       |          | 44/2371            | 27715 | AccessShareLock  | t
    virtualxid |          |          |      |       | 96/1390    |               |         |       |          | 96/1390            | 27745 | ExclusiveLock    | t
    relation   |    16386 |    17054 |      |       |            |               |         |       |          | 55/0               | 27748 | AccessShareLock  | t
    relation   |    16386 |    17071 |      |       |            |               |         |       |          | 72/5232            | 27587 | AccessShareLock  | t
    relation   |    16386 |    16920 |      |       |            |               |         |       |          | 72/5232            | 27587 | AccessShareLock  | t
    relation   |    16386 |    16906 |      |       |            |               |         |       |          | 37/7678            | 27728 | AccessShareLock  | t
    relation   |    16386 |    16932 |      |       |            |               |         |       |          | 18/0               | 27758 | AccessShareLock  | t
    relation   |    16386 |    16405 |      |       |            |               |         |       |          | 29/6478            | 27723 | AccessShareLock  | t
    relation   |    16386 |    17002 |      |       |            |               |         |       |          | 72/5232            | 27587 | AccessShareLock  | t
    relation   |    11564 |    10969 |      |       |            |               |         |       |          | 14/5931            | 27760 | AccessShareLock  | t
    relation   |    16386 |    17058 |      |       |            |               |         |       |          | 49/3355            | 27537 | AccessShareLock  | t
    virtualxid |          |          |      |       | 14/5931    |               |         |       |          | 14/5931            | 27760 | ExclusiveLock    | t
    relation   |    16386 |    16830 |      |       |            |               |         |       |          | 49/3355            | 27537 | AccessShareLock  | t
    relation   |    16386 |    16405 |      |       |            |               |         |       |          | 81/286             | 27749 | AccessShareLock  | t
    relation   |    16386 |    16932 |      |       |            |               |         |       |          | 50/3710            | 27759 | AccessShareLock  | t
    relation   |    16386 |    16902 |      |       |            |               |         |       |          | 72/5232            | 27587 | AccessShareLock  | t
    relation   |    16386 |    16440 |      |       |            |               |         |       |          | 1/3870204          | 27578 | AccessShareLock  | t
    relation   |    16386 |    17197 |      |       |            |               |         |       |          | 49/3355            | 27537 | AccessShareLock  | t
    relation   |    16386 |    17195 |      |       |            |               |         |       |          | 49/3355            | 27537 | AccessShareLock  | t
    relation   |    16386 |    16918 |      |       |            |               |         |       |          | 81/286             | 27749 | AccessShareLock  | t
    relation   |    16386 |    17063 |      |       |            |               |         |       |          | 55/0               | 27748 | AccessShareLock  | t
    virtualxid |          |          |      |       | 1/3870204  |               |         |       |          | 1/3870204          | 27578 | ExclusiveLock    | t
    virtualxid |          |          |      |       | 50/3710    |               |         |       |          | 50/3710            | 27759 | ExclusiveLock    | t
    relation   |    16386 |    16405 |      |       |            |               |         |       |          | 96/1390            | 27745 | AccessShareLock  | t
    relation   |    16386 |    16469 |      |       |            |               |         |       |          | 40/1870            | 27674 | AccessShareLock  | t
    relation   |    16386 |    17067 |      |       |            |               |         |       |          | 72/5232            | 27587 | AccessShareLock  | t
    relation   |    16386 |    16452 |      |       |            |               |         |       |          | 95/2961            | 27663 | AccessShareLock  | t
    relation   |    16386 |    17028 |      |       |            |               |         |       |          | 49/3355            | 27537 | AccessShareLock  | t
    relation   |    16386 |    16904 |      |       |            |               |         |       |          | 15/0               | 27439 | AccessShareLock  | t
    virtualxid |          |          |      |       | 40/1870    |               |         |       |          | 40/1870            | 27674 | ExclusiveLock    | t
    relation   |    16386 |    17076 |      |       |            |               |         |       |          | 20/4820            | 27725 | AccessShareLock  | t
    relation   |    16386 |    16916 |      |       |            |               |         |       |          | 72/5232            | 27587 | AccessShareLock  | t
    relation   |    16386 |    16452 |      |       |            |               |         |       |          | 12/14404           | 27727 | AccessShareLock  | t
    relation   |    16386 |    17058 |      |       |            |               |         |       |          | 40/1870            | 27674 | AccessShareLock  | t
    relation   |    16386 |    17055 |      |       |            |               |         |       |          | 49/3355            | 27537 | AccessShareLock  | t
    relation   |    16386 |   118405 |      |       |            |               |         |       |          | 72/5232            | 27587 | AccessShareLock  | t
    (133 rows)
    

Dan, I've added your suggestions to the cron job, so the next time we are over about 90 connections we'll have that data too.

[nnhubbard]

Chris or Dan? And ideas on what could be going on here?

[Byron_Claiborne]

Further information:

 

We have been getting a great deal of these 'relation' entries with AccessShareLock and ExclusiveLock on them.

 

------LOCKS-------

could not change directory to "/root"

   locktype    | database | relation | page | tuple | virtualxid | transactionid | classid | objid | objsubid | virtualtransaction |  pid  |       mode       | granted

---------------+----------+----------+------+-------+------------+---------------+---------+-------+----------+--------------------+-------+------------------+---------

 relation      |    16386 |    20873 |      |       |            |               |         |       |          | 39/242             | 18081 | AccessShareLock  | t

 relation      |    16386 |    20881 |      |       |            |               |         |       |          | 27/242             | 18040 | AccessShareLock  | t

 relation      |    16386 |    16450 |      |       |            |               |         |       |          | 26/254             | 18039 | AccessShareLock  | t

 relation      |    16386 |    16506 |      |       |            |               |         |       |          | 26/254             | 18039 | AccessShareLock  | t

 relation      |    16386 |    16403 |      |       |            |               |         |       |          | 27/242             | 18040 | AccessShareLock  | t

 relation      |    16386 |    20719 |      |       |            |               |         |       |          | 48/242             | 18102 | AccessShareLock  | t

 relation      |    16386 |    57374 |      |       |            |               |         |       |          | 9/4126             | 17982 | AccessShareLock  | t

 virtualxid    |          |          |      |       | 39/242     |               |         |       |          | 39/242             | 18081 | ExclusiveLock    | t

 

from PS AUXF we get a large number of...

 

------PS AUXF-------

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND

www-data 17791  0.1  1.1 235164 34944 ?        S    11:25   0:00  \_ /usr/sbin/apache2 -k start

www-data 17802  0.0  0.8 225648 25988 ?        R    11:25   0:00  \_ /usr/sbin/apache2 -k start

www-data 17839  0.1  0.9 229508 30072 ?        R    11:26   0:01  \_ /usr/sbin/apache2 -k start

www-data 17843  0.1  0.6 218480 18852 ?        S    11:26   0:00  \_ /usr/sbin/apache2 -k start

www-data 17911  0.0  1.2 240220 39712 ?        S    11:27   0:00  \_ /usr/sbin/apache2 -k start

www-data 17924  0.0  0.8 225280 24760 ?        S    11:27   0:00  \_ /usr/sbin/apache2 -k start

 

-- AND --

 

postgres 18005  0.0  0.4 103096 13020 ?        Ss   11:28   0:00  \_ postgres: matrix_web mysource_matrix [local] idle                                                                        

postgres 18015  0.0  0.1 101740  3900 ?        Ss   11:28   0:00  \_ postgres: matrix_web mysource_matrix [local] idle                                                                          

postgres 18018  0.0  0.1 101740  3896 ?        Ss   11:28   0:00  \_ postgres: matrix_web mysource_matrix [local] idle                                                                          

postgres 18034  0.0  0.1 101740  3896 ?        Ss   11:28   0:00  \_ postgres: matrix_web mysource_matrix [local] idle                                                                          

postgres 18037  0.0  0.4 103348 14040 ?        Rs   11:28   0:00  \_ postgres: matrix_web mysource_matrix [local] idle                                                                          

 

My first guess was backups locking the DB but it happens at random times and NOT near our scheduled DB backup.

[Chris_Smith]

Further information:

 

We have been getting a great deal of these 'relation' entries with AccessShareLock and ExclusiveLock on them.

 

------LOCKS-------

could not change directory to "/root"

   locktype    | database | relation | page | tuple | virtualxid | transactionid | classid | objid | objsubid | virtualtransaction |  pid  |       mode       | granted

---------------+----------+----------+------+-------+------------+---------------+---------+-------+----------+--------------------+-------+------------------+---------

 relation      |    16386 |    20873 |      |       |            |               |         |       |          | 39/242             | 18081 | AccessShareLock  | t

 relation      |    16386 |    20881 |      |       |            |               |         |       |          | 27/242             | 18040 | AccessShareLock  | t

 relation      |    16386 |    16450 |      |       |            |               |         |       |          | 26/254             | 18039 | AccessShareLock  | t

 relation      |    16386 |    16506 |      |       |            |               |         |       |          | 26/254             | 18039 | AccessShareLock  | t

 relation      |    16386 |    16403 |      |       |            |               |         |       |          | 27/242             | 18040 | AccessShareLock  | t

 relation      |    16386 |    20719 |      |       |            |               |         |       |          | 48/242             | 18102 | AccessShareLock  | t

 relation      |    16386 |    57374 |      |       |            |               |         |       |          | 9/4126             | 17982 | AccessShareLock  | t

 virtualxid    |          |          |      |       | 39/242     |               |         |       |          | 39/242             | 18081 | ExclusiveLock    | t

 

If you include the relation name you can see what things are competing for:

 

select pg_class.relname, pg_locks.mode, pg_locks.relation, pg_locks.pid     from pg_class, pg_locks
    where pg_class.relfilenode = pg_locks.relation
;
 

(works in postgres 8.4 at least).

 

Did you try Dan's suggestion of using memcache for sessions?

 

Does stracing any of those pid's show anything interesting?

Is autovacuum running?

Anything interesting in postgres or matrix logs?

[Byron_Claiborne]

 

If you include the relation name you can see what things are competing for:

 

select pg_class.relname, pg_locks.mode, pg_locks.relation, pg_locks.pid     from pg_class, pg_locks
    where pg_class.relfilenode = pg_locks.relation
;
 

(works in postgres 8.4 at least).

 

Did you try Dan's suggestion of using memcache for sessions?

 

Does stracing any of those pid's show anything interesting?

Is autovacuum running?

Anything interesting in postgres or matrix logs?

I am currently unable to get into psql to run queries. the role 'root' doesn't exist so i will have to figure out the role and password used.

 

From what I can see, memcache is not installed on the server.

It has been 3 days since the last issue so stracing shows nothing atm.

autovacuum does appear to be running.

 

postgres logs just before the issue:

 

         VALUES (($1), ($2), ($3), ($4), (TO_TIMESTAMP($5, 'YYYY-MM-DD HH24:MI:SS')), ($6))
2013-09-13 10:52:01 PDT ERROR:  current transaction is aborted, commands ignored until end of transaction block
2013-09-13 10:52:01 PDT STATEMENT:  DEALLOCATE pdo_stmt_0000004b
2013-09-13 10:52:01 PDT ERROR:  duplicate key value violates unique constraint "cache_pk"
2013-09-13 10:52:01 PDT DETAIL:  Key (cache_key, perm_key, url)=(5020276c0dc317ea39ef6f14e52d1348, e3fbb7821272919aaa2564e2e0485a19, FQDN/api/iphone/current-news.xml?v=1) already exists.
2013-09-13 10:52:01 PDT STATEMENT:  INSERT INTO sq_cache(cache_key, perm_key, url, assetid, expires, path)
        VALUES (($1), ($2), ($3), ($4), (TO_TIMESTAMP($5, 'YYYY-MM-DD HH24:MI:SS')), ($6))
2013-09-13 10:52:01 PDT ERROR:  current transaction is aborted, commands ignored until end of transaction block
2013-09-13 10:52:01 PDT STATEMENT:  DEALLOCATE pdo_stmt_0000005a
2013-09-13 10:52:05 PDT ERROR:  duplicate key value violates unique constraint "cache_pk"
2013-09-13 10:52:05 PDT DETAIL:  Key (cache_key, perm_key, url)=(b1b88cbd21b35d0d07f9f9655d503b19, e3fbb7821272919aaa2564e2e0485a19, FQDN/api/iphone/photos-list2.xml) already exists.
2013-09-13 10:52:05 PDT STATEMENT:  INSERT INTO sq_cache(cache_key, perm_key, url, assetid, expires, path)
        VALUES (($1), ($2), ($3), ($4), (TO_TIMESTAMP($5, 'YYYY-MM-DD HH24:MI:SS')), ($6))
2013-09-13 10:52:05 PDT ERROR:  current transaction is aborted, commands ignored until end of transaction block
2013-09-13 10:52:05 PDT STATEMENT:  DEALLOCATE pdo_stmt_00000069
2013-09-13 11:27:55 PDT WARNING:  pgstat wait timeout
2013-09-13 11:36:56 PDT FATAL:  remaining connection slots are reserved for non-replication superuser connections
2013-09-13 11:36:56 PDT FATAL:  remaining connection slots are reserved for non-replication superuser connections
2013-09-13 11:36:56 PDT FATAL:  remaining connection slots are reserved for non-replication superuser connections
2013-09-13 11:36:56 PDT FATAL:  remaining connection slots are reserved for non-replication superuser connections
2013-09-13 11:36:56 PDT FATAL:  remaining connection slots are reserved for non-replication superuser connections
2013-09-13 11:36:56 PDT FATAL:  remaining connection slots are reserved for non-replication superuser connections
2013-09-13 11:36:56 PDT FATAL:  remaining connection slots are reserved for non-replication superuser connections
2013-09-13 11:36:56 PDT FATAL:  sorry, too many clients already
2013-09-13 11:36:56 PDT FATAL:  sorry, too many clients already
2013-09-13 11:36:56 PDT FATAL:  sorry, too many clients already

I haven't seen the Matrix logs yet.
 

[Benjamin_Pearson]

Look at data/private/conf/db.inc for the database name and user, then use (as root): psql -U USERNAME [-h IPADDRESS] DATABASENAME (you only need -h if postgres is not running on a UNIX socket). Also you can use the default postgreSQL user as well (psql -U postgres DATABASENAME).

[nnhubbard]

Look at data/private/conf/db.inc for the database name and user, then use (as root): psql -U USERNAME [-h IPADDRESS] DATABASENAME (you only need -h if postgres is not running on a UNIX socket). Also you can use the default postgreSQL user as well (psql -U postgres DATABASENAME).

 

Thanks, I was able to provide him with the login info.

[Byron_Claiborne]

I was able to get into PostgreSQL and test the script written by Ryan last year. I think it is setup and ready to go when the issue recurs. I also looked back over the postgresql logs and the two errors from just before the crash were occurring for about an hour before hand. From the looks of them, they appear to be db caching errors. I will try the memcache change suggested and see if it happens again.

 

Thanks,

Byron