Did my dream come true?

Squiz Matrix General Discussion
1

[font=“Tahoma”][size=1]

[quote]determined

adjective

…

3. devoting full strength and concentrated attention to;

…

5. strongly motivated to succeed[/quote]

www.dictionary.com

[/font][/size]

[font=“Tahoma”][size=2]



I would like to start this topic to discuss advances in web applications design.

Lot's of things happening in this area now and it is very hard not to become lost -

  • many PHP frameworks evolved

    (PHP section)
  • JavaScript frameworks

    (PHP section)
  • Google Code

    (PHP section)

    etc.





    It takes some time before you start asking yourself questions - what we do, why we do it this way and what are the other ways to do it?



    First, let me to define 'web applications' as I understand them now. Web application (as at today() :slight_smile: ) - is a system which works via internet (browser-client –> web server - server) and carries complicated business logic.

    Internet Banking, Amazon and EBay interfaces as retail customers (buyers) see them are not considered as web applications by me.

    These systems (with functionality they provide) are perfectly fine delivered by browser - anybody can reach these services via any major browser. Some minor issues and browser inconveniences (as I call them) can be resolved by professional developers. They have relatively few data entry screens and most of the things they do - display us structured content. Browser evolved as a tool for browsing content.

    These days browser is used more and more as a delivery framework for complex intranet and internet applications. There is a lot of excitement about it. If you look at PHP section - you will see that they call themselves ‘proven leader in on-demand customer relationship management ‘. After I discovered some problems on my system I decided to have a look how they handle such situations. Guess what? They do NOT handle it at all.

    Want to know what was checked?



    1. Loaded account details to edit in two different browser windows. First user changed record and saved it. Second user did not change record and saved it after first record.

    Result - second user overwrote first user changes unintentionally

    Correct result - second user should get a message that record was changed after he loaded it for editing and provided interface to merge his changes into updated record (merge) if necessary.



    2. Quite common schema while working with data entry to forms is.

    (x) Data entry page -> (y) Confirmation page -> [(z) Details page]



    (x) Form - data entry has buttons ‘Save’ and ‘Preview’

    (y) Confirmation page - displays data as text (without form elements) to let user to check entered data. Has buttons ‘Back’ and ‘Save’

    (z) - display data which were inserted and extracted back from database. This step is sometimes omitted.



    There is a tendency - users spot a mistake at a moment they hit ‘Save’ button on Confirmation page. They hit ‘Back’ and do not realize that data were already added to database. Than they correct errors and trying to Save again.

    Result - Salesforce application went to indefinite loop. Data can not be saved but there is no any error message to a user similar to “Record already exists!”

    Correct result - after first submission all events from same page should be blocked by JavaScript to eliminate any activity from this page.



    Nobody is perfect. Just proves how hard sometimes to code business apps in browser and it is under looked even by big companies.



    Look at a picture below. This is well known YahooMail. If you open any mail in Send folder (shown) you can see HREF ‘Add to Address Book’ next to …. From email address. Does it make sense? Won’t it be your own email address and does it make sense to add yourself to address book?



    [center][/center]

    [color=“#483D8B”][center]Fig 1 Natural position for 'Add to Address Book'[/center]



    What were major driving factors of using web for developing enterprise applications? Most common ones I have heard of are rapid development and easy deployment. It appeals as a perfect tool to replace company’s shared Excel files and Access databases - merging data to enterprise data storages away from complexities of managing many single files and databases.

    But, as experienced developers, we know that there is no way to get Yin without Yang.

    On a Yang side we have most of the logic located on server side and with super processor and Gb’s of memory we have to wait server response, isolation from OS - no access to local file system, Forward/Back buttons - browser navigation conflicts with application logic, differences in browser functionality in a combination OS-Browser, no way to work offline to at least preserve data while network is not available.

    May be for future projects I will reconsider technology choice - at the end of the day we have Eclipse and PHP section and I am sure you know many more successful cross platform applications. I agree that Java is not perfect but in comparison with .Net it gives a feel of freedom which any developer likes :).



    In my situation we already wrote hundreds of thousands of lines in PHP. To overcome browser inconveniences I started thinking of putting a browser inside a box which will allow developing in HTML and JavaScript on one side and will allow removing browser navigation and intercepting events before browser will see them. I found .Net WebControl and I thought I can sacrifice support OS-Browser in favor to .Net and IE. It is like you will say that Matrix backend supported on .Net and IE (Windows) only. I develop corporate applications and I do not consider it as a big disadvantage as majority of corporate desktops have unified setups read Windows/IE combination.



    I found simple VBasic application which uses Microsoft WebControl and asked my developers to remove from it all navigation and leave only frame (I called it IENude project) and add functionality to read XML file with URL of web application server and web application name.

    When we launch it we will see only web page ina frame. This frame will have our application name instead of Form1 shown on a screenshot.

    I am sorry for typos inside screenshot - was typing it at 2 am.



    [center][/center]

    [color=“#483D8B”][center]Fig 2 Night wish[/center][/color]



    I though finally I can get rid of ugly URL’s IE7 shows on each and every page, can ping web server and tell customer speed and network availability from my IENude, I can now replace ‘Page can not be displayed’ with nice interface to allow user to save typed data locally and provide interface to resubmit later. I have no ‘Back’ and ‘Forward’ buttons, I can easily intercept double clicks or ability to click many buttons on slow networks (problem #2 described above). I can easily filter events to browser and it looks like such filtering process will NOT be business specific. It is all will make it IEImproved for me. I even can ping clients from server to find out who is still online, I can send messages to them and how happy I am!



    [center][/center]

    [color=“#483D8B”][center]Fig 3 Night wish implementation with Matrix login inside - AIR will be discovered shortly after that[/center][/color]



    This week I came across to … Adobe AIR and it reminded me something. They did more than I wanted but nobody forces me to use more than I need (again my finger will not be itching while I type ‘.Net’ any more :slight_smile: ) I was a bit skeptical about their push for Flash and Flex before for web application development. I even subscribed for 2 weeks trial of PHP section site which is fully designed using Flash technology and I did not like it. It reminded me times when people used Java applets inside web pages.

    As to me these are perfect tools for promotional sites and companies which business is in a design and media/entertainment.



    Could it be this time ADOBE get it right?

    If you look at PHP section (scroll down to Adobe AIR features) won’t it remind you my IEImproved.

    They promise it to be lightweight and cross platform. It uses PHP section as HTML engine. Good thing is - no more OS-browser tests. All logic can still be on a server - we can use AIR browser extension to render HTML and JavaScript to keep AIR application updates to a very minimum.

    BTW: application itself can detect if a newer version is available and update itself on startup.

    It is still in Beta but potentially I see it as a tool to AIR browser Yang sides and replace my IEImproved.



    Would really like to hear what you think. Squiz developers and customers all welcome to discuss or contribute here.



    Thank you,



    Igor

    [/size][/font]
2

My thinking is a little different. Those sites do contain non-trivial business logic and programming, and have to scale well beyond the average site.



If you took even basic e-commerce sites out of the web browser and into a traditional application environment, you would call them applications as they are not trivial. The web, and the explosion of Content Management Systems and frameworks, has allowed anyone to create a web application. Even those that didn't require any programming to get started are based on code written by someone. Because of that, I see no real difference between running an e-commerce site and running a point-of-sale desktop application that you didn't develop yourself. They run in different environments but are both equally qualified to call themselves "applications".


[quote]After I discovered some problems on my system I decided to have a look how they handle such situations. Guess what? They do NOT handle it at all. [snip] Nobody is perfect. Just proves how hard sometimes to code business apps in browser and it is under looked even by big companies.[/quote]

To me, this is another reason why even simple sites that interact with users should be classified as web applications. The web has provided a lot of challenges for developers. I'd say that the two you've pointed out here are pretty common. The fact that developers need to take these things into consideration makes the sites they write non-trivial. It's a different world for application developers, with a whole range of new problems to solve.


[quote]What were major driving factors of using web for developing enterprise applications? Most common ones I have heard of are rapid development and easy deployment.[/quote]

I'd add "the ability to access an application from any internet-enabled computer" to that list. Certainly, web-mail applications like Gmail and YahooMail are popular because you have access to all your mail whenever you connect to the web application. You don't have to worry about not having access to emails you sent/received at work from your home computer.


[quote]I though finally I can get rid of ugly URL’s IE7 shows on each and every page, can ping web server and tell customer speed and network availability from my IENude, I can now replace ‘Page can not be displayed’ with nice interface to allow user to save typed data locally and provide interface to resubmit later. I have no ‘Back’ and ‘Forward’ buttons, I can easily intercept double clicks or ability to click many buttons on slow networks (problem #2 described above). I can easily filter events to browser and it looks like such filtering process will NOT be business specific. It is all will make it IEImproved for me. I even can ping clients from server to find out who is still online, I can send messages to them and how happy I am![/quote]

Again, there is no way to get Yin without Yang. By giving your users a new application to use to access your web application, they are losing the ability to use your web application from any internet-enabled computer. You are also changing the user experience that they are used to. They will still know they are accessing a web application, but they wont have the familiar tools available to them, including any browser plugins they might use.


[quote]This week I came across to … Adobe AIR and it reminded me something. Could it be this time ADOBE get it right?[/quote]

Maybe, or are we just adding yet another piece of complexity for web developers to deal with?



If I was to use AIR, I would use it to create an alternate interface to my web application. I would make the interface different (both functionality and process) to match the expectations users have of desktop applications. Now, I can add all that great local machine integration that the browser doesn't give me, and I don't have to stray to far from my existing web development skills. However, I do need to start thinking differently; like a desktop application developer. I also need to maintain two versions of my interface so I get the benefits of both browser-based and desktop-based web applications.



A good example of this type of approach is eBay Desktop.



AIR has it's place, but for some web applications, the developers are going to be under even greater pressure to get things right. For some companies, they will have the resources to manage the additional complexity. For others, they may end up bitting off more than they can chew and making a greater number of mistakes; not just the ones you mentioned previously for Salesforce.



Don't get me wrong though; AIR is an exciting platform that I'll be keeping my eye on.

3

[font=“Tahoma”][size=2]
We might look at it from slightly different corners.

In your business (there are no doubts) web content produced by CMS has to work properly in ANY major web browser. It is a challenging task by itself (very challenging).

In my previous post I looked at Matrix (just for clarity) but not at content Matrix produces and tried to generalise with what we (within my company) do.



Who are Matrix users (please correct me if I am wrong):

– administrators who look at site or its pieces as a single entity. They allow editors to access certain areas to change/create content.

– content maintainers – people who actually fulfil site with content.



Lets think only about these two groups of people. They are Matrix users and you have to support Matrix for all major OS-BROWSER combo to allow them to access Matrix from any internet connected computer. Is it important for them? Most of the people have computer at work and at home plus may be laptop. Is it a real requirement (advantage) for such users to go to internet access point or somebody else’s computer to fulfil their duties?

I know that this is not a requirement for my system users (corporate intranets). I am not advertising AIR here. It is only in beta now – too early to make political decisions about its use. For corporate applications such lightweight frame simplifies development and significantly improves application quality. On top of removing a head ache of supporting all OS-browser (which is time to market for developers) it brings a lot of things which are not available or hard to implement in a browser and still be HTML/CSS/JavaScript based.



For example, I never ever type such long forum posts or emails inside browser (compose message/post screen) just because I know I can accidentally close it or new window will be opened on top of my text or network connection might disappear after submit. So it is always a MS Word for me with 1 min auto save. So yes, internet forums are accessible from any internet connected computer and are a great communication tool but I personally need a desktop editor if my post is bigger than one paragraph. To be truly supportive in this area application has to have access to local file system.



I am not trying to convince you - just to explain the way I think as well as show how I look via technology filter on my business requirements which could be different from yours.

[/size]

[/font]

4

I can't talk for all our clients, but I certainly find it an advantage to be able to update the Matrix and Squiz websites no matter where I am. It is certainly something that is important for a percentage of web application users, which means it is not important for another percentage. I was just making a point about the other advantages of web apps.


[quote]I am not trying to convince you - just to explain the way I think as well as show how I look via technology filter on my business requirements which could be different from yours.[/quote]

You might have misunderstood me. I think that custom interfaces for web apps, like the one you produced or those produced using AIR, have their place. I was just making the point that there are other things to consider also.

5

As I said I see AIR as a browser engine inside a specialised frame which gives me some more functionality and simplifies/speeds development. You are right - at a price of 9M install and inability to look at it via browser. If ADOBE will make it right I know my customers will accept it.


One can design AIR application and have some HTML/JavaScript files to be a part of application (stored locally) and retrieve only data from server but I am not going to use it this way. Well - may be I will include JavaScript files with mechanism to refresh them as they change on server.



I will definitely try this technology to see how problems I described above and others will be handled by AIR. If you like I can post screenshots or even swf's as at this stage all my writings based on impressions only. Promises discipline me :slight_smile:



BTW.

I know you are working on MS SQL support for Matrix. Do you use FK's in database? Didn't you hit a following situation?

_______________________________________________

table users (PK user_id, …)

table log_table (PK log_id, …, created_by, …, modified_by, …)



MS SQL will not allow to create two FK's from log_table.created_by --> users.user_id AND from log_table.modified_by --> users.user_id, assuming you want typical ON DELETE RESTRICT ON UPDATE CASCADE. I was shocked when I found it when we moved our application from PostgreSQL to MS SQL. MS support says it is by design. I still do not see danger they saw.



Error on [log_table] save will look like:


[quote]'users' table saved successfully

'log_table' table

  • Unable to create relationship 'FK_log_table_users1'.

    ODBC error: [Microsoft][ODBC SQL Server Driver][SQL Server]Introducing FOREIGN KEY constraint 'FK_log_table_users1'

    on table 'log_table' may cause cycles or multiple cascade paths.

    Specify ON DELETE NO ACTION or ON UPDATE NO ACTION, or modify other FOREIGN KEY constraints.

    [Microsoft][ODBC SQL Server Driver][SQL Server]Could not create constraint. See previous errors.[/quote]

    ______________________________________________

    Solution was found by using triggers instead of FK.

    We basically created a table in database (like a register) where we store all FK relationships. We also have a PHP script which uses this table and generates a SQL script which regenerates all FK (triggers) on our system. Only later we realised that we not only can control FK implementation with single PHP script but also have 'like FK' on columns which allow NULL values. In my example above [modified_by] column can not be validated with server FK if it alows NULL values. If we generate triggers from PHP script we can easily allow change from NULL to NOT NULL and validate that value to be a valid users.user_id and allow changing back from not null to null.



    If you want to look how it works I can publish/pass PHP script and table.
6

I'd love to see what you come up with.


[quote]I know you are working on MS SQL support for Matrix. Do you use FK's in database? Didn't you hit a following situation?[/quote]

We use MS SQL for MySource4 only. We use so many DBs for My4 that we come across all sorts of DB problems :). Instead of using foreign keys, we build the relationship logic into DB triggers and PHP code.

7

The first thing I want to try is to handle timeouts and network disappearance as well as all errors like HTTP 4XX and 5XX.
Idea is to intercept it, not to show it and offer user friendly way to continue - wait, save, offer to notify by email when service will become available or by the other words something very very cool.

[quote]We use MS SQL for MySource4 only. We use so many DBs for My4 that we come across all sorts of DB problems :). Instead of using foreign keys, we build the relationship logic into DB triggers and PHP code.[/quote]

So you use similar approach - very pleasant to hear. If I will be pushed to support many DB's for my system I will just supply a switch DB_VENDOR to PHP script and let it build triggers for particular supported database.
Later in our development carears :) we stopped deleting items from database but marking rows with deleted_by and deleted_tstamp.
As nothing deleted physically FK's will be no help if used. With triggers approach we control triggers creation database wide and we came up with a beautiful solution to check if deleted_by can be set to not NULL.

Let's say I have following constraint recorded in my system:

[center][img]http://members.optusnet.com.au/~igor_kryltsov/table.PNG[/img][/center]
Let's also assume that we missed validation in a code and PHP wants to delete record in [hr_leave_reason] which is still referenced in [hr_exit].
We can easility now modify our triggers generator to look in [hr_exit] table just before [deleted_by] is going to be changed from NULL to NOT NULL. Basically we use ON UPDATE trigger to double validate our way of deletion (setting deleted_by rather than executing DELETE).

If to look even further (we haven't touched it yet) and allow to keep track of all versions of a particular record - again such unified approach allows to register data in a table (similar to one shown above) and use PHP to generate database specific SQL implementation.
8

[quote]Later in our development carears :slight_smile: we stopped deleting items from database but marking rows with deleted_by and deleted_tstamp.
As nothing deleted physically FK's will be no help if used. With triggers approach we control triggers creation database wide and we came up with a beautiful solution to check if deleted_by can be set to not NULL.[/quote]

How do you clean the deleted records? If you do not clean the records then the database size will get bigger and bigger. Isn't that a problem?

9

Thank you Danis for joining us :slight_smile:


In my system it is not a problem (live DB is 300M after being live for 5 years) but because we store deleted_by in a register table to produce triggers:

  • we can provide web interface for sysadmin to set a time period to store deleted records before purging them out and let system do it based on admin setup
  • note as we record all deleted_by columns with their respective tables in register table, shown in my post above, we can even look via tables and tell how many deleted records database wide or table by table.
  • we can even (subject of customer requirements) allow admins to browse them with browser
  • restoration could be quite challenging as there are data dependencies



    Admins (I evolved from sysadmin area) like to be able to tell management who deleted those important records.



    Also in my system for simple lists (simple table with values used later in drop downs) if you will delete record and than another user will create it (exactly the same one) - system is actually smart enough to remove deleted_by and modified_by with corresponding timestamps and will update created_by and created_tstamp - sort of self cleaning. Obviously we do not program it every time - part of our framework/standards.



    Hope this answers your question.
10

Inline Editing


I know you are going to implement it.

Just hit SalesForce Inline Editing Implementation link which might give you some hints from people who can afford to do usability testing :wink:

11

[quote]I know you are going to implement it.
Just hit SalesForce Inline Editing Implementation link which might give you some hints from people who can afford to do usability testing ;)[/quote]



Without giving too much away, our inline editing for MySource 4.0 is better than that. :slight_smile:

12

To Avi:
It was not my intention to point you to gurus.

I know you are very smart bunch of people (no irony here).

Just sometimes some minor interface solution(s) can be grabbed and used for a happy ever after :slight_smile:

13

Oh, sure. I was just gloating a bit too. :)
14

Tracking page load by client


I guess you are adding Google Analytics code injection automation in a new version of Matrix - looks as pretty straightforward feature for CMS if not to track all page events.

Google has not implemented Real client load time measurement functionality yet as far as I can see.

Looks like very handy thing as you can report for pages which were never fully loaded as well as compare server side statistic with client side statistic.

Imagine - 2 lines graph.

X - each web site page, Y - time. One line represents time server spends to generate page, second - average time same page loaded to client.

Another (third line) may represent hit count for each page. Lot’s of information to absorb for web analyst.

What if server spends lots of time but than it takes a fraction to load to client or backwards?

If represent it as stacked bars - problems will be represented by most tall bars and priority by hit count line (basically we fix first most popular pages in trouble).



Another very important thing in measuring client load time (could be enbled for a limited time only as it will degrade performance slightly) you can calculate (client_time - server_time) latency of customers. Not sure if this figure will represent something meaningful but could give you an idea how speedy are your customer’s networks.



I am looking for it for a long time but never had time to implement.



Does it make any sense?

15

Before ‘diving into AIR’ test we decided to look (I should say way too late) how our system handles HTTP 4XX and 5XX when this happens just before fancy AJAX ‘Loading…’ .
Funny to say - we are running PHP on IIS 5 and it is a first time when we actually wanted to get HTTP 500 (Internal Server Error). I think vendor will patent HTTP 500 as IIS trademark one day.



Very interesting to see what users will see if not to take care of it.



At least now we know what can and can not be done in a browser and how this should be done for what it can be done.



B.A. (Before Ajax era) it was easy - user gets ‘Page can not be displayed’, hits Back and it is nothing to do with developers. A.A. (Anno Ajax era) user can see a page and has no clue what is HTTP and all mantra behind it.

16

Here is Stage 1 for my new developer and TAF© - ThinAirFramework.


This is a way we work - no time spent in front of computer (PPS technology - Paper, Pencil, Scanner).

I do not advertise technology but when I look at bottom of my web server on a picture (shown below) - I remember what I was thinking about when I was doodling it :). Power Point does not have such feature yet.



The only time we do it nice way (HTML pages, Visio diagrams, nicely formatted MS Word) is on customers sign off for their business logic.



[center][/center]

[color=“#000080”][center]TAF Stage 1[/center]



For those who will not be able to translate it I will briefly describe what it is all about.



We use simple registration page - username, first/last name.



We use two PHP scripts:

  • registration.php – displays and client side validates user input
  • registration_handler.php – checks that user name is unique and saves data in table [users] than it redirects back to registration.php or displays server side validation error and redirects again to registration.php



    Our TAF goal is:
  • look at TAF possibilities to handle HTTP 4XX and 5XX etc
  • look at TAF possibilities to store and retrieve form data locally (user initiated - just wants to save unfinished work or problem initiated - server not available)



    On Stage 1 we do not save anything yet. We only start application and trying to load data entry page.

    We probe all possibilities and different handling techniques.

    We utilise HTML and JavaScript within AIR (not loaded from server but part of TAF) to handle above.



    Keep watching - source code and screenshots will be provided.
17

What can stop us to have some fun while we are waiting for first results on TAF?


I have been working on my LinkedIn profile (want to thank Greg again to show this site to me).

I was in a good mood and I delivered this entry to cover period before I popped up in Australia.



[center][/center]



This was smart and funny and I was proud to do it using my second language but …

it was even more fun when I saw this…



[center][/center]

Computers do not have an intellect but they definitely have a sense of humor :slight_smile:

18

That made me laugh :slight_smile:

19

Things you want to check but have no time to do


My development center is based in one of Technical Universities and we have strong relationships (my Alma mater).



We are lucky enough to have an opportunity to specify subjects for student projects every six months.

Student’s specialty is "Software design and development" (Masters in 4th and 5th year of studies, some already work part time in software industry)

They learn real life tasks and we experiment with stuff we have never had time to do.

Bonus is - our tasks will be passed to the best students.



Please consider it as an open invitation post YOUR things (don’t be shy)

The only condition - task has to be well defined and not too complicated. It is not a diploma project it is semester project for students.

Your description and solution offered will be posted here after completion.



[center]This offer was published 8 Nov 2007 and closed 20 Nov 2007[/center]

20

Back to connectivity problems handling between browser and server


Greg,



Didn’t you think to add functionality to save form data in a cookie if connectivity to server was lost? Something similar to this approach.

Imagine - user works on a form and there is 1 min (for example) AJAX background process which sends all entered form data to server (auto save).

Than network disappears and JavaScript notifies user and saves form data in a cookie on a client.

At this stage you have 1 server side snapshot and 1 client side snapshot of data.

Obviously on top of actual data (element-value) we can save date/time and what page/form it is.

Web page has an ability once loaded to browser to look inside cookies to check for unsaved data.



More complicated but smarter way is to actually track changes since last auto save. If there were no such changes - last auto save inside session will be current. This tells us that there is no need to save anything in a cookie if connectivity was lost.



Another thing to think about - session is not a perfect way to store form auto save data. I mean it is but it has to be some process which periodically moves such data from sessions to database.

Successful submission will clean both locations.



Do you consider this as an important thing to worry about (implement) for your product?