Archive for July, 2005

Tag Filters (Update on Query Problems)

Tuesday, July 26th, 2005

From the couple responses I got to my posting about Query Problems it looks like there is no great solution to tag filtering. It does look like using a subquery for each tag you want to filter by works reasonable well up to at least hundreds of thousands of tags, so that approach should work just fine for me.

The actual filtering is in my AJAX resources, and is currently just on there attributes (language they are written in, etc) not resources other tags. I think to really make the filtering work nice i’ll use some sort of multi link approach were you can add the filter as an or link, as an and, or view view that tag. I’ve also been thinking about adding some AJAX to the page just to show it off, but I’m not sure its really helpful to usability besides possible faster loads times, I’ll have to think about it a little more.

An example of why the filtering was added is now you can see all the PHP AJAX libraries, where before all the Ruby or Perl ones would be mixed in.

AZPhp Tomorrow

Monday, July 25th, 2005

Phoenix’s PHP User’s group is having its monthly meeting tomorrow. Its at the usual time and place, Walt’s TV at 7:00pm. The presentation was going to be “Connecting PHP and Java through a bridge” but it seems that the presenter is going to be in Colorado which messes up that plan a bit.

But being a well organized bunch Eric and I have a backup plan, tomorrows presentation will be replaced by a round table on:

Neat PHP Hacks/tricks

It would be great if everyone who comes to the meeting could bring some exciting PHP code they’ve written to show off. I’ll be showing off PHP Flash app I’ve been working on (not all that neat, but definitely a hack) as well as some AJAX stuff.

Query Problems

Monday, July 25th, 2005

I’ve been attempting to add some filters to my AJAX resource list. I’ve made an attempt an added them, but they don’t work the way I would like. So lets say your on the library list, a filter that would make a lot of sense, would be to say show me every library thats written in PHP and Provides AJAX. Now if you goto that link you’ll get every library thats written in PHP or Provides AJAX. Now I think the UI worked out pretty well but the or thing stinks, and that all comes down to database queries.

So the database is 3 tables:

Bookmarks
  bookmark_id
  user_id
  url
  ...

bookmark_tag
  bookmark_id
  tag_id

tag
  tag_id
  user_id
  tag

So I end up with a that is basically:

select b.*
  from
bookmark b
  inner join bookmark_tag using(bookmark_id)
  inner join tag using(tag_id)
where
  tag.tag = 'library' and
  b.bookmark_id in(
    select bookmark_id from bookmark_tag inner join tag using(tag_id) where tag.tag in ('language:php','provides:ajax')
  )

This works great but I export for multiple filter tags means or, since they are in an in list, or subquery for filter which seems like it would be horrible for performance.

Am I just missing a simple solution to the problem or is a subquery for each filter the only option?

Btw: I’m using mysql 4.1.something

AJAX Resource Updates

Thursday, July 21st, 2005

I finally started clearing out my incoming bookmarks folder, still a long ways to go but a lot of whats left are reference links i’ll need when I do the writeup about my embedded mozilla project (thing pdf generation).

Anyhow here is the summary of todays updates.

library

Ajax.NET - The free library for .NET (C#)
Full Featured C# ajax lib
DWR - Direct Web Remoting
AJAX library for Java
Accesskey Underlining Library
Easy way to add access keys to any html page without changing the actual html
script.aculo.us - web 2.0 javascript
Effect and Drag n Drop library built on top of prototype
SACK - Simple AJAX Code Kit
JavaScript library for performing AJAX
youngpup.net - DOM-Drag
Lightweight JS Drag n Drop lib

article

adaptive path » ajax: a new approach to web applications
Original article that Created the AJAX name
The Man in Blue > This is not another XMLHttpRequest article
Article discussion the decision of JavaScript for “the extras” or for primary features, focuses on Gmail and Google maps
Using the XML HTTP Request object
Article covering the ins and outs of using XMLHttpRequest on various browsers, various requests such as GET POST HEAD are covered

oss

Monket Calendar
GPL AJAX online Calendar written in PHP and JavaScript

Linux Developers don’t get it

Saturday, July 16th, 2005

Since the begining of the project i’ve been subscribed to the fedora devel list. I read it too see whats in the new releases and to make sure I know all the tricks when im upgrading my server. I don’t post often but its a pretty good way to get an idea of whats happening in my distro of choice.

Over the last week a thread has been going on that has me convinced I know the reason Linux isn’t ready for the desktop. Many people just don’t get what developing an operating system is about, I’m not even sure they get what using a computer is a about. And even if they aren’t a majority they slow down the approach to sanity so much thats its taken Gnome since 1997 to get to windows 95 levels of usability (and hardware integration isn’t even to that level yet).

To sum up the thread, Gnome is removing the open terminal item from the right click menu of Nautilus, and a number of people think thats such a big deal that Fedora should make an exception from its policy of using upstream code whenever possible.

The reasons they give are odd in a number of ways.
The terminal is the most important part of unix (if your a graybeard maybe)
The right click menu is the easiest way to open it (kinda odd since a terminal junky would always say a keyboard shortcut is better)
The terminal should be used by everyone since its so much more efficient then a gui
That having to use a terminal to get something done isn’t a bug

Now whats also odd, is that there is already a package to give you this open terminal functionality back if your really want it only its better since if your in a window in Nautilus it open in that directory.

So whats my problem with these arguments.
Wake up, Fedora and Gnome are about being cutting edge, how is pushing the terminal as part of normal usage part of that.
The pointless rightclick argument, its already in the menu and you can drag it to the toolbar for a launcher
The terminal being more effiecient might be true, but thats only after you’ve managed the learning curve.
To change to a directory in nautilus move a file all I have to do is point and click, then drag and drop, its a pretty basic skill that everyone knows and since it has good visual feedback its reasonable to figure out.
To do the same thing in a terminal I have know the following cryptic commands (they are meaningless just because you and I have long memorized them doesn’t mean there not)
cd, ls, mv

I’m not sure why someone who uses the computer for surfing the net, checking email, and writing papers would want to learn these commands, but a very vocal group of people on fedora devel thing were doing them a disservice by not making them memorize them. Many even seem to think everyone should learn bash scripting and awk and crap I don’t even have a need to learn in order to be good computer users.

The reality is most people don’t want to become computer geeks.
A computer is a tool, just like there toaster is a tool.
It should work well, most of the time, and not be too difficult to figure out.

Thats what people want out of computers and thats why they use Windows or Mac OSX, neither of these are perfect but at least there developers have the right idea.

HTML_AJAX

Friday, July 15th, 2005

If your on the pear-dev mailing list you should have already seen the post but i’ll blog about it here as well hoping to get some feedback.

A proposal was made on pepr for an HTML_AJAX library in PEAR. I was less then happy with the implementation and since once a package is in PEAR its hard to replace I thought i’d step up and provide a package that I think is ready for widespread use. I wasn’t really considering doing this before this came up, since its a lot of work, and JPSpan has been working fine for me (thats not to say I wouldn’t do some things differently). So here is an alpha release of my HTML_AJAX library.

Anyhow im actively looking for someone to come aboard as a co-maintainer of this package, if your interested let me know.

Its goal is to over OO javascript proxy and proxyless operation. Sync and Async AJAX calls, with an optional JS library offering things like a content replacement api. It should be usable in a server style setup (JPSpan) or directly in a page (Sajax). Its curently doing all communication using JSON, but that might change if someone sees a good reason.

The current feature set is:

  • Ability to register multiple classes on one page
  • JSON is used for communication in both directions (I made add JPSpan xml from JavaScript to server as well)
  • Class are exposed to Javascript as classes
  • POST is used for sending requests
  • Decent JavaScript error handling (right now there is an alert catch on the errors, but we might just leave the exceptions alone at some point)

Next Steps are:
In the Javascript code:

  • Decide if JavaScript JSON implementation has a license that is ok (json.js)
  • At least switch to a smushed json.js (3k instead of 10k)
  • Clean up HTML_AJAX.js, I pull in a bunch of classes from JPSpan and I think they can be made smaller through refactoring
  • Expose onLoad etc events
  • Complete Async Callback support
  • Create an addon api that allows for, proxyless operation, direct content replacement from results

In the PHP Code:

  • Pearify JSON.php (this currently has a pepr proposal in the draft state, the author said he will try to move it forward, but who knows if he will have time)
  • Complete the api so it that is supports inpage operation like test.php, or out of pager operation (including the javascript proxy files)
  • Write code to deliver the js files through php with the proper caching headers, so you don’t have to copy the js files around unless you want too
  • Write a cache for javascript proxy files (I did this for JPSpan today so it should be too hard)

Improving JPSpan page load performance

Friday, July 15th, 2005

If you use JPSpan to augment a normal website you may have noticed that your page loads now take quite a bit longer. The reason for this is loading the JPSpan client javascript code, its generated on each load and so it can’t be cached on the client. This file can become huge if you register lots of classes with lots of methods to create stubs for.

So the solution is to allow this page to be cached, to allow for that, im using HTTP_Cache, i’ll also be caching the output generated by PHP since the client files shouldn’t change often.

When you start caching the most important question to answer is, “How do I know when I need to regenerate my cache”. In the JPSpan case two things can cause a cache regen, the server url changing (the url of the JPSpan server is embedded in the file) or the api your exporting changing. The server url changing might not ever happen for some people, but in my usage the ip is different depending on if your using the VPN or the public url so this needs to be taken into account. Its also good to take into account so that a stale cache file from development doesn’t ruin things.
Now the api is a little harder, but since JPSpan has to know what to export, its something that must be available.

In fact its in the descriptions array on the JPSpan_Server_PostOffice Object. So to know if its change you just make a hash of it, and check for a change in that. Code to make our API hash is shown below.

<?php

// create a hash from the api of the handlers
    // turn the descriptions into a string
    $api = “”;
    foreach($S->descriptions as $key => $val)
        $api .= $key;
        foreach($val->methods as $method) {
            $api .= $method;
        }
    }
    $apihash = md5($api);

?>

Now that we have the information we need on when to regenerate our cache its just a matter of implementing it. First will implement the client side cache, to do this we Use HTTP_Cache to send 304 codes when nothing is changed. The code to do this is shown below, notice that were not compressing the generated JavaScript yet (strip whitespace comments) since its a really expensive operation, that will have to wait until we cache that part too. Also take not that were not calling displayClient anymore, its echo’s it output and then calls exit, so that makes its pretty useless for caching.

<?php

        // setup HTTP_Cache give it our custom etag and see if we need to generate the client
    require_once ‘HTTP/Cache.php’;

    $cache = &new HTTP_Cache();
    $cache->setEtag($etag);

    if (!$cache->isValid()) {

        // Compress the output Javascript (e.g. strip whitespace)
        //define(’JPSPAN_INCLUDE_COMPRESS’,TRUE);

        // Display the Javascript client
        $G = & $S->getGenerator();
        require_once JPSPAN . ‘Include.php’;
        $I = & JPSpan_Include::instance();

        // HACK - this needs to change
        $I->loadString(__FILE__,$G->getClient());
        $client = $I->getCode();
        header(‘Content-Type: application/x-javascript’);

        $cache->setBody($client);
    }
    else {
        // something is setting Cache-Control
        header(‘Cache-Control: must-revalidate’);
    }

    $cache->send();

?>

Now if things are working correctly the web browser shouldn’t be fetching the entire page after the first time, it should just be getting a 304 header as the result. There are a couple ways to test that this is working, the easiest it too just view the client page, get the page info in firefox and see if its in your browsers cache, if not then something is broken. In debugging these types of problems i’ve found the LiveHTTPHeaders firefox extension to be useful. What your looking for is the server to send the ETag and then the client to respond with it on the next reload. The server should then respond with a 304 instead of a 200.

Now the next step in the process is to add a file cache in php for the generated client stubs so that we can turn on whitespace stripping and make the first download faster. Thats pretty simple stuff, just make your filename, check if the file exists, if so use it, otherwise generate the client as write it out to file for latter use. I’m using file_put_contents to write to the file so if your on php4 you’ll want to replace that code or check out PHP_Compat. Complete code showing both client side caching using HTTP_Cache and 304, and server side caching is shown below.

<?php

// Include JPSpan and setup your server here
// now our new updated client serving code with lots o caching
if (isset($_SERVER['QUERY_STRING']) && strcasecmp($_SERVER['QUERY_STRING'], ‘client’)==0) {
    // cache dir
    $cacheDir = APP_ROOT.“/tmp/”;

    // create a hash from the api of the handlers
    // turn the descriptions into a string
    $api = “”;
    foreach($S->descriptions as $key => $val) {
        $api .= $key;
        foreach($val->methods as $method) {
            $api .= $method;
        }
    }
    $apihash = md5($api);

    // get the host the request is being made with since it gets embedded in the client file
    $server = preg_replace(‘/[^a-zA-Z0-9._]/’,,$_SERVER['HTTP_HOST']);

    // create the filename
    $cacheFile = “client-$apihash-$server.js”;

    // create the etag
    $etag = md5($cacheFile);

    // setup HTTP_Cache give it our custom etag and see if we need to generate the client
    require_once ‘HTTP/Cache.php’;

    $cache = &new HTTP_Cache();
    $cache->setEtag($etag);

    if (!$cache->isValid()) {

        if (!file_exists($cacheDir.$cacheFile)) {
            // Compress the output Javascript (e.g. strip whitespace)
            define(‘JPSPAN_INCLUDE_COMPRESS’,TRUE);

            // Display the Javascript client
            $G = & $S->getGenerator();
            require_once JPSPAN . ‘Include.php’;
            $I = & JPSpan_Include::instance();

            // HACK - this needs to change
            $I->loadString(__FILE__,$G->getClient());
            $client = $I->getCode();

            file_put_contents($cacheDir.$cacheFile,$client);
        }
        else {
            $client = file_get_contents($cacheDir.$cacheFile);
        }

        header(‘Content-Type: application/x-javascript’);

        $cache->setBody($client);
    }
    else {
        // something is setting Cache-Control
        header(‘Cache-Control: must-revalidate’);
    }

    $cache->send();

} else {

        // This is where the real serving happens…
        // Include error handler
        // PHP errors, warnings and notices serialized to JS
        require_once JPSPAN . ‘ErrorHandler.php’;

        // Start serving requests…
        $S->serve();

}

?>

Update: added data cleaning around HTTP_HOST since I use that in a file name

This circle expands additional navigation
You should really buy my AJAX book.
You'll learn not only the technology you need for implementation but also an understanding that will help you get the most from AJAX.
You'll also have my eternal thanks :-)