There and Back Again

Webthumb updates

Webthumb got off the ground yesterday without too many problems. It ran into problems right out of the gate since a large number of sites don’t seem to want to finish loading correctly. I was able to fix this by adding an extra timeout handler to the browser code. If we stop making progress loading a site for 5 seconds we just give up and snapshot what is currently loaded.

I also updated the server config, Webthumb now supports flash (version 7), and has a lot more fonts installed so sites that need non-latin characters should be working correctly.

My current feature goals are:
Better Url’s, something like /thumbs/url/time
5 minute cache on thumb generation (if anyone else has requested the same site in the last 5 minutes just return those images)
A way to get a thumb of a site without going through the queue, if its already been generated
A way to search for current thumbs
User accounts with:
Snap at time
Snap recurring

I’m also looking at prototying and API for access, though it may require paying a fee to use.

If you have any feature suggestions or input on an API leave a comment.

52 thoughts on “Webthumb updates

  1. Malte Steckmeister

    Feature suggestions (besides better filenames):
    – selectable size(s) of capture area (800×600,1024×768,…,custom)
    – selectable size(s) of thumb
    – possibility to download all the generated thumbs from a url as one zip
    – option to turn vertical and/or horizontal autoscroll on, so that a full page can be captured even if it exceeds the client window of the capturing browser

  2. Pingback: PHPDeveloper.org

  3. tobozo

    nice work ^^

    by any chance, is there any API snippet code that would not crequire url and php5 to use this service ?

    curl is not installed on most webservers (given the multiple security issues), and php4 is still more popular than php5 ..

    be well

  4. Joshua Eichorn Post author

    I haven’t written any but it should be really easy to take the code from Hasin, and update it too use HTTP_Request. I just use a little JavaScript tester for making sure things work so I don’t have any PHP code that talks to the api.

  5. User

    WTF?

    Notice: Undefined index: referrer in /home/htdocs/jeichorn/bluga.net/webthumb/index.php on line 32

    Warning: Cannot modify header information – headers already sent by (output started at /home/htdocs/jeichorn/bluga.net/webthumb/index.php:32) in /home/htdocs/jeichorn/bluga.net/webthumb/index.php on line 39

  6. Joshua Eichorn Post author

    I wasn’t handling a missing referrer correctly, im guessing your run a privacy proxy or something. Anyhow it should be fixed now.

  7. Janakan Arulkumarasan

    Hi! I love the site. I would be willing to pay for a service which let me download large thumbnails quickly for use (for example) in a search engine like Snap.com. I don’t mind if they are cached for up to a month, so long as you’re hosting them!

  8. Joshua Eichorn Post author

    MarcAndre, for the time being there isn’t much i can do for sites like that. Something about there layout makes they take an extremly long time to render even after all the content is downloaded.

    I’ll be looking at better load detection techniques in the future but there doesn’t seem to be any easy solutions.

  9. RD

    How does one purchase credits for webthumbs, I suspect I’ll need 5000 per month with my project

  10. Aaron

    Hey quick question, what browser is the snapshot generated from? My site seemes to display ok in ie and firefox on mac and pc but the snapshot being generated is showing an alignment issue that I can’t seem to locate.

    Thanks

  11. Joshua Eichorn Post author

    Its a mozilla browser running on linux. I think its the equivalent of firefox 1.0.8 The fonts might also be slightly different.

  12. Ben

    Wow this is awesome.

    Is there a time-delay or are the images generated LIVE on-the-fly?

    I mean if I generate a thumbnail, does the image represent the site as it was when I clicked or it from some time ago. If it is not live, is there an indication of how reent the image is?

    Thanks,

  13. Joshua Eichorn Post author

    They are generated on the fly, thumbnails normally take 5-20 seconds to generate depending how busy the server is.

    There is one exception if your using the recently added easythumb features images are cached for at least 24 hours.

  14. Loopion

    Hi,

    We launch a service called Menthe-fresh (http://www.menthe-fresh.fr) and we use your service. We would like to mention that there is a little bug on your price list.

    You mention on your home page that it is $20 for 15,000 credits.
    “Registering gives you API access and free 250 credits a month, additional credits can be purchased at any time, and cost $20 for 15,000 credits.”

    I am a register user and I see that it is :

    $20 – 5,000 WebThumb Credits
    $50 – 15,000 WebThumb Credits

    How much should I pay ?

  15. Joshua Eichorn Post author

    Loopion:
    Sorry about that the list on the purchase page is the correct amount:
    $20 – 5,000 WebThumb Credits
    $50 – 15,000 WebThumb Credits

  16. matbennett

    Excellent resource. Struggling to get the API running without curl, but it definitely looks like it is worth percevering with. Has anyone had any joy converting Hasin’s code for non-curl environments?

  17. Bill

    I tried a page that displays a quicktime movie from Darwin Streaming Server. Actually, there is a poster movie (still) displayed first. The thumbnail shows the “Get Plug In” dialog.

    Can you provide a list of content types and plugins that are supported?

  18. Joshua Eichorn Post author

    Bill:
    I’m only running the flash, and PDF plugins right now. Other then that I support everything firefox 1.5 does.

    I’m looking at adding support for direct thumbnailing of movies but that will be a related service.

  19. Bill

    Coming up with a good thumbnailer for movies is very tough. When we posted a movie to Youtube, we found they grabbed a frame from the middle (I think). But the middle frame wasn’t a good representative, so I had to add black frames to the end of the movie until the middle was a decent image.

    The page I was trying to thumb was http://www.raisethebar.com/play.taf?_uid1=2, which is a streamed, autostart quicktime movie (mpeg4). I was mistaken that we had a poster movie for it, as this page dynamically embeds the movie player. With autorstart, I don’t know what you would have grabbed, even if you had an mp4 plugin installed, it would be very timing dependent.

    The same is true if I had asked you to thumb the movie directly http://www.raisethebar.com/movies/Buzz_Report.mov (rather than embedded in a page). I think this is a problem that requires some eyeballs to solve.

  20. Joshua Eichorn Post author

    Bill: I was actually planning on doing a thumbnail set on the direct movie thumbnail service. Plugin wise i may be able to support more media types for people who want the video in a page to look right, but that depends on me getting the new server setup which i’ve been dragging my feet on.

  21. daniel englert

    This is really great, is there anyway to get a complete full size image?
    I’d love to use this with a commercial project I’m slated to start on in a few weeks in which we’d need to purchase a lot of keys. It would be a highly trafficked site with a lot of requests to your capture engine. I don’t think we’d have time to setup any of the open source alternatives on our own servers.

    The only issue is I would need to actually get the full screensize back, not a scaled thumbnail.

    Is there anyway to do this? It seems like on the server side it always scales the images.

    In any event, cool project!
    -dan

  22. Joshua Eichorn Post author

    Dmitri:
    The PHP API wrapper is open source

    An older version of the custom mozilla browser is open source, i have a private branch of that code at this point.

    The api etc isn’t open source

    daniel:

    There is an undocumented feature that lets you get the full image, it makes requests cost 2 credits instead of 1.

    You just add <fullthumb>1 to the request like:
    <webthumb>
    <apikey>yourkeyhere</apikey>
    <request>
    <fullthumb>1</fullthumb>
    <url>webthumb.bluga.net</url>
    </request>
    </webthumb>

  23. marques

    You thumbnail service (which is great!) does not capture YouTube videos, could this issue be fixed? I would like to create thumbnails of flash video (if you have not already guessed it).

  24. Marques

    Did you disable your flash plugin? You mention in your post that you have added flash support. I just started using this service (which rocks!) and I remember that I was able to see the flash objects, but now I cannot (maybe I am just going crazy).

    Marques

  25. Joshua Eichorn Post author

    marques:
    For videos you would really need a specific version just for them to get the results you want. I’ve looked into creating one but thats a matter of finding the time to get it up and running.

    The Flash Plugin is working again. The service leaks shared memory and when it hits its limits the plugin doesn’t work right anymore.

  26. Marques

    Thanks Joshua for fixing the flash plugin, actually I already have a solution for creating thumbnails for wmv/mov/flv files directly, if that is what the other posters are looking for. Contact me and I can give you details.

    Thanks!

  27. Marques

    I looked in the pageprint file and found a possible solution to the errors that I am getting when using your services.

    // quoted out existing code and kept as a reference/backup
    //$command = “$dir/PagePrint-bin -i $procUniq $cookieHack$args 2>&1”;
    // My cheat,
    $command = “$dir/PagePrint-bin -i $procUniq $cookieHack -s ‘$argv[2]’ $argv[3] 2>&1”

  28. JiveBay

    I’ve noticed that your computer doesnt have the font “Trebuchet MS” installed. Is this because the server is on Unix/Linux? I’m not sure if you could add it, but it would be cool if you could. 🙂

  29. Joshua Eichorn Post author

    JiveBay:
    Its a Linux box, so it has some different fonts installed from the default windows set. I’ll see what options i have for adding some Microsoft fonts.

  30. Matthias

    Hi Joshua,

    we love your service and use it for our worldwide database of scuba diving shops.

    A nice feature would be to be able to capture the full page, if it is longer than one screen. Something like autoscrolling.

    Any chances to get that into the service?

    Matthias

  31. Joshua Eichorn Post author

    Matthias:
    Adding autoscrolling for full page capturing sounds like an interesting feature. But i’m not sure when i’ll have time to work on it.

  32. Ketlan

    An extremely useful service though it appears to have stopped working properly. Are you letting it go or do you want the facility mirrored elsewhere?