August 1, 2009

Appengine Task Queue API & Gotchas

I recently added a new feature for Appsd. It shows the recent iphone related news from twitter. For this we search twitter every 10 minutes and gather all links. It shows the highest tweeted links. I used task queue api and cron for crawling. Here are things I learned from task queue api.
  1. As of now Task queue api is in experimental state. So you have to import it from google.appengine.api.labs.taskqueue. When it got stabilized you have to import it from google.appengine.api.taskqueue. So proper way of importing is, first try to import from google.appengine.api.taskqueue. If it fails then import it from labs.

  2. If the response of the request is not HTTP 200 OK then appengine tries to reexecute the task after some time. This is a big problem. It has two side effects. Task might have failed because of some temporal problem or because of some bug or exception. If it is the second case, reexecuting the task after some time won't solve the problem. Second problem is, it basically exhausts your 10000 quota limit.

  3. Now there is no way to increase the quota of tasks from 10000 requests. So if each task is not taking more time then pack more tasks into single task. For instance I process multiple urls in single task.

  4. If you change schema, don't forget to support the old schema. (Refer 2nd point)

  5. when you change something from post to get then do support get request and return nothing otherwise task will fail. (Again refer 2nd point)
Update: Task queue uses exponential backoff scheme to prevent single error exhausting your quota. see this post.

5 comments:

  1. Hi,

    I am pretty sure that point 2 is incorrect, I was pretty certain that the quota is based on insertions into the queue - at the point of error it is already in the queue and is not technically requeued.

    3. You can ask google to increase it like any other quota increase that yo might hit.

    ReplyDelete
  2. Hi paul,

    Its technically requeing right ?. That's why i thought it will count.

    Any way i asked the same in appengine group.

    http://groups.google.com/group/google-appengine/browse_thread/thread/5e32c52b18ac8085#

    If any body replies i will update the post accordingly.

    3. You are true.

    ReplyDelete
  3. HI I have just seen a reply.

    I was wrong, they are deducted from your quota.

    ReplyDelete
  4. This comment has been removed by the author.

    ReplyDelete
  5. Update blog to highlight this point.

    ReplyDelete