August 1, 2009

Appengine Task Queue API & Gotchas

I recently added a new feature for Appsd. It shows the recent iphone related news from twitter. For this we search twitter every 10 minutes and gather all links. It shows the highest tweeted links. I used task queue api and cron for crawling. Here are things I learned from task queue api.
  1. As of now Task queue api is in experimental state. So you have to import it from google.appengine.api.labs.taskqueue. When it got stabilized you have to import it from google.appengine.api.taskqueue. So proper way of importing is, first try to import from google.appengine.api.taskqueue. If it fails then import it from labs.

  2. If the response of the request is not HTTP 200 OK then appengine tries to reexecute the task after some time. This is a big problem. It has two side effects. Task might have failed because of some temporal problem or because of some bug or exception. If it is the second case, reexecuting the task after some time won't solve the problem. Second problem is, it basically exhausts your 10000 quota limit.

  3. Now there is no way to increase the quota of tasks from 10000 requests. So if each task is not taking more time then pack more tasks into single task. For instance I process multiple urls in single task.

  4. If you change schema, don't forget to support the old schema. (Refer 2nd point)

  5. when you change something from post to get then do support get request and return nothing otherwise task will fail. (Again refer 2nd point)
Update: Task queue uses exponential backoff scheme to prevent single error exhausting your quota. see this post.


  1. Hi,

    I am pretty sure that point 2 is incorrect, I was pretty certain that the quota is based on insertions into the queue - at the point of error it is already in the queue and is not technically requeued.

    3. You can ask google to increase it like any other quota increase that yo might hit.

    1. Machine Learning Projects for Final Year machine learning projects for final year

      Deep Learning Projects assist final year students with improving your applied Deep Learning skills rapidly while allowing you to investigate an intriguing point. Furthermore, you can include Deep Learning projects for final year into your portfolio, making it simpler to get a vocation, discover cool profession openings, and Deep Learning Projects for Final Year even arrange a more significant compensation.

      Python Training in Chennai
      Python Training in Chennai

      Angular Training Project Centers in Chennai

  2. Hi paul,

    Its technically requeing right ?. That's why i thought it will count.

    Any way i asked the same in appengine group.

    If any body replies i will update the post accordingly.

    3. You are true.

  3. HI I have just seen a reply.

    I was wrong, they are deducted from your quota.

  4. This comment has been removed by the author.

  5. Update blog to highlight this point.