Appengine Task Queue API & Gotchas

I recently added a new feature for Appsd. It shows the recent iphone related news from twitter. For this we search twitter every 10 minutes and gather all links. It shows the highest tweeted links. I used task queue api and cron for crawling. Here are things I learned from task queue api.

1. As of now Task queue api is in experimental state. So you have to import it from google.appengine.api.labs.taskqueue. When it got stabilized you have to import it from google.appengine.api.taskqueue. So proper way of importing is, first try to import from google.appengine.api.taskqueue. If it fails then import it from labs.
   


2. If the response of the request is not HTTP 200 OK then appengine tries to reexecute the task after some time. This is a big problem. It has two side effects. Task might have failed because of some temporal problem or because of some bug or exception. If it is the second case, reexecuting the task after some time won't solve the problem. Second problem is, it basically exhausts your 10000 quota limit.


3. Now there is no way to increase the quota of tasks from 10000 requests. So if each task is not taking more time then pack more tasks into single task. For instance I process multiple urls in single task.


4. If you change schema, don't forget to support the old schema. (Refer 2nd point)


5. when you change something from post to get then do support get request and return nothing otherwise task will fail. (Again refer 2nd point)

Update: Task queue uses exponential backoff scheme to prevent single error exhausting your quota. see this post.