I recently added a new feature for Appsd. It shows the recent iphone related news from twitter. For this we search twitter every 10 minutes and gather all links. It shows the highest tweeted links. I used task queue api and cron for crawling. Here are things I learned from task queue api.
Update: Task queue uses exponential backoff scheme to prevent single error exhausting your quota. see this post.
- As of now Task queue api is in experimental state. So you have to import it from google.appengine.api.labs.taskqueue. When it got stabilized you have to import it from google.appengine.api.taskqueue. So proper way of importing is, first try to import from google.appengine.api.taskqueue. If it fails then import it from labs.
- If the response of the request is not HTTP 200 OK then appengine tries to reexecute the task after some time. This is a big problem. It has two side effects. Task might have failed because of some temporal problem or because of some bug or exception. If it is the second case, reexecuting the task after some time won't solve the problem. Second problem is, it basically exhausts your 10000 quota limit.
- Now there is no way to increase the quota of tasks from 10000 requests. So if each task is not taking more time then pack more tasks into single task. For instance I process multiple urls in single task.
- If you change schema, don't forget to support the old schema. (Refer 2nd point)
- when you change something from post to get then do support get request and return nothing otherwise task will fail. (Again refer 2nd point)
Hi,
ReplyDeleteI am pretty sure that point 2 is incorrect, I was pretty certain that the quota is based on insertions into the queue - at the point of error it is already in the queue and is not technically requeued.
3. You can ask google to increase it like any other quota increase that yo might hit.
The article highlights several practical issues that can easily be overlooked when working with background tasks. The discussion of automatic retries after non-200 responses, task quota limitations, batching multiple URLs, and maintaining compatibility when changing schemas or request methods provides useful lessons for building more reliable web applications.
DeleteThe retry behavior described here is particularly relevant to RESTful API Course concepts, because HTTP response handling directly influences how a task is considered successful or failed. The recommendation to return an appropriate response instead of allowing unnecessary retries is a useful design consideration for API-based applications.
The article also provides a good reminder that changing request methods or application behavior requires backward compatibility, something worth considering when developing services with FastAPI Course concepts. Handling older requests gracefully can prevent avoidable task failures and repeated processing.
DeleteHi paul,
ReplyDeleteIts technically requeing right ?. That's why i thought it will count.
Any way i asked the same in appengine group.
http://groups.google.com/group/google-appengine/browse_thread/thread/5e32c52b18ac8085#
If any body replies i will update the post accordingly.
3. You are true.
HI I have just seen a reply.
ReplyDeleteI was wrong, they are deducted from your quota.
This comment has been removed by the author.
ReplyDeleteUpdate blog to highlight this point.
ReplyDeletekuşadası
ReplyDeletesamsun
marmaris
muğla
malatya
QNCF
bergama transfer
ReplyDeletegüzelbahçe transfer
foça transfer
mordoğan transfer
aliağa transfer
balıkesir transfer
NCW07L
Niice blog thanks for posting
ReplyDeleteGreat post! If anyone is searching for top VLSI training institutes in Ameerpet, I highly recommend checking this program.They provide practical-oriented training with real-time projects, which is essential for mastering ASIC design and verification.
ReplyDelete