WebScraping

File: 1455271614.b8acc93ea98149f2bf821d4b49a68b07.jpg (30KB, 755x363px) Image search: [Google]

30KB, 755x363px

Lets say i have a simple webscraping tool that i have created as a tutorial project.

I scrape a house reseller site every 5-10 min (random value), take all new houses added since last scrape and add them to my database.

I have couple of questions, in different areas:

1. What's the best way to take all new houses (with it's parameters such as sq. meters, price etc) and store them in a database? How do i store only new houses, all old ones will already be in a database

2. Should i store every house as a separate query, or should i first create a batch of new houses and use ACID transactions to save them? If batch is the solution, what would be the best way to implement it?

3. How to not get banned? My plan is to set random delay time, from 5-10 min and list of real useragents. Anything else i can do?

4. Any general tips for total noob?