This is for harvesting urls. And for private, dedicated proxies.
So I have read a few threads on this and from what I can gather:
- Be careful with 'advanced operator' searches in Google (eg. In url) in your footprints
- A previously burnt out proxy will probably burn out quicker
- Test and optimize
My question is regarding the third point. For testing and optimizing, I want to make sure I am tweaking the right things. Here is what I am tweaking:
- Connections for the Harvester - and more specifically the ratio of connections:proxies
- Harvester timeout setting
Are there any other settings I should be playing around with for my goal of not burning out my proxies?
So I have read a few threads on this and from what I can gather:
- Be careful with 'advanced operator' searches in Google (eg. In url) in your footprints
- A previously burnt out proxy will probably burn out quicker
- Test and optimize
My question is regarding the third point. For testing and optimizing, I want to make sure I am tweaking the right things. Here is what I am tweaking:
- Connections for the Harvester - and more specifically the ratio of connections:proxies
- Harvester timeout setting
Are there any other settings I should be playing around with for my goal of not burning out my proxies?