Free proxies are slow, an issue that can cause frequent request timeouts. Some do not offer high uptime; it’s in fact not uncommon to come across a free proxy list showing free proxies with uptimes of less than 50%. Given the poor performance and unreliability, free proxies are only ideal for small-scale web scraping, educational purposes, and testing web apps across different regions. Nonetheless, you can consider using proxy rotation to enhance the free proxies’ effectiveness as you perform these limited activities. This article walks you through the steps to follow to set up proxy rotation using free proxies. Let’s get started.
Table of contents
What is Proxy rotation?
Proxy rotation is the process by which the assigned IP address changes with each request. The process is a tad random. This means you cannot easily tell which IP address in the free proxy list will be assigned to your connection.
There are two ways in which you can implement proxy rotation:
- Third-party, ready-made proxy rotator
- Custom script created using Python or any other scripting language
Ready-made proxy rotators are available for sale. And paying for such a tool wouldn’t make sense if you simply wanted to use free proxies as a cost-saving measure. A paid tool would nonetheless be perfect for rotating paid proxy servers. This leaves us with the second option: scripting using Python. And we have provided a step-by-step guide on how to set up such a script. But first, let’s discuss the benefits.
Benefits of proxy rotation
- Proxy rotation handles timeouts: It picks the next proxy in the free proxy list when the previous one takes too long to complete requests.
- The rotator reattempts to connect to free proxies whose connection wasn’t successful previously. This way, they route traffic through as many proxies in the free proxy list as possible.
- Proxy rotation increases scraping success by enabling you to achieve a human-like pattern of sending web requests; it does this by limiting the number of requests associated with a single IP address
Implementing proxy rotation using Python
Here are the steps to create a proxy rotation script.
Step 1: Install Python and libraries
Given that you’re working with Python, ensure you have installed the latest version of Python. Your IDE (e.g., Visual Studio Code) lets you download and install the Python package. Next, type pip or pip3 install requests on the terminal.
Step 2: Import free proxy list
Create a free proxy list by visiting free proxy providers’ websites. You can either save this list as a .csv or .txt file.
Step 3. Handle timeouts and send requests
The third step entails defining the maximum time you are willing to wait for a single free proxy to load. This is captured in some of the scripts below. The scripts also use some of the methods defined in the Requests library. We have used some of these methods/functions, including requests.get, response.status_code, and response.text, in the scripts below.
Step 4: Proxy rotation
Rotate free proxies sequentially
Sequential proxy rotation evenly distributes the number of requests between the available proxies. It involves creating a for loop that sends the requests and uses Python’s itertools.cycle to cycle through the proxies. The code below sends three requests per proxy.
Rotate free proxies randomly
To rotate free proxies randomly, import the Random library. This library contains the random.choice method that randomizes the free proxy list inside the for loop. And like the sequential method, it distributes the requests among the randomized proxies.
Synchronous proxy rotation
The synchronous method checks all the proxies in a free proxy list one by one. It also tackles one request at a time and only moves to the next one once it completes the preceding. This makes the synchronous method extremely slow.
Asynchronous proxy rotation
The asynchronous method sends all requests simultaneously and in a non-blocking way and waits for the responses. It’s, therefore, faster than the synchronous method. Before writing the code, use the pip or pip3 install aiohttp command to install the aiohttp package. Next, use the code below.
Conclusion
Proxy rotation helps prevent websites from associating a single IP address with large volumes of traffic. This way, it prevents blacklisting. It can be used to deal with, at least to some extent, the unreliability of free proxies. You can create a script that rotates the proxies on a free proxy list. If you’re looking for options to start with, consider exploring some of the best free proxy providers.
Your script can define the number of requests that should originate from each server. Additionally, you can determine how the script selects the proxies in the free proxy list. It can take a sequential or random approach. The proxy rotator in the script can also capture the time you are willing to wait for a single proxy to respond. The script selects the next available free proxy if this timeout limit is surpassed. Still, and despite these efforts, paid proxy services are still preferred over free proxies.
