In order to help you make the most out of Luminati, we collected the best practices that proved themselves over and over again.
Here is a general list of tips that help improve your results using proxy, no matter what is your specific goal
Slower connections may get a timeout from the super-proxy server (in the header) or from the target website (in the https). To avoid timeouts set timeout period to be longer (in the request). When you do that, you may want to increase the amount of IPs that you use to send request proportionally to the change in timeout to keep the pace at which you send requests.
- IP Country: Set the country of your IP (in the Zone) to the same country as the destination site. It makes your request look more like that of a human user, which in some cases prevents getting blocked or mislead.
- IP City: Setting the city targeting of your IP ensures that when your Residential IP changes, the new IP will come from the same city and won't look like you traveled hundreds of kilometers in one moment, reducing the chance of getting blocked.
- Rate Limit: When sending multiple requests via the same IP to the same destination, make sure to send up to 10 requests per day and avoid getting blocked or mislead.
Sometimes target sites can detect users as a bot or a crawler. In this case include headers and cookies in your http/https requests in order to mimic a human behavior.
When you automatically collect public data from professional networking sites you may get an error response, followed by a JS redirect to an endpoint that sets cookies that block your IP for several hours, and then get redirected again to the login page. If the login flow is completed it means that they let you access to the requested resource. Find below some tips for collecting public data successfully with Luminati:
- Use gIPs
- Rotate between as many unblocked IPs as possible
- Make sure your requests look like those coming from a browser
- Cookie behavior must be that of a real user
- Rotate User Agent
- Periodically throttle (slow down) your request rate
- Some websites set trap links, make sure to avoid them
- Crawl pages in the same order that a real user would (e.g. do a search before viewing a profile)
- Use a Referer header that matches your crawling pattern
- Randomize crawling patterns, do not use the same one all the time
- Use remote DNS
- Use a headless browser (if there is a mismatch between the implemented user-agent and the browser used you can be flagged)
- Use the Luminati Proxy Manager and check the Success Rate of your requests
- Ask your success manager about using the Proxy Manager in debug mode
While using a script for checking product prices of an online store, some of the requests may get blocked or cloaked (altered prices). In order to improve the chance to have successful requests, try out these tips:
When using proxy for purchasing goods in product releases while keeping your anonymity, you'd probably want to:
- Make many requests simultaneously
- Speed up the requests as much as possible
- Keep the same IP alive throughout the session of each request
In case your requests get blocked:
- Use remote DNS. It will slow down your requests, but it costs the same and the probability that the request will go through is much higher
- If they still get blocked, go to the Control Panel's Zones page, edit your Zone (pencil icon), and in Plan tab under Permissions choose City. Then go to the LPM's proxies list and edit your port, go to Targeting, and select a country, a state, and a city. City level targeting is more expensive than the default residential country level targeting, but it significantly increases the chance of a successful request
- Power tip: Consider switching automatically to the alternative settings mentioned above upon failure of a request using Rules, it will optimize both your results and your proxy costs
- Set the country of the request IP to your country with a 2-letter country code, e.g "us" = United States, to make the requests go shorter distance and arrive faster
- Set a different session ID to each request with sequential number (1,2,3,4...) to keep previous sessions alive
- Find fast Super-Proxies close to you by running the following command in CMD/terminal: nslookup servercountry-COUNTRY.zproxy.lum-superproxy.io (replace COUNTRY with a 2-letter country code)
- Take the Super-Proxies that you found and rotate between them with every request sent
- Request examples:
- First request
curl --proxy 18.104.22.168:22225 --proxy-user lum-customer-somecompany-zone-somezone-country-us-session-0001:s0m3passw0rd "http://lumtest.com/myip.json"
- Second request
curl --proxy 22.214.171.124:22225 --proxy-user lum-customer-somecompany-zone-somezone-country-us-session-0002:s0m3passw0rd "http://lumtest.com/myip.json"
- Third request
curl --proxy 126.96.36.199:22225 --proxy-user lum-customer-somecompany-zone-somezone-country-us-session-0003:s0m3passw0rd "http://lumtest.com/myip.json"
- Fourth request
curl --proxy 188.8.131.52:22225 --proxy-user lum-customer-somecompany-zone-somezone-country-us-session-0004:s0m3passw0rd "http://lumtest.com/myip.json"
Creating and managing many accounts from your office requires each account to behave like one person is managing it.
- Create long single sessions by using the "Long single session (IP)" preset in the Proxy Manager, to keep the same IP throughout the session
- If you use a bot, you can create a port in the Proxy Manager and then multiply it as many times as needed
- When approaching more sensitive websites:
- Here are 2 of the most popular account management softwares amongst Luminati customers:
Due to the nature of the residential network the IPs will rotate. Therefore, using Mobile IPs is important because target sites are less restrictive to block those IPs even though they are being changed frequently.
When using our Mobile IPs network it is helping with increasing success rates and getting less blocked by target sites.
- Create a new zone in the CP with mobile permissions
- In the Proxy manager multiply ports per gIP
- Rotate IP every 10-20 requests to enlarge your success rate
Verifying that your ads are displayed and behave as intended requires to access them anonymously from various devices and locations, to click them and check the link's destination.
- Check how many redirects there are until you reach the final landing page and adjust max requests per IP accordingly
- Keep the same IP throughout the session by using the "Long single session (IP)" preset in the Proxy Manager
- After verifying the final destination, automatically change IP and test again. For best results use the Proxy Manager: Create a port for each of the tests you want to carry out, and set a rule for each port that triggers the next relevant port and then the next, until the last test is done
- Use remote DNS to perform the DNS resolution at the Proxy Peer
- Different User Agents (browser property) give different results, so test with all the relevant User Agents one, and also sending many subsequent requests with the same User Agent may get them eventually blocked or cloaked
The following use-cases will be added to this Best Practices guide in the next few weeks:
- Sales Intelligence
- Brand Protection
- Data Collection
Including all headers and cookie behavior. See bash example, using Luminati Proxy Manager:
curl --proxy http://127.0.0.1:PORT_NUM "http://lumtest.com/myip.json" -H 'pragma: no-cache' -H 'accept-encoding: gzip, deflate, br' -H 'accept-language: en-US,en;q=0.8' -H 'upgrade-insecure-requests: 1' -H 'user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36' -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp, image/apng,*/*;q=0.8' -H 'cache-control: no-cache' -H 'cookie: bcookie="v=somekindofcookiexxx";' -v
How to generate bash request from your browser
Switch an IP when it gets an error, and use it again only after it cools down several hours.
For optimal results use Luminati gIPs.
To do this you have 2 options:
- Use our API to control the IP rotation:
Use our Proxy Manager to achieve the same by setting a simple Rule:
- If you use gIPs, get all available gIPs per zone: /api/get_route_vips
curl "https://luminati.io/api/get_route_vips?" \ -H "X-Hola-Auth: lum-customer-lum_qa-zone-static-key-s6u19zy0zrys"
- Use the IP until it gets an error (it is recommended to stop after 10 times)
- Manage a log of blocked IPs: Document the IP and time it got an error
- Get a different IP that didn’t get an error in the last X hours
- Check the IP by session ID using lumtest.com or using our API
- Check in your log when was the last time it got an error
- Repeat this as many times as needed
- Go to Edit Proxy > Rules
- Under 'Rule Type' choose 'Status-code'
- Under 'Status Code string to be scanned' choose the status code that you get when the request get blocked, or choose custom and write it manually in the field that appeared below (e.g "888")
- Under 'Action type' choose 'Retry request with new IP'
- Under 'Number of retries' write the amount of times you want to retry sending a request using the same settings and a new IP
Port Rules allow you to define a custom action for specific request response.
Use Port Rules to automatically switch from one port to another in case of a failed request.Consider using the Luminati waterfall approach or building other rule sets, in order to maximize request success rate and reduce the price you pay.
Using Remote DNS gives you a DNS resolve server similar to that of the Proxy peer (the origin of the Proxy's IP). This makes your request look more credible as the resolve server is usually located geographically close to the IP, whereas without using Remote DNS the resolve server is one of Luminati's Super-Proxies which may be located in another country. To use Remote DNS:
- Via Proxy Manager, go to Proxies tab > Edit Proxy > Request Speed tab > set "DNS Lookup" to "Remote - resolved by peer"
- Via API simply add it to your username string. For example:
- Create as many duplicates of the proxy that you created as you need. Click Edit (pencil icon) on the proxy that you created, then go to "General" tab, and under "Multiply" write the amount of times that you want the proxy to be duplicated.
- You can then download the proxies list (contains IPs and ports) by clicking the CSV button that's on the top of the list.