Luminati learning hub

Videos

Best Practices

A guide to buying IPs to fit your needs

Network types

• When to buy Residential IPs?

  Residential IPs are private IP addresses, meaning that your requests are routed through a real user when accessing the web. Buying residential IPs ensures that you will never get blocked or misled.

• When to buy Data Center IPs?

  Data Center IPs are also known as static IPs.

  A Static IP means you can keep using a specific IP for as long as you need (one long single session). These are more economical than Residential IPs.

• When to buy Mobile IPs?

  Mobile IPs are a segment of residential IPs that are connected to the Internet via cellular (3G / 4G) connection, rather than WiFi or cable. Mobile IPs are commonly used to verify that your ads are being displayed correctly on mobile devices with different providers.

Exclusivity types

• When to buy Shared IPs

  Shared IPs are the easiest way to start using Luminati network. They refer to global pool of IPs in various countries that are shared amongst many users to access numerous sites. you can use this option as the first entry point and scale to more exclusive one when getting blocked or misled.

• When to buy Exclusive IPs, with exclusive domains

  Exclusive Domains promise that you will be the only one using those IPs for the target sites that you have chosen.

• When to buy Dedicated IPs, with unlimited domains

  Unlimited Domains refers to IPs that are completely dedicated to you. No one can use those IPs while you are using them. Buying dedicated IPs ensures that your results will not be affected by another user of proxy services.

How to allocate IPs?

Allocation of IPsis done through the Zones section of your account.
To get started you need to create a new zone and among other settings, choose the number of IPs you want and save.
You can also update number of allocated IPs by editing the number of IPs in your zone.

When turning a zone off all IPs allocated to it are released, so when you turn the zone back on you will have new IPs allocated to it

How to download a list of my IPs?

In order to download a list of all gIPs or datacenter IPs allocated to one of your Zones there are 2 possible actions:

1. Download your IP list through your Luminati CP Zones section by entering the specific zone and clicking on 'Download gIPs list' (for gIPs) or 'Download IPs list' (for datacenter) under settings
2. Download your gIP list via API using the following command:
• For gIPs:
  curl "https://luminati.io/api/get_route_vips?" \ -H "X-Hola-Auth: lum-customer-ACCOUNT_ID-zone-ZONE-key-ZONE_PASSWORD"
• For datacenter IPs:curl "https://luminati.io/api/get_route_ips?" \ -H "X-Hola-Auth: lum-customer-ACCOUNT_ID-zone-ZONE-key-ZONE_PASSWORD"

Cost effectiveness of residential IPs

How did we calculate this table?

Your company needs to collect information from the web by sending 1,000 http requests per hour to a specific website. You write the scraper code and run it through a server. The target website allows 50 requests per minute from the same IP before blocking your scraper. Now, you have to purchase more proxies.

Assuming you choose datacenter proxies:
You don’t want to share IPs, so you buy 200 dedicated data-center IPs. You code the integration of the scraper with the new data-center proxies for 2 hours and then run the new program. This time, it takes 3 days for your target website to detect your scraper. Once your proxies are detected, you’ll have to purchase new proxies and repeat this process again, checking each day to make sure the proxies haven’t been detected. Cost per month (all numbers are from real customers):

• 200 dedicated IPs: $500
• Bandwidth: (1000 request X 20KB per request X 24 hours X 30 days = 14.5GB): $3 per month
• Developer (2 hours integration X 10 + 1 hour every 3 days for managing the proxies): ~3 days of work a month or ~$900

Your total cost per month will be at least $1400 for just these things alone, and the salary of $30 per hour for a developer is very conservative. Additionally, this doesn’t account for a lack of information reliability if your target website sends misinformation before blocking you or if your information flow is cut every few days, which can be detrimental to your brand or your revenue stream.

Assuming you choose Luminati residential proxies:
You buy a basic package of 40GB with access to unlimited residential IPs. It takes 2 hours to integrate your scraper. Due to an average of 3 million residential IPs available each day, your target website can’t detect your scraper, allowing you to focus on other projects.

The bandwidth and unlimited IPs cost just $500 per month. Your information is always reliable because your requests are always successful and access is never cut in the middle of the month. When your business grows as a result of this scraping and your projects exceed 600MB each month, the difference in costs can be much higher than just ~$1000.

Luminati also allows you to suspend your account when not in use, so your cost can be lower than $500 per month. Start by using the $5 voucher for free data-center traffic to test Luminati’s benefits and then ask for access to our residential network for cheap and reliable data collection.

Become a Luminati professional

How to crawl a website without getting blocked or misled (cloaked)?

Why should I care?
When a target website detects crawlers from a proxy (datacenter) IP, it typically

• Blocks the IP, or
• Presents the IP with purposely misleading information, or
• Throttle down the response rate

How does the target website identify my crawling activity?
Target websites log the IPs of whomever visits them and analyzes the activity of these IPs. Assuming you are using a traditional data center proxy, the target website can:

1. Identify that the activity from a single IP (the rate of requests) is much greater than what a human can accomplish in a given timeframe
2. Identify that the IP address originated from a proxy server list, which these target websites have access to
3. Identify that the IPs have the same subnet block range

How to prevent being detected?

1. To prevent being detected by the amount of requests per IP, you can reduce the number of requests per second. However, this will reduce your crawling speed
2. To prevent the target from identifying your IP as coming from a proxy server, you must rotate your requests through residential IPs. You should be able to circulate through enough IPs that the target website can not detect your activity
3. When using residential IPs there is no subnet block range

By using a traditional proxy solution, it’s only a matter of time before the target website will identify your crawling activities, and can block or provide you with the wrong information.

How to get an IP in a specific city

Why should I care?
Example: If you are responsible for testing Yelp’s city level service - you need to check the site from 10,000 different cities around the world.

How to get an IP in a specific city?
If you only use a traditional data center proxy solution for your information collecting tasks, you are limited to where these datacenters are located. Large residential networks can get you IPs in any specific city in the world.

How to accelerate your web scraping

Why should I care?
If the number of requests you rotate through a single IP are higher than what target websites allow, the website you target will identify your IP and block or mislead you with false information. It means that your information collecting can be much slower than what you’re used to.

How do I improve the speed of my data harvesting?
Assuming you're running 10 million requests, 1 request per second per IP with 1000 data center IPs, your routine can take about 3 hours. With 10,000,000 residential IPs, your routine can potentially take 1 second.

Guidelines to rotate multiple parallel sessions through Luminati’s residential network:

1. Open Luminati Proxy Manager
2. Go to the ‘proxies’ tab
3. Check the port of your residential zone
4. Edit in the port settings ‘preset’ to ‘round-robin (ip) pool’
5. Route your requests to 127.0.0.1:{portnum} where the {portnum} is the port of the residential zone

How to rotate your IP address

Why should I care?
When many requests are sent to a website from the same IP, the website can tag the IP used as a crawler and send misleading information or block you. Periodically changing, or rotating, your IP address helps prevent a target site from identifying your IP as a crawler. Rotating your IPs can drop your failure rate to below 1%.

How do I rotate my IP address?
With its easy-to-use proxy manager, Luminati allows you to control:

• Number of requests permitted before changing the IP
• Amount of time between IP changes
• Quantity of IPs in your rotation pool

Guidelines for rotating your IP address with Luminati:

1. Open Luminati Proxy Manager
2. Go to the ‘proxies’ tab
3. Click on the proxy you want to edit, then on the edit button
4. Browse rotation options under 'Preset' or 'IP Policy'
5. Route your requests to 127.0.0.1:{portnum} where the {portnum} is the port of the residential zone

How to use SOCKS5 with Luminati

Why should I care?
A SOCKS server is a proxy server that works for any type of network protocol on any port and establishes a connection to a server on behalf of a user, then routes traffic between the user and the server.

Why use SOCKS5 with Luminati?
When you use SOCKS5 with Luminati, the proxy manager converts any requests to port 80 or port 443 to http and https requests, so you don’t have to worry about which format is accepted by your target site. With any other port, the traffic is sent as-is between the user and the host.


Guidelines to use SOCKS5:

• Open Luminati Proxy Manager
• Click a port, then 'Edit Proxy'
• Under 'General' add SOCKS port number
• Go to "Performance" - Set Reverse Resolve: DNS
• Add a resolve file
  • Add '--resolve PATH' to the program code
  • Follow the hostsfile format
  • List domains you are going to use and their IPs
  • Add domains and IPs from the resolve file editor in the proxy manager 'tools' page

How do I avoid subnet block range?

What is a subnet block range?
Smaller sections of a network are called subnets, which are useful for grouping hosts together and managing them all at once. Subnets are based on IP address, making it easy for websites or malicious users to target or block an entire subnet. For example, blocking 223 would block all IP addresses that begin with 223, while blocking 223.1 would only block IP addresses that begin with 223.1, and so on - this is the subnet block range.

Why should I care?
Because a database’s IPs are often all within the same subnet, they are easily blocked by websites, especially if they are known proxies.

Avoiding subnet block range:
Residential proxies cannot be blocked this way, because they don’t have subnet ranges. Using Luminati’s residential IPs ensures you won’t be affected if a website uses the subnet block range method.

How to switch from API to Proxy Manager

Why should I care?
Using the Luminati Proxy Manager offers advanced features that are not readily available in the API. Instead of having to manually code mechanisms for tasks like keeping an IP as long as possible or rotating your IP after each request, you can simply click a button in the proxy manager.

Guidelines to to switch from API to Proxy Manager:

1. Install the Luminati Proxy Managerhere
2. Change the code to send HTTP requests directly to the specified port (for example, 127.0.0.1:24000) instead of to zproxy.lum-superproxy.io:22225
3. Configure the settings for your custom proxies and Zones through the LPM dashboard. You do not need to send the user parameter (lum-customer-customer_name-zone-zone_name…) alongside your requests, as all the needed data is wrapped within the manual proxy configuration.

How to configure Luminati raw API with iPhone

Guidelines for configuring Luminati raw API with iPhone

1. Open your device Settings app and go to Wi-Fi
2. Tap the name of the Wi-Fi network you are connected to
3. Scroll to the bottom and you will find a section for 'HTTP Proxy'. This is set to 'Off' by default. Set it to 'Manual':
• In the server slot enter:zproxy.lum-superproxy.io
• In the port slot enter: 22225
• Select Authentication: DISABLE
4. Open Browser
5. Add the target URL
6. In the Pop Up, tap settings
7. Add the username (e.g. lum-customer-lum_qa-country-us-zone -city)
8. Zone Password and tap OK

How do I know if I’m getting cloaked? (misleading information)

Why should I care?
Getting cloaked means that you’re getting misleading information from the website you are scraping.
Example: If you are collecting comparative competitive information to feed your automatic pricing algorithms, the target website can return artificial lower prices to your requests, to skew your pricing and profits.

How to know when you’re getting cloaked
When using traditional proxy networks (data center based IPs), your target websites may identify your activity quite easily and may cloak your requests. Therefore, the only way to ensure you’re not getting cloaked is to rotate your requests through residential IPs.

Guidelines for rotating requests through millions of residential IPs:

1. Open Luminati Proxy Manager
2. Go to the ‘proxies’ tab
3. Check the port of your residential zone
4. Edit in the port settings ‘preset’ to ‘round-robin (ip) pool’
5. Route your requests to 127.0.0.1:{portnum} where the {portnum} is the port of the residential zone

Introducing: Luminati Chrome Extension

Why should I use it?
Use the Chrome extension to self-test your website, verify your ads, or simply browse a site as if from another country. It’s a powerful complement to the Luminati Proxy Manager and an easy-to-use tool for less technical users. You can also have people who don’t have access to the dashboard use the extension without knowing the account credentials.

What features are available in the extension?
Luminati Chrome Extension supports datacenter and residential IP browsing, allowing you to search from any country. You can adjust the user agent, customize configuration of the DNS to maximize discretion or speed, and set random IP rotation.

Guidelines to download Luminati Chrome Extension:

• Download Luminati Chrome Extensionhere
• The extension will appear next to your search bar
• Click the dark blue button in the extension
• Set up extension parameters in the dedicated set up page

FAQ