Unlocking the Potential of Competition Tracking: The Value of Web Scraping

Daria Jemli

14 February 2023

Passenger

Share

Why efficient web scraping is key for competitive success across industries

In today’s rapidly evolving market, companies across various industries are constantly seeking ways to stay ahead of the competition and uncover new revenue streams.

At Wiremind, we specialize in pricing strategies for a variety of sectors including sports, air freight, and transportation. Through our work, we have witnessed the growing importance of monitoring competitors, deciphering pricing algorithms, and identifying untapped opportunities for growth. In particular, the transportation industry has become increasingly competitive with the rise of dynamic pricing and complex fare structures, making it more essential than ever to stay on top of industry trends.

To achieve this, a robust web scraping system plays a crucial role by providing the different departments within a transportation company with the data they need to improve their operational margins. From the revenue management team that needs real-time insights to adapt to market changes, to the pricing and marketing teams that need a clear understanding of the fare structure, and the scheduling team that wants to analyze company routes – all can benefit from efficient data gathering.

Scraping solutions are expected to meet the highest standards of quality and comprehensiveness not just by transportation operators, but by any company operating in a competitive environment. However, some providers often fall short in delivering the data required to meet these expectations. They often struggle with low-quality data and limited sources and don’t offer an effective way to analyze, visualize, and make sense of billions of data points.

From data scraping to data delivery: Wiremind’s journey to maximizing data quality

Wiremind, as a Revenue Management Solution provider, recognized the impact of poor data quality on revenue uplift, especially as machine learning algorithms took center stage. Training and using a model requires the utmost level of quality to ensure accurate decision-making.

To overcome this challenge, Wiremind developed their own Fare Tracking & Competition Data service, CAYZN Tracking. This solution overcomes the limitations of existing Fare Tracking tools by using cutting-edge web scraping technologies, custom undetectable browsers, in-house proxies, travel-specific data mapping, and real-time integration with the company’s Revenue Management tool, ensuring that it meets the highest standards of quality and comprehensiveness.

With a 97% success rate over the scraping perimeter from publicly available sources, and more than 20 million data points collected daily over the past few years, CAYZN Tracking has received high praise from its customers. Wiremind’s commitment to clean data and modern UX and Workflow standards ensures that our clients can quickly and easily analyze, visualize, and make informed decisions from the data we provide. And, as a result, they see a significant impact on their ROI compared to other providers who struggle with low-quality data and limited sources.

The data processing methodology: from scraping to delivery

The workflow of data processing in CAYZN Tracking is a multi-step approach that covers the entire journey from defining the scraping perimeter to delivering the processed data to the client and making it available through our dedicated application. The process starts with the agreement between Wiremind and the client regarding the scope of the data to be scraped. This file is then loaded into the platform through a user interface, enabling the user to keep track of the currently implemented perimeter, modify it, and track changes.

Our scraping engineers develop and maintain code known as spiders, which carry out the actual scraping process. These spiders emulate requests made by a human user on travel websites, parsing the HTML or Javascript code of the resulting pages to extract the relevant information.

Raw data collected by the spiders is stored in its original format in a data store. A background process then maps and parses this raw data, creating uniform values across different websites and eliminating duplicate data. The parsed data is also subjected to automated rigorous steps to ensure consistency, accuracy, and standardization.

Finally, the gathered data is stored in a specialized database optimized for full-text search operations, which is used by our application. This setup enables us to deliver massive amounts of data to our clients on a daily basis.

The secret of successful web scraping: our robust framework

Web scraping has become a ubiquitous tool for extracting publicly available data from the internet. However, many websites employ advanced anti-scraping measures to prevent bots from accessing their information. This is especially true in the transportation sector, where websites are protected with state-of-the-art anti-bot techniques.

The evolution of these anti-scraping measures has made the process of web scraping increasingly complex, particularly as the number of requests grows. At Wiremind, we are equipped to tackle these challenges head-on with a dedicated team of experts who work tirelessly to maintain our scraping and antiban framework. This allows us to stay ahead of the ever-evolving cat-and-mouse game that has become scraping in 2023.

Our web scraping approach is built on three key components:

• The proxy layer. To avoid being blocked by website protections, we use a mix of globally dispersed proxies to simulate real user connections. Our system includes a blend of external services and our own in-house proxy farms. We have also developed our own intelligence to route requests through the right proxy at the right time, based on real-time metrics and the parameters of the crawl requests. For example, we may use a US-based proxy for scraping requests targeting North American websites.
• The Javascript rendering engine layer. Advanced protections can identify and ban users based on their unique fingerprints. To avoid detection, we use our own proprietary, internally developed browser solution instead of widely known open-source options. This approach has shown tremendous improvement in our success rate on highly defended websites.
• The orchestration layer. This involves the management of the scraping process and the constant monitoring of success rates and changes to scraped websites. We have multiple checks in place to ensure the highest possible success rate, including retrying crawl requests with different proxy providers and browser fingerprints upon detecting a ban and using a per-website dynamic rate limit to control the speed of scraping.

With these three key components in place, we are able to emulate millions of human users daily without ever being detected.

Setting a new standard in web scraping

Wiremind’s CAYZN Tracking solution leverages web scraping technology to unlock the potential of competition tracking. By providing real-time insights and comprehensive data analysis, CAYZN Tracking helps companies stay ahead of the competition and maximize their revenue uplift. From defining the scraping perimeter to delivering processed data, the multi-step approach ensures that the data meets the highest standards of quality and comprehensiveness, with a 97% success rate over the scraping perimeter of their customers.

Interested to find out more about CAYZN Tracking and how it can help you stay ahead of the competition? Reach out to our team at [email protected] for a free trial. This will give you a firsthand experience of the innovative technology that is shaping the future of competition tracking.