With the golden quarter around the corner, ecommerce organisations now prioritise web scraping when making decisions.
Over 82% of ecommerce organisations are now using web scraping to gather external data to help guide their decision-making, overtaking internal data collection (81%), which stands as the number one method in nearly every other sector. This is one of the key findings from the new Oxylabs white paper, Alternative Data Defines Competition in the US & UK Ecommerce Sectors, which demonstrates how web scraping is shaping the industry.
In cooperation with Censuswide, Oxylabs surveyed 500 UK-based and 501 US-based senior data decision-makers from UK and US ecommerce companies, gathering insights on attitudes towards data collection in the sector.
Not only has web scraping entrenched itself as the primary vehicle for decision-making, it has also become the most popular way to collect data in the ecommerce industry. When asked what methods the respondents use to acquire data, 75% of ecommerce organisations selected web scraping while 74% are using the runner-up, internal data collection. Other methods, such as the procurement of data from third-party aggregators, fall further behind at 65% for traditional, and 59% for alternative data. Such a trend isn’t surprising as having control of the entire pipeline allows for greater flexibility and adaptability to business goals.
When asked which type of data do respondents use to make business decisions, third-party aggregators came in third behind web scraping and internal data, with only 66% viewing it as the most important source of data-driven decision-making. Such a trend might be caused by ecommerce companies and retailers mostly dealing with regular consumers. Third-party aggregators are often oriented towards B2B data, making their offerings more useful to internal decision-making rather than direct revenue generation.
Gediminas Rickevičius, VP of Global Partnerships at Oxylabs, said: “Web scraping overtaking internal data collection in any industry is a milestone moment. External data collection is an area where businesses across many sectors will compete fiercely with one another in the future, with ecommerce an important trendsetter in this respect. As external data offers exciting, unique, and profitable insights, getting to them first should be a priority for every business, whether involved in ecommerce or not. The reliance on data is clearly important, therefore it is no surprise that the budgets are also expected to increase over the coming year.”
As the data access needs increased (cited by 85%, with 30% citing significant increases), so did the budgets for data departments (cited by 73%, with 27% citing significant increases). Additionally, 70% of respondents expect to continue expanding their data department budgets in the coming 12 months.
Rickevičius continued: “Ecommerce isn’t the only industry heading towards significantly increasing the procurement of external data. These trends are visible throughout all industries, showing that external data, especially its advanced forms, has become an essential part of daily operations. Today’s business optimisation is achieved through careful data-driven decision making, showing the immense importance of departments that support such processes.”
“In ecommerce, where margins are thin and the competition is fierce, companies will take absolutely any advantage they can get. Web scraping has now proven itself as the way to do so. Through external data, businesses can monitor the competition, adapt to changing environments, and even predict consumer behaviour, all of which bring it closer to optimal performance,” concluded Rickevičius.
