By Tomas Montvilas, Chief Commercial Officer at Oxylabs
The years of the pandemic have resulted in the ecommerce industry reaching new heights. With so many formerly brick-and-mortar businesses moving online and joining those already there, the competition got enormous. To get a competitive edge, companies are now extensively turning towards external data.
External data describes information gathered from data sources that come from outside of the organization. Most businesses are used to analyzing the data collected in various internal systems – CRMs, billing and accounting system logs, even website browsing activity. Such an approach allows them to understand key business metrics and optimize internal processes.
External data comes on top of that, offering new information that was not available internally. The simplest possible example of an external data source is a competitor’s website – it alone offers the possibility to get information on a competitor’s stock, pricing, even website layout and amend your own strategy accordingly to get ahead in the competitive game.
However, such information is very difficult to acquire manually, even when you only have one competitor. Usually, there’s more than one competitor to check, thus automated web scraping processes are necessary. Fortunately, this also brings endless possibilities.
Collecting external data becoming a mainstream practice
Using external data in ecommerce is nothing new. Some companies have been utilizing it for years already and are now simply expanding the ways in which they benefit from it. However, it is also now being increasingly discovered by those who haven’t tried it before.
To get even better insight into the industry trends, we have recently conducted a survey in cooperation with Censuswide on the UK ecommerce and retail sectors. We have surveyed 251 senior data decision-makers (Directors of Engineering, CTOs, VPs of Engineering, Chief Data Officers, Heads of Business Intelligence and Heads of Data & Analytics) from leading UK-based retail and e-commerce businesses.
This is how we found out that using external data is becoming almost mainstream – a majority of respondents (57%) are currently scraping external public web data to gain market insights. Companies are interested in pricing strategies, shipping information, promotion data and customer sentiment. They are also interested in new ways to harness the power of external data for their benefit.
Unlocking competitive advantages with web scraping
The most popular use case for external data in ecommerce is dynamic pricing. Different dynamic pricing strategies are now being implemented by many e-commerce and retail companies. It is no longer enough to simply change prices at regular intervals – you need to have a clear understanding of your competitors’ whole pricing strategy to offer a better option to your customers.
Compared to other industries, e-commerce businesses are in a unique position – with web scraping technology utilized, they can constantly monitor their competitors and get insights into their pricing strategies, market trends and consumer demand, as this information is mostly publicly accessible.
With these insights at hand, companies can make real-time price adjustments to outperform their competitors and increase ROI. For example, in supply-limited dynamic pricing, prices can be changed when supplies are limited or expiring, and to do that, you must follow the supply residue all over similar online stores. In price matching dynamic pricing, prices are changed accordingly to competitors’ pricing changes, which can happen multiple times a day. Other parameters can be considered too, e.g., customer behaviour or geographic location.
To benefit from dynamic pricing, having accurate real-time data is essential. However, according to our survey, it’s not always that easy – 33% of respondents claimed that getting data in real-time remained challenging, especially at scale.
This is quite often caused by the lack of human capital. Some organizations get discouraged by high entry costs driven by building an in-house team dedicated to web scraping. However, it is now easier than ever to outsource ready-to-use data extraction solutions – a cheaper and more efficient option. Using these solutions greatly reduces the initial costs and overall management burden.
Getting insights for product catalogue development
Another competitive way to use external data in ecommerce is by analyzing it for product catalog development. It’s crucial that your catalog is well-managed and meets customer expectations when they search for a specific product. This might even be the biggest determiner of your online store’s success.
Using web scraping tools, you can find out the assortment of your competitors, as well as how they are categorizing and tagging products, what related or alternative products they are suggesting. This information, matched with your internal data, might help when developing your own catalog structure or expanding it.
Choosing the right names for the categories and subcategories is another essential step. To make sure that customers find your products, you first need to determine the keywords under which they search them. This is where web scraping tools are indispensable. Efficient SEO research will ensure that you use the right wording.
Managing stock with demand forecast
One of the most common challenges e-commerce businesses face is demand forecasting. How do you know which products will sell out too quickly and which might stay in the warehouse for ages? The latter might cause severe financial losses, as some of unsold merchandise eventually needs to be liquidated. The right approach to data collection might make the forecasting process more effective.
Many e-commerce businesses still base their demand forecasting on historical sales data of a single product. However, in today’s fast-moving market, this is no longer enough. New products are added to the competition every day and consumer preferences change rapidly.
Thus, more data points are required for sounder decisions.
This is where external data sources come in. With large scale web scraping you can monitor how fast certain products or even product categories sell out in other marketplaces. Many online shops provide the number of items left in stock when it moves to the lower side. By constantly monitoring it you may find out the average time it takes to sell a single item. Other factors, such as product sentiment, positive or negative online reviews, influencers or media outlets recommending a certain product will give you even more context. This might help you forecast the demand for your own e-shop.
The future is data driven
E-commerce is already one of the most progressive industries when it comes to harnessing the power of external data. However, with the competition rising, we’ll be increasingly seeing new ways in which companies benefit from it.
According to our recent survey, e-commerce companies are investing increasingly more money into their data departments – the majority of the respondents have either increased their budgets in the past year or are planning to do so. Thus, most businesses in this sector understand the importance of striving to become truly data-driven.
