Market Analysis Tool screen 2

Market Analysis Tool

Market analysis solution for a Fortune 500 American manufacturer of industrial tools and household hardware
usa USA
Business intelligence web crawling desktop app Marketing Retail & Wholesale

Project Background

Online shopping has become a lifestyle. The convenience of researching, buying, and shipping, and the enormous assortment of goods available from small retailers to large brands drive more consumers online year by year.

About one-third of consumers said the top reason to shop online is the ability to shop 24/7, followed by lower price, convenience, and save time.

Reasons to shop online

Reasons to shop online
Reasons to shop online

In a rapidly changing retail environment, it is essential to stay on top of things. Every day, marketers should monitor the market situation, including changes in product prices, currently running promos, availability status, customer reviews, and rating of the products, in order to react immediately.

Product History

  • 2007 The project started as a small task of data crawling from the marketing department of a large American manufacturer of industrial tools and household hardware. The main goal was to facilitate the process of price formation for the vendor’s goods.
  • 2008 After a while, it became more evident that the automatisation of data collection adds value while minimising the labour input. A dedicated database developer appeared at that time. The database was relatively small, and only five crawlers collected data.
  • 2009 As the number of crawlers increased, and the database grew, it soon became necessary to hire an exclusive software developer with data crawling expertise.
  • 2011 A dedicated server was established that allowed for conducting the first launch of the system in full power. Before that, a traditional PC was used as a server to host the solution, causing frequent system failures and outages. Two more software developers joined the team.
  • 2014 A dashboard for data analysis and reports generation using various set criteria was then implemented. The team started to work with not only the US office but also European, Latin American, Australian, and Asian divisions.
  • 2015 - 2016 Support of the existing solution.
  • 2017 Interest in data quality and analysis has increased, and the team has expanded to include a Data QA/Analyst and Project Manager. The team cooperated extensively with the company’s marketing department, extending the reporting and analytical functionality. Power BI and QlikView were integrated into the system for data visualisation, and the infrastructure was upgraded to achieve more scalability and better performance. Hybrid infrastructure, combining on-premises servers and Microsoft Azure cloud storage, as well as web services, was chosen for the system.
  • 2020 The system is in the active upgrade phase that involves the new technology stack to meet the increasing requirements and respond to present-day realities.
  • NOW This experience allowed SSA Group to develop valuable expertise in data crawling and data analysis solutions that significantly helps in other projects.

Project Description

It is essential to remain competitive and increase market share demands, and this can be achieved by the business being informed as soon as possible about any changes in the market to improve marketing strategy and sales.

Implemented by SSA Group, this scalable business intelligence solution allows automated extraction of e-commerce-related data from retailers’ and competitors’ online stores, giving marketers the opportunity to monitor and maintain brand competitiveness, improve decision-making and predict future sales.

Market Analysis Tool screen

Project Challenges

Collection

Collection

of e-commerce related data quickly and frequently
Encompassing

Encompassing

hundreds of online resources
Overcoming

Overcoming

website protection measures and updating crawlers upon website changes
Storing

Storing

massive amounts of data
Report generation

Report generation

and data visualisation

Solution Architecture

The software engineers designed a high-loaded OLTP/DW system with the following architecture:
Market Analysis Tool architecture

For deployment, we have used a hybrid infrastructure combining on-premises servers and the Microsoft Azure cloud computing service.

The powerful crawlers implemented in the system collect over two million products per day into the allocated database, which is connected to the desktop application for managing crawlers.

Data then flows to the cloud database connected to the web application intended for reporting and analysis. The visual reports allow businesses to monitor product ratings and customer reviews, price dynamics, and changes in competitors’ product ranges.

SSA Group consultants are currently involved in the project, continually adding new features, updating components and improving the system’s overall performance and productivity.

Team Structure

Expertise

  • Desktop Application
    Desktop Application
  • Data collection
    Data collection
  • Web Crawling
    Web Crawling
  • REST API
    Data Warehouse
  • Database Design & Development
    Database Design & Development
  • DBA
    DBA
  • Quality Assurance
    Data Quality Assurance
  • Microservices
    Data Modelling
  • Dashboard
    Data Analysis
  • Web Application
    Web Portal
  • Reporting
    Reporting
  • Project Management

Technologies

WEB CRAWLING

  • C#
  • RegEx
  • XPath
  • XML
  • Selenium
  • SimpleBrowser
  • Google Driver
  • Firefox Driver
  • Tesseract
  • Microsoft SQL Server

PROXY & CAPTCHA SOLVING

  • Tor
  • Polipo
  • Luminati
  • 2Captcha
  • Death By Captcha

WEB & DESKTOP

  • .NET
  • ASP.NET
  • DevExpress
  • JavaScript
  • HTML/CSS
  • WPF
  • WCF
  • Windows Services
  • Power BI
  • QlikView

CLOUD

  • Microsoft Azure
  • Azure Functions
  • Azure Databricks
  • Azure Data Lake Storage Gen2
  • Azure HDInsight
Market Analysis Tool

Value

SSA Group has developed a powerful solution enabling marketers to act quickly and make confident business decisions based on real-world data. It automates the collection of all e-commerce related data needed for:

  • Tracking competitors’ activity
  • Monitoring retailers following MAP policy to protect brand value
  • Boosting the company’s marketing strategy
  • Management of product ranges and minimising financial losses
  • Improving the calculation of the Consumer Price Index

The provided solution allows businesses to take e-commerce to a new level, thereby increasing revenue and customer satisfaction.

Numbers

230+ Websites crawled
28 Countries
2TB Warehouse
2mln+ Products crawled per day
<1h Crawl frequency
12 Dedicated & Cloud servers
Market Analysis Tool

View similar projects

Secret Horse main image
Gambling
uk UK

Secret Horse

Smart solution for horse racing winner prediction

Leadership tool
Information Technology & Services
usa USA

AI-driven Leadership Tool

Team productivity tool measuring workforce performance and making recommendations for its improvement

Your message has been sent!

Keep a close watch on your inbox. We’ll get in touch with you in no time.

Have a question?

Please fill out the quick form and we’ll be in touch with lightning speed

    By clicking “Send” below you confirm you have read and accepted the Privacy Policy

    you're currently offline