Over 35 years of entertainment

Optimising big data queries for large volumes of complex data

22 July 2022 | Noor Khan

Optimising big data queries for large volumes of complex data in Amazon Athena

Key Challenges

Our client's existing system was unable to handle the large volumes of data they were dealing with on a daily basis and required a flexible, robust data reporting engine for time-efficient results.

Key Details

Service

Data Engineering

Technology

Cloud Watch, Cloud Trail, IAM, DynamoDB, SQL, Athena, S3 Data Lake, Angular, API Gateway, Lambda, AWS Batch

Industry

Media

Sector

Broadcasting

Key results

  • Ability to export a large amount of data to Excel
  • Multi-factor authentication to validate users for security purposes
  • Ability to save preferences and default settings for searches increasing efficiency when carrying out future searches
  • Speedy and accurate data results
  • Email alerts to inform users when data results are ready so they do not have to wait around
  • High level of flexibility when carrying out searches with capabilities of combination searches

Multinational broadcasting company

Entertaining and inspiring audiences since 1984

Our client are a well-established, multi-national broadcasting company reaching 300 million subscribers across 150 countries, through 50 channels and in 37 languages. They deal with large sets of complex data audience data ranging from streaming to TV audiences. They need to be able to efficiently carry out searches, for research and sales purposes. Their data sets are sourced from Nielsen AMRLD which is stored in their data warehouse.

Optimising big data queries for large volumes of complex data

Program Search and Market Break Ranker tool

Huge data requires a powerful system

Our client’s existing data search tool performed the search queries extremely slow, frequently timed out because of the large volumes of complex data and did not offer the flexibility required. They needed a tool that would help them gain insights from large amounts of data to help support the research and sales efforts of the business.

Optimising big data queries for large volumes of complex data

Flexible, user-friendly tool that handles data searches on large volumes of complex data

Making big data searches, speedy

The client was looking for an improved data search tool that allowed them to have a Program Search as well as a Market Break Ranker. These two different components were essentially search engines for the two types of data the company dealt with .The Ardent engineering team took on board the requirements and created a tool that optimised big data queries for large volumes of complex data with a user-friendly interface, using API Gateway and Angular technologies. We offered flexibility by adding filters and drop-down menus, allowing the users to select multiple categories for the data results see t.As the client wanted the ability to be able to carry out Boolean logic combination searches, we needed to build a system that could search vast data sets to pull out relevant data results. We were able to provide a solution on Lambda and then moved to AWS Batch to ensure a quick and efficient search of the data.

More on our technology partners.

Optimising big data queries for large volumes of complex data

Ardent's data engineering expertise

Drill down searches with boolean logic

Our expert engineers made recommendations on boolean logic to streamline data searches as we have a vast amount of experience working on a similar scale of data and clients.

Optimised big data queries

Searching huge data sets quickly and easily

Our experienced engineers delivered a fit for purpose data solution, allowing our client to optimise big data queries on large, complex big data with a number of features to make the search process easy and straightforward. This enabled our client to have a quick turnaround time on analysis of the data (household viewing figures), increasing efficiency in the data search process. Overall, they were able to save time, be efficient, and save costs allowing them to focus on other important business goals.

Explore our data engineering services or get in touch to find out how we can help you optimise your big data.


More Success Stories

Automating data collection with OCR technology (1)

Success Story

Automating data collection with OCR technology

Market Research | Retail

Accelerating market research by automating data collection with OCR technology. [...]

Read More... from Over 35 years of entertainment

Ensuring timely data availability for real time mission critical data

Success Story

Monetizing broadcasting data

Media | Media

A market leader, internationally renowned media and broadcasting company Founded in 2002, our client has been around for over two decades and is an internationally known company dealing with broadcasting data for commercial use. With a mission of making high-quality technology and content affordable for everyone, they have established themselves as a market leader. [...]

Read More... from Over 35 years of entertainment

Fine art storage & preservation software

Success Story

Making logistics simple

Logistics | Logistics, Software

Leader logistics software provider Our client is a leading logistics software provider in the UK. With over 3 decades of experience in the industry, they continuously look to innovate with technology. Their range of software products includes a warehouse management system and removal management software. They aim to remove the complexity of software and bring [...]

Read More... from Over 35 years of entertainment

Ardent Insights

Which Platforms Are Ahead in AI-Ready Data Pipelines?

At Ardent, we have spent years helping organisations design, modernise and operate the data foundations behind critical reporting, analytics and decision-making. That experience gives us a clear view of what now separates AI-ready businesses from those still struggling to get value from their data. It is not the amount of data they hold, or even [...]

Read More... from Over 35 years of entertainment

Making Your Existing Data Pipelines AI-Ready

From Stable Infrastructure to Adaptive Intelligence Most organisations do not need more data. They need their existing data to work better. At Ardent, we spend a significant amount of time inside large-scale client data platforms that are already mature, operational, and delivering value. These are not greenfield environments. They are complex ecosystems built over years, [...]

Read More... from Over 35 years of entertainment

AI-Powered ETL in Amazon Redshift

When the Warehouse Starts Doing the Work In our previous piece, we explored how ETL (Extract, Transform, and Load) is evolving into adaptive, intelligent systems. In Redshift environments, we are now seeing what that shift looks like in practice. For most of its life, Amazon Redshift has been treated as the final step in the [...]

Read More... from Over 35 years of entertainment