Multilingual Social Media Ingestion Platform
Client: Media-intelligence company
A PySpark pipeline on Databricks that brings social and news sources in many languages into one standard schema, written to Delta tables.
The Challenge
Analysts needed a single narrative feed covering social and news sources in many languages, but each source had its own format.
Our Solution
Built a PySpark pipeline on Databricks that pulls Meltwater (X, Facebook, Instagram, Weibo, Douyin, WeChat, news), Reddit, Telegram and Webz into one standard schema, enriches it, validates metrics, and writes Delta tables.
Outcomes
One standard schema across 10+ source platforms
Multilingual support including Arabic, Chinese, Japanese and Korean
Metric validation and timeout alerting
Technologies
Have a Similar Project?
Tell us about your data or AI challenge and we'll show you how we would approach it.