Jigar Shah: Pioneering Data Engineering with Innovation and Mentorship

Jigar Shah, a Principal Data Engineer with over 15 years of experience, has witnessed firsthand the evolution of big data and AI. His passion for innovation and understanding the growing role of data in today’s world has guided him through various ventures. In this exclusive interview, Jigar shares insights into his journey, challenges, and the future of data engineering.

On his motivation to become a data engineer, Jigar explains, "Data engineering, as it is now, has a very specific set of activities with a clear path to pursue once you understand the role of data in the global economy. My journey began at Infosys, where I worked with financial clients. I was fascinated by the power of analytics to process large amounts of data and make it actionable for organizations. I wanted to drive strategy and innovation through data."

At VideoAmp, Jigar’s role evolved from Senior Data Engineer to Principal Data Engineer. "I was involved in building PySpark data pipelines to process linear viewership data for analytics within Snowflake’s data lake. One of my proudest achievements was designing the Commingle Linear TV Data Panel for Currency reporting, alongside the Data Science and Product teams. Leading these projects while mentoring junior engineers has been highly fulfilling," he shares.

The challenges at John Hancock were different, especially when it came to designing a fault-tolerant, scalable data platform. "The Hadoop ecosystem posed reliability and performance issues, and integrating heterogeneous data sources was complex. However, these challenges taught me resilience and the importance of a solid data strategy in overcoming obstacles," he reflects.

Jigar's role at TiVo marked a turning point in his career. "As a Senior Big Data Engineer, I worked on developing data processes for media campaign management. The volume and complexity of the data were unparalleled. This experience underscored the importance of efficiency in data engineering and the need for cross-functional collaboration."

At Staples, Jigar contributed to the Enterprise Data Warehouse (EDW) team, developing use cases for real-time sales data and integrating it into the Hadoop data lake. "This project was crucial for enhancing the company’s data analytics capabilities and supporting business functions with real-time data. It was a key milestone in the evolution of modern data architecture," he notes.

Reflecting on his early days at Infosys, Jigar says, "At Infosys, I gained invaluable exposure to distributed systems, frameworks, and data architectures. Leading projects that involved credit card transaction reporting and clickstream data laid the foundation for my expertise in data engineering."

When asked about the technical skills required for data engineers, Jigar emphasizes, "A data engineer must be proficient in big data technologies like Spark, Databricks, and the Cloudera stack. Cloud platforms such as AWS, GCP, and Azure are essential for managing data solutions. Strong programming skills in Python and Scala, alongside SQL and data modeling knowledge, are also critical."

To stay up-to-date with industry trends, Jigar actively participates in conferences, webinars, and online forums. "I engage with the data engineering community and stay informed by reading research papers and blogs. This proactive learning helps me stay ahead in the ever-evolving field of data engineering."

Finally, Jigar offers advice to aspiring data engineers: "Master the technical skills, gain practical experience through projects, and always remain open to learning. Communication and teamwork are just as important as technical expertise in ensuring success within a team."

Jigar Shah’s dedication to his craft, his mentoring, and his ability to innovate make him a trailblazer in data engineering, driving change in the field while inspiring the next generation of engineers.

0