Develop end-to-end data solutions (storage, integration, processing, visualization) in Azure – building Azure/ AWS data pipelines
Build data pipelines using Databricks & Snowflake- scheduling Databricks jobs –real-time streaming using Kafka – API integration with external sources
Develop Spark applications using Spark – SQL in Databricks for data extraction, transformation and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into the customer usage patterns
Estimate cluster size, monitoring, and troubleshooting of the Spark Databricks cluster. Set up Azure Data Lake Storage (ADLS GEN2) using Role Based Access mechanism. Develop solutions on Azure using Azure Data Platform services
