Technical Situation:

Client was facing slowdown in their day to day business activities as their Hadoop Cluster was not stable and also goes down for longer duration. It had major performance issues as it would fail to perform to the meet the client SLA and impact the business users. It used to take 4 hours for processing the data. Also there was need for client’s Big Data to be migrated from Traditional different sources of Databases to Hadoop. It was a major migration for one of the biggest retailer. They needed help in performance tuning of their Hadoop system and Migrating of data from Traditional legacy different sources to Hadoop.

Openmind Solution:

• Openmind Team worked with the client hand in hand to help the delivery team with respect to the Big data system performance tuning and migrations.
• We aligned with client’s systems engineering team to propose and deploy new hardware and software environments required for Hadoop and to expand existing environments.
• Our Big data experts conducted a performance review of the existing system and used Hive and Knife to optimize the system.
• Solution was provided by fine tuning the Hive queries for full performance optimization and set the process for Batch Processing of data.
• We upgraded their current system and migrated from one hadoop cluster to other cluster and also from traditional databases to Hadoop which included incremental loading.
• Our capability team delivered the project on time within budgeted needs in the following phases which consisted of Performance tuning of hadoop cluster and hive/Spark queries, Monitoring and helping business analysts write/tune hive/Spark hql’s.
• We monitored and managed the Hadoop cluster using Ambari.
• We ensured , managed and reviewed hadoop log files, conducted File system management and monitoring,
• Openmind Solution provided client business custom built application that enabled user’s access digital analytics data in real-time using Kafka, MemSql, spark connectors.
• We developed a Kafka streaming integration, Kafka Java producer and consumer with a various backend platform and also created and configure Kafka integration connector prototypes.
• We implemented Flume to import streaming data logs and aggregating the data to HDFS through Flume.
• Our team worked on configuration based Database pipeline project using Scala.
• We also worked with data delivery teams to setup new users by managing security needs.