Automated Data Processing: Energy & Utilities

Client had implemented IOT enabled electric meters in a specific area to pilot its functionality such as ability to collect and transfer errors faster to take decisions on service delivery. Meters collect about 50 parameters (once in 15 minutes) and transfer that data to its processing unit where it would be formatted and transferred to Customer’s storage. Customer’s IT team would split the data into multiple tables and store them on to Oracle SuperCluster for reporting. The bottleneck in the processing was that the storage was limited and data had to be deleted after processing, which meant if there were errors, then the cycle starts again. Also, the time taken for reports was running into hours.

Cloud Kinetics moved the local storage location to S3 and created a mechanism to receive, process and move data to Amazon Redshift automatically. The data in S3 was also maintained without deletion for enable re-processing in case of errors. The automated data processing script will take into consideration the sequence of files and even if there are errors, the data can be retrieved and processed from the place where it stopped. Existing reports were fine-tuned to generate reports faster.

• ETL process left untouched and ETL-ed data moved automatically to Cloud-Native Data Warehouse
• Data aggregation at Cloud Data Warehouse created with intention to receive data periodically
• J2EE application used to generate the required report(s) by connecting directly to the Data Warehouse
• Ability to generate reports that wasn’t possible earlier
• Architecture designed to be highly resilient
• Fullest potential of Cloud to help Customer produce reports

• Major performance improvement – Up to 50% improvement in Report loading time
• Automatic data transfer with provision for update of missing data
• Business Users enabled to take informed decisions quickly