My Energy

My Energy and My Energy Live were Smart Energy Analytics platforms which Hive built and ran for British Gas. This platform took all smart energy meter readings as well as customer and billing data for British Gas customers. Processing this data to provide customers with easy to understand information on their energy consumption and usage.

Over the course of our involvment with the project grew from processing data from 600k meters to 3.5M meters. This was mostly all processed inside PostgreSQL with heavy use of stored functions to process the data. The database grew from 600GB to 7TB over the project lifetime. The overall system was pretty simple, reliable and high performant even considering the size of the PostgreSQL database.

Data Loading

Chris was instrumental in rebuilding the batch data loading systems for the project. Going from a system which routinely failed and took over 24h to load a day's data, to a system which reliably loads millions of records in under 3h. The new custom data loading framework, written in Java 8, took care of parsing and strictly validating incoming CSV and XML files, then using multi-threaded loading into PostgreSQL. The application has since evolved to manage a number of key data flows in the product.

AWS Migration

Over 6 months Chris and Ian led the successful migration of the my energy product from a Data Centre to AWS. This involved migrating a complex, high performance 7TB PostgreSQL database along with the complete application and utility stacks, and containerizing the application on ECS. During this process the system went through a full security review, resulting in significant replatforming as part of the migration. The migration was a success, with 4M users seeing no downtime or loss of data. It was tricky to migrate whilst the team were still developing features.