Data Platform Architect at Octopus Energy (London, UK)

Data Platform Architect at Octopus Energy (London, UK)

Add To Bookmarks
Location: London, UK
Type: Full Time
Created: 2021-08-08 05:00:46

Apply Here

Kraken is the technology behind Octopus Energy. We invented, designed and built Kraken Technologies to ensure that we had the most innovative, data-centric and customer-focused technology to support our top-class customer service. Kraken Technologies now delivers transformational Energy-Tech to businesses around the world to make the global transition to green energy quicker and more affordable. We’re now licensing Kraken Technologies to businesses globally. You can read about our partnership with E.On HERE as an example to see how Kraken is evolving our business. You can learn more about how the migration works in practice HERE. Our aim is to have 100 million customers on the Kraken Platform by the year 2030.Our group includes Octopus Energy, Octopus for Business, Octopus Electric Vehicles and Octopus Energy Services as well as our new net zero Centre - Octopus Centre for Net Zero. We recently secured over £500 million of investment from Origin, Australia's largest Energy Company and Tokyo Gas, to help us continue our mission. This will help power our growth into new markets, invest in Kraken Technologies, build more tech to make energy greener and increase our investment in service and growth in our existing markets, the UK and Globally. At Octopus we’ve developed a data platform that provides data services to the business in the UK and our retail energy businesses around the world. The platform enables self-service of data analytics to hundreds of data hungry users as well as automation of all our data workflows from simple ETL jobs to ML training and prediction. The data platform team works across the whole customer domain on anything from natural language understanding of our customer communications to processing billions of smart meter readings to build customised smart energy tariffs.As the volume, scope and geographical range of our data services rapidly expand, we’re looking for an experienced data platform architect to join the team to help us build and maintain our platform, pipelines and data sources.This is a fantastic opportunity to work on data problems that genuinely move us closer to Net Zero with a company that is passionate about building great technology to change the way customers use energy.

What you'll do

    • Build new data sources and data pipelines that deliver key data and insights to the business
    • Optimise our data pipelines from data extraction to processing
    • Optimise our data storage systemsWork with the business to scope and deliver new data engineering projects and requirements
    • Maintain and build on our existing data infrastructure and tools
    • Support the internationalisation of our data infrastructure as we continue to grow globally

Our Data Stack

    • Python as our main programming language
    • Kubernetes for data services and task orchestration
    • Airflow purely for job scheduling and tracking
    • Circle CI for continuous deployment
    • Parquet and Delta file formats on S3 for data lake storage
    • Spark for data processing
    • dbt for data modelling
    • Presto and SparkSQL for analytics
    • Streamlet for data applications

What you'll have

    • First and foremost, we want our data engineers to be great software engineers with a passion for writing high quality code
    • Experience with terraform
    • Spark and databricks
    • Postgres databases management
    • In depth knowledge of AWS data services i.e. DMS, Firehose, glue, etc
    • Experience in assuring data qualityClear understanding of how data is stored in different systems and how to optimise the process for processing, securing, and accessing it.
    • The projects will be varied and we’re looking for someone who can work autonomously and proactively to scope problems and solve and deliver pragmatic solutions