Data Platform Engineer

Background

estie has built one of the largest office real estate data bases in Japan, and is developing and operating multiple products based on this data.

We are building an in-house data management infrastructure to enable multiple product teams to collect, process, and store data at high quality and high speed, and to create an environment in which the data built by other products can be used securely.

To build and continuously improve these infrastructures, we are looking for a data analysis infrastructure engineer to build a high-level infrastructure based on their knowledge and experience in data engineering.

Responsibilities

As a data engineering specialist, you will be primarily responsible for the development and operation of a data management platform.

• Building and operating a data pipeline using a modern data stack, including Snowflake and dbt.
• Establishing guidelines for external teams, such as product development teams, to use the data management platform, and creating a system to improve the development experience.
• Designing and operating the security and governance of the data management platform.

Topics to tackle in the future

• Promoting the use of AI in the data management platform, including Snowflake Cortex AI.
• Building a data management platform to promote data democratization.
• Implementing a data catalog to efficiently access the ever-increasing volume of diverse data as the business grows.
• Building a single source of truth data management platform that can be trusted and used across the entire company.
• Designing and building a data structure for company-wide use of multi-product data, and implementing flexible permission management.

Technical Environment

• DWH: Snowflake
• ETL/ELT: Fivetran, dbt
• IaC: Terraform
• BI: Tableau, Streamlit in Snowflake
• Code Management: GitHub
• CI/CD: GitHub Actions
• Task Management: JIRA
• Communication Tools: Slack
• Other: GitHub Copilot

Requirements

• Practical experience in system development using cloud services
• Practical experience in one or more of the following:
  
  • Experience building and operating data infrastructure using languages such as SQL and Python
  • Experience building and operating infrastructure such as web systems and batch systems

Preferred Skills

• Experience in data manipulation using advanced SQL
• Experience in designing and operating security and governance for data infrastructure and infrastructure
• Knowledge of and operational experience with IaC (Infrastructure as a Computing) such as Terraform
• Knowledge of and development experience with AWS
• Knowledge of and development experience with Snowflake
• Knowledge of RDBMS, table design, and database tuning experience
• Experience using and selecting BI tools such as Tableau and Looker
• Experience using and building data processing tools such as dbt