Middle/Senior Data Engineer (Tallinn, Estonia)

Middle/Senior Data Engineer (Estonia)

Our client is an Estonian custom software development company headquartered in Tallinn. It was founded in 2018.

They have 100 employees already and continue hiring engineers as they are actively growing. There are Data engineers, and Data architects, back-end and front-end developers, Product owners, QA, DevOps, SecOps engineers.

Their main project is a fully customisable and scalable B2B iGaming platform. It allows business clients to configure payment methods and games from different integrated providers, customize site design, set up fraud rules, and designate markets to start earning money.

The platform is built using modern technologies, a microservice architecture with an event-sourcing approach. There is no legacy. Scrum is applied as the main Agile framework.

Role Purpose

The Data Engineer’s goal is to assist in the switchover to near real-time processing, non-organic growth of data amount, and more ETL and reverse-ETL pipelines.

The primary duty will be data-pipeline building, but many tasks will require acting on the edges of Data Analytics and DataOPS.

Your responsibility will vary from refactoring and maintaining existing tools to the development and deployment of new services, including coding or implementing new data services from the market.

There are currently 3 data engineers, a product manager and a scrum master in this team (with plans to grow this team in the future).

Responsibilities

• Build and manage efficient and reliable (batch and real-time) data pipelines from disparate data sources (Kafka, and 3rd party tools).
• Design, develop and launch data ingestion and storage systems with high availability and reliability that can scale.
• Drive the advancement of data infrastructure by developing and implementing underlying logic and structure for how data is set up, cleaned, and stored.
• Architect, launch and manage automated extraction and transformation processes.
• Build a scalable data aggregation layer from streams and batches of data for data visualization.
• Collaborate with development teams on the design, architecture, and expansion of infrastructure.
• Work as an SME Operational Data Stores, Data Warehouse, and Data Marts development; guide the development design activities with input and data dependencies.

All DWH is around 2TB. The average query consumes 30GB. Around 10M events go through the ingestion pipeline daily. There are ~30 working pipelines.

Technology stack

• Data storage: Google Cloud Storage, Google BigQuery
• Data sources: Apache Kafka, REST API
• ETL: Spark, Airflow, DBT.
• Programming languages: SQL, Python, Scala
• Infrastructure: different SaaS and self-managed K8S

Qualifications

• Data management/ETL development (data pipelines, architecture, modelling, governance, quality fields) with Scala/ Java / Python.
• Using Apache Spark or Pyspark is a must (including Databricks, Dataproc, Amazon EMR).
• Architecture design based on streaming data technologies for low-latency data processing (Apache Kafka, Hadoop ecosystem).
• Data pipeline orchestration (Apache Airflow).
• CI\CD, and containerization (knowledge of core Kubernetes concepts).
• Data quality tools, monitoring and alerting.
• Data from various sources (RDMS, APIs, files) in various formats (JSON, Avro, Parquet, Delta).

• Interest in diving into igaming (ideally a previous background in the same area).
• Current European residence.
• English level B2 and above.
• Readiness to relocate to Estonia.

Perks

• Employment contract with visa sponsorship.
• Full relocation support for candidates and their families (tickets, luggage, moving costs, help with finding a new home, payment for temporary accommodation for up to 3 weeks (usually an AirBnb apartment), a relocation bonus).
• Office/hybrid/remote work (but attending meetings at the office at least once a week).
• 25 days of holiday + national holidays.
• Private healthcare insurance covers most of the paid medical services.
• Quarterly sports package.
• 13th salary scheme.
• Professional training.
• Team building events.
• Summer Days / Christmas Party.
• Birthday Celebrations.
• Sports events.
• Board game evenings.
• Games room.
• Free parking.
• Free refreshments and organic fruit daily.
• All staff breakfast once a week.
• Humidifiers/Air purifiers across the office.