Middle/Senior Data engineer (relocation to Estonia)

Middle/Senior Data engineer (relocation to Estonia)

We are looking for a Middle/Senior Data engineer for relocation to Estonia.

About the company 

Our client is an Estonian custom software development company headquartered in Tallinn. It was founded in 2018.

They have 80 employees already and continue hiring engineers and actively growing. There are back-end and front-end developers, QA, DevOps, SecOps and Data engineers. 

Their main project is a fully customisable and scalable B2B igaming platform. It allows business clients to configure payment methods and games from different integrated providers, customize site design, set up fraud rules, and designate markets to start earning money. 

MVP was released less than 2 years ago and guys extend it with new features.

The platform is built using modern technologies, a microservice architecture with an event-driven approach. There is no legacy. Scrum is applied as the main Agile framework.

About the role 

The main data engineer’s goal is to help the team with achieving the next tasks: switch to near real-time processing, non-organic growth of data amount, and more ETL and reverse-ETL pipelines.

The primary duty is data-pipelines building. But many tasks require acting on the edges Data Analytics and DataOPS. Your responsibility will vary from refactoring and maintaining existing tools to developing and deployment of new services including coding or implementing new data services from the market.

There are 2 data engineers, a product manager and a scrum master in this team (coming more).

Your future tasks

• Build and manage efficient and reliable (batch and real-time) data pipelines from disparate data sources (Kafka, and 3rd party tools).
• Design, develop and launch data ingestion and storage systems with high availability and reliability that can scale.
• Drive the advancement of data infrastructure by developing and implementing underlying logic and structure for how data is set up, cleaned, and stored.
• Architect, launch and manage automated extraction and transformation processes.
• Build scalable data aggregation layer from streams and batches of data for data visualization.
• Collaborate with development teams on the design, architecture, and expansion of infrastructure.
• Work as an SME Operational Data Stores, Data Warehouse, and Data Marts development; guide the development design activities with input and data dependencies.

All DWH is around 2TB. The average query consumes 30GB. Around 10M events go through the ingestion pipeline daily. There are ~30 working pipelines.

Technology stack

• Data storage: Google Cloud Storage, Google BigQuery
• Data sources: Apache Kafka, REST API
• ETL: Airflow, DBT, Spark.
• Programming languages: SQL, Python, Scala
• Infrastructure: different SaaS and self-managed K8S

We are waiting from you to be experienced with

• Data management/ETL development (data pipelines, architecture, modelling, governance, quality fields) with Scala/ Java / Python.
• Architecture design based on streaming data technologies for low-latency data processing (Apache Spark/Flink, Apache Kafka, Hadoop ecosystem).
• Data pipeline orchestration (Apache Airflow).
• CI\CD, and containerization (knowledge of core Kubernetes concepts).
• Data quality tools, monitoring and alerting.
• Data from various sources (RDMS, APIs, files) in various formats (JSON, Avro, Parquet, Delta).

It’s also important to us 

• Interest in diving into igaming (ideally a previous background in the same area).
• English level B2 and above.
• Readiness to relocate to Estonia.

Company offers

• Employment contract with visa sponsorship.
• Full relocation support for candidates and their families (tickets, luggage, moving costs, help with finding a new home, payment for temporary accommodation for up to 3 weeks (usually an AirBnb apartment), a relocation bonus).
• Office/hybrid/remote work (but attending meetings at the office at least once a week).
• 25 days of holiday + national holidays.
• Private healthcare insurance covers most of the paid medical services.
• Quarterly sports package.
• 13th salary scheme.
• Professional training. 

Other perks

• Team building events.
• Summer Days / Christmas Party.
• Birthday Celebrations.
• Sports events.
• Board game evenings.
• Games room.
• Free parking.
• Free refreshments and organic fruit daily.
• All staff breakfast once a week.
• Humidifiers/Air purifiers across the office.