Big Data Engineer

hace 2 semanas


Madrid, Madrid, España BASF SE A tiempo completo
Job Description

As a Data Distribution Specialist, you will be responsible for designing, implementing and maintaining data distribution strategies using Unity Catalog with Azure Databricks and Azure Data Factory. You will work closely with the data science team to understand their data needs and provide suitable solutions.

The ideal candidate has a strong background in automation & scripting with tools like Azure DevOps pipelines, Azure Data Factory, etc., as well as experience in Agile way of working with a DevOps mindset. Additionally, they should have knowledge on Unity Catalog for Data Distribution through Databricks and Big Data concepts.

Responsibilities:
  • Design and implement data distribution strategies using Unity Catalog with Azure Databricks and Azure Data Factory.
  • Maintain and monitor data distribution across various platforms and systems with Azure Data Factory.
  • Develop, extend and optimize our data distribution pipeline.
  • Implement quality control measures to ensure accuracy and consistency of data.
  • Collaborate with the data science team to understand their data needs and provide suitable solutions.
Requirements:
  • Bachelor's degree in computer science, Information Technology, Engineering, Business, or related fields.
  • Minimum 3-4 years related working experience as DataOps Engineer using Azure, Python, Azure Data Factory and Databricks.
  • Strong knowledge in automation & scripting with tools like Azure DevOps pipelines, Azure Data Factory, etc.
  • In-depth know how with Big Data concepts and Databricks.
  • Knowledge on Unity Catalog for Data Distribution through Databricks.
Estimated Salary: €80,000 - €110,000 per year

We offer a wide range of benefits, including flexible work schedule, home-office options, learning and development opportunities, and relocation assistance to Madrid.



  • Madrid, Madrid, España Telefónica Iot & Big Data Tech S.A. A tiempo completo

    Descripción del EmpleoEstamos buscando un estudiante universitario para unirse a nuestro equipo de Ciencia de Datos y Big Data en Telefónica IoT & Big Data Tech S.A. en Madrid.Objetivos del EmpleadoEl objetivo principal de este puesto es formarse en el tratamiento, modelización y explotación de datos a partir de los datos generados por la red móvil de...

  • Data Engineer

    hace 3 semanas


    Madrid, Madrid, España Basf Se A tiempo completo

    About the JobWe are seeking a highly motivated and detail-oriented Data Engineer who will be responsible for the development of new and existing software and data pipelines with Python and Databricks and maintaining/developing CI/CD pipelines and processes in an ambitious team. Key Responsibilities:Develop, analyze, improve and test quality software to meet...

  • Data Engineer

    hace 1 mes


    Madrid, Madrid, España Basf Digital Solutions S.L. A tiempo completo

    About UsBASF Digital Solutions S.L. is a leading digital solutions provider that develops innovative digital solutions for BASF. Our team is passionate about creating new exciting customer experiences and business growth, and driving efficiencies in processes to strengthen BASF's position as the digital leader in the chemical industry. We believe in the...


  • Madrid, Madrid, España Inetum Spain A tiempo completo

    Inetum Spain busca a un Data Engineer con experiencia en Big Data y tecnologías de procesamiento en memoria para desarrollar y mantener plataformas de datos y productos de datos.El candidato ideal será responsable de diseñar e implementar pipelines de datos robustos utilizando herramientas como DataStage, asegurando la calidad y fiabilidad de los datos....


  • Madrid, Madrid, España Inetum A tiempo completo

    About InetumInetum is a leading company in digital services and technology.Job SummaryWe are seeking a highly skilled Big Data Engineer to join our team. This role will involve developing and maintaining big data platforms and data products using DataStage.Key ResponsibilitiesAnalyze technical requirements and data needs to develop robust data...

  • Data Engineer

    hace 1 mes


    Madrid, Madrid, España Basf Se A tiempo completo

    About the Job:We are seeking a highly motivated and detail-oriented candidate who will be responsible for the development of new and existing software and data pipelines with Python and Databricks and maintaining/developing CI/CD pipelines and processes in an ambitious team.Responsibilities:Develop, analyze, improve and test quality software to meet both...


  • Madrid, Madrid, España Basf Digital Solutions S.L. A tiempo completo

    We are seeking a highly motivated DataOps Engineer to join our Data Foundation - Big Data Management team at Basf Digital Solutions S.L. In this role, you will be responsible for the development of new and existing software and data pipelines with Python and Databricks, as well as maintaining and developing CI/CD pipelines and processes.About the JobThe...


  • Madrid, Madrid, España Amazon A tiempo completo

    Job SummaryAmazon is seeking a highly skilled Software Development Engineer to join our Open Data Analytics Fundamentals team. As a key member of our team, you will design and build next-generation infrastructure technologies to ensure AWS is the best environment for running large-scale big data processing workloads.About the RoleAs a Software Development...

  • Data Engineer

    hace 1 mes


    Madrid, Madrid, España Inetum Spain A tiempo completo

    About the Role:We are seeking a highly skilled Data Engineer to join our team at Inetum Spain. As a Data Engineer, you will be responsible for designing, developing, and maintaining large-scale data pipelines and architectures.Your Mission:To build efficient and scalable data pipelines that ingest, transform, and prepare data from various sources into our...


  • Madrid, Madrid, España Basf A tiempo completo

    About UsBASF Digital Hub Madrid is a leading digital solutions provider, driving innovation and business growth in the chemical industry.Job SummaryWe are seeking a highly motivated Cloud Data Engineer to join our ambitious team. As a key member of our Big Data Management unit, you will be responsible for designing, implementing, and maintaining data...


  • Madrid, Madrid, España Basf A tiempo completo

    About UsBASF Digital Hub Madrid is a leading digital solutions provider for the chemical industry. We believe in innovation, creativity, and collaboration to drive business growth and efficiency.Job DescriptionWe are seeking a highly motivated Cloud Data Engineer to join our team at BASF Digital Hub Madrid. The successful candidate will be responsible for...


  • Madrid, Madrid, España Ravenpack A tiempo completo

    RavenPack is a leading provider of big data analytics for the financial services sector. Our solutions empower financial professionals to make informed decisions with speed and accuracy. With over 20 years of experience, we have established ourselves as a trusted partner in the industry.Our latest initiative, Bigdata Store, aims to revolutionize the way we...

  • Data Engineer

    hace 1 mes


    Madrid, Madrid, España Ravenpack A tiempo completo

    About UsRaven Pack is a leading provider of big data analytics solutions for the finance industry. With 20 years of experience, we empower our clients to enhance returns, reduce risk, and increase efficiency by integrating public information into their models and workflows.We are now launching a new suite of Gen AI and SaaS services, designed specifically...


  • Madrid, Madrid, España beBee Professionals A tiempo completo

    Descripción del Puesto:Estamos buscando a un/a Desarrollador/a de Big Data para unirse a nuestro equipo de Madrid. El candidato ideal tendrá una pasión por la tecnología y una mentalidad innovadora para ayudarnos a construir una plataforma de Big Data de última generación.Como Desarrollador/a de Big Data, tendrás la oportunidad de trabajar con una...

  • Data Engineer

    hace 1 mes


    Madrid, Madrid, España Ravenpack A tiempo completo

    About UsRavenPack is a leading provider of big data analytics solutions for the finance industry. With 20 years of experience, we empower our clients to enhance returns, reduce risk, and increase efficiency by integrating public information into their models and workflows.We are now launching a new suite of GenAI and SaaS services, designed specifically for...


  • Madrid, Madrid, España BASF A tiempo completo

    About UsBASF Digital Hub Madrid is a hub for innovation, where we develop cutting-edge digital solutions for BASF, create new and exciting customer experiences, and drive business growth by streamlining processes and strengthening BASF's position as a digital leader in the chemical industry. We believe in the power of creativity, collaboration, and...


  • Madrid, Madrid, España Amazon A tiempo completo

    Job Title: Software Development Engineer - Big Data AnalyticsAbout the Role:We are seeking a highly skilled Software Development Engineer to join our team in Amazon Web Services. As a member of our team, you will be responsible for designing and building next-generation infrastructure technologies to ensure AWS is the best environment for running large-scale...

  • Datastage Data Engineer

    hace 4 semanas


    Madrid, Madrid, España Inetum A tiempo completo

    About the RoleInetum seeks a highly skilled Datastage Data Engineer to join our team. As a Datastage Data Engineer, you will be responsible for developing and maintaining new big data platforms and data products. You will work closely with our data squads to build data pipelines that ingest, transform, and prepare data from various sources into our datahub...


  • Madrid, Madrid, España Data Engineer Jobs A tiempo completo

    Job OverviewWe are seeking an experienced Data Engineer to join our team in Jersey City, NJ. The successful candidate will have a strong background in designing and developing cloud data pipelines using AWS services.About the RoleThe Data Warehouse Dimensional Modeler is a mid-to-senior level position that provides data pipeline expertise and business...

  • Arquitecto Big Data

    hace 1 mes


    Madrid, Madrid, España Cmv Consultores A tiempo completo

    Arquitecto Senior Big DataSe requiere experiencia en proyectos de implantación de Big Data con al menos 5 años de experiencia. Se busca un profesional con habilidades en diseño de soluciones Big Data y Data analytics según los requerimientos del cliente.Se valoran habilidades de comunicación efectiva, capacidad para adaptar el mensaje al feedback del...