Data Engineer Job Description
Data Engineer Duties & Responsibilities
To write an effective data engineer job description, begin by listing detailed duties, responsibilities and expectations. We have included data engineer job description templates that you can modify and use.
Sample responsibilities for this position include:
Data Engineer Qualifications
Qualifications for a job description may include education, certification, and experience.
Licensing or Certifications for Data Engineer
List any licenses or certifications required by the position: AWS, CCNP, CCNA, ITIL, II, IAT, GCP, AZURE, PMP, SQL
Education for Data Engineer
Typically a job would require a certain level of education.
Employers hiring for the data engineer job most commonly would prefer for their future employee to have a relevant degree such as Bachelor's and Master's Degree in Computer Science, Engineering, Technical, Statistics, Mathematics, Information Systems, Education, Business, Math, Science
Skills for Data Engineer
Desired skills for data engineer include:
Desired experience for data engineer includes:
Data Engineer Examples
Data Engineer Job Description
- Handle and report from, large complex datasets in collaboration with domain experts
- Data extraction through RESTful services
- Design end-to-end big data solutions by understanding business requirements and translating them into technical components as building blocks of the to-be solution
- Analyse data with sourcing, capturing, transformation and preparation techniques
- Work with new technologies such as Kafka, Flink, Storm, Spark, Hive, Sqoop, Oozie, Impala, Akka, Elastic search…
- Become a real data architect
- Build, deploy and manage big data solutions that can adequately handle the needs of a rapidly growing data driven company
- Streamline data access and security to enable data scientists and analysts to easily access to data whenever they need to
- Build out scalable and reliable ETL pipelines and processes to ingest data from a large number and variety of data sources
- Maintain and optimize the performance of our data analytics infrastructure to ensure accurate, reliable and timely delivery of key insights for decision making
- Extensive background extracting and transforming complex data sets (ETL process design and administration)
- Slow data loads and data bugs result in keyboard facerolls
- Previous experience with the Hadoop eco-system (HDFS, Map Reduce, Pig, Hive)
- Passionate video gamer and in-depth knowledge of Blizzard games, products, and services
- Database and data warehousing experience, both in RDBMS and NoSQL environments
- Experience with Unix / Linux bash
Data Engineer Job Description
- Influence and evolve Big Data security models
- Capture, maintain and integrate technical metadata in a Big Data ecosystem and external metadata repositories
- Assist and enable the integration of business metadata in a Big Data ecosystem and external metadata repositories
- Participate in PoC/PoT efforts to integrate new Big Data management technologies, software engineering tools, and new patterns into existing structures
- Create custom software components
- Create Big Data warehouses that can be used for reporting or analysis by data scientists
- Influence/recommend ways to improve data reliability, efficiency and quality
- Collaborate with other data management and IT team members on project goals
- Document detailed Big Data design solutions conformant to enterprise standards, architecture and technologies
- Oversee handover to operational teams
- Experience in remote software installation
- Bachelor’s Degree in Computer Science, Information Systems, Mathematics, Statistics, Finance, Business, related field or equivalent working experience
- Excellent verbal/written communication & data presentation skills, including ability to succinctly summarize key findings and effectively communicate with both business and technical teams
- Industry experience as a Data Engineer or related specialty
- Experience communicating with senior management (top level decision makers) with colleagues from engineering, analytics, and business backgrounds
- Experience with Business Intelligence tools and systems
Data Engineer Job Description
- Implement best practices around database operations
- Programming in Python, Go, Java (and learning other languages as needed) to process large amounts of structured and unstructured data
- Understanding emerging trends in the constantly evolving and growing Big Data field
- Delivering systems and features with the best performance and scaled to our constantly growing business needs
- Build data expertise and own data quality for the pipelines you build
- Architect, build and launch new data models and data marts that provide intuitive analytics to your customers
- Design and develop new systems and tools to enable folks to consume and understand data faster
- Have a clear understanding of the reports/analyses/insights to be driven by data and build data solutions to optimally support the analytics needs
- Integrate third party data to enrich our data environment and enable new analytic perspectives
- Work across multiple teams in high visibility roles and own solutions end-to-end
- 3+ years development experience with notable BI reporting tools like Oracle BI Enterprise Edition (OBIEE), Tableau
- Experience working with Distributed databases and AWS or other cloud technologies would be a plus
- Use business intelligence and visualization software (e.g., OBIEE, Tableau Server, ) to empower non-technical, internal customers to drive their own analytics and reporting
- 1+ years of Demonstrable Experience in developing natural language processing models, corpus and models
- Experience with NLP tools such as NLTK, OpenNLP, Stanford CoreNLP and similar open source solutions
- Experience with NLP tagging methods and techniques such as CCG, Penn TreeBank
Data Engineer Job Description
- Managing AWS resources including EC2, RDS, Redshift and
- Design, develop, implement, and support our emerging big data analytics capabilities through the development and maintenance of advanced data ingestion, processing, modeling, and reporting capabilities
- Partner effectively with both WM and Enterprise IT teams, with key business stakeholders and help apply leading edge technology to solve complex business problems
- Design and implement new or enhanced data processes, tools or models
- Working with stakeholders, provide operational and process execution and ongoing user support
- Write a query to do almost anything
- Process unstructured data into a form suitable for analysis – and then do the analysis if needed
- We use AWS extensively, so experience with RDS and other web services will help you hit the ground running
- Define and develop an approach to database upgrades that allows for always on services
- Assisting with solutions by helping to define database physical structure and functional capabilities, database security, data back-up, and recovery specifications
- Ability to apply combinations of classifiers, LDA, Naïve Bayes, Decision Tree, k-NN, and SVM
- Experience with Machine Learning applications and tools such as Mahout, Mallet, PyBrain
- Experience developing and applying machine learning/deep learning methods using tools such as Scikit, R or similar languages
- Experience in analyzing vast data stores and uncovering insights
- 3 years’ experience as a Database Engineer with exposure to Big Data solutions
- Candidate must have a deep understanding of logical and physical data modeling for OLTP and OLAP systems
Data Engineer Job Description
- Track success criteria, communicate results with senior leaders, support adhoc requests, bridge technical and business program metrics
- Work with bag data technologies like Cassandra, Accumulo, HBase, Spark, Hadoop, HDFS, AVRO, MongoDB, or Zookeeper
- Profile data to measure quality, integrity, accuracy, and completeness
- Develop and implement tools, scripts, queries, and applications for ETL
- Increase “speed to deliver” by implementing workload/workflow automation solutions
- Collaborate with other analysts and audit teams to identify opportunities to design and implement solutions maximizing productivity and revenue
- Convert & cleanse “big data”
- Contribute to group knowledge sharing platforms and best practices
- Restore client data from various external sources to our file system
- Manage technical day to day with 3rd party data providers and demand side platforms, troubleshooting any issues that may arise
- Self-starter, able to tackle problems alone, and also to work well with internal and external development partners
- Must integrate well into a multi-cultural team environment
- Ability to travel to destinations in the United States to Europe
- Experience in a data warehouse development and/or data engineering role
- BS/MS in Computer Science, Engineering
- Experience in handling large data sets (multiple TBs)