Data Platform Engineer Job Description
Data Platform Engineer Duties & Responsibilities
To write an effective data platform engineer job description, begin by listing detailed duties, responsibilities and expectations. We have included data platform engineer job description templates that you can modify and use.
Sample responsibilities for this position include:
Data Platform Engineer Qualifications
Qualifications for a job description may include education, certification, and experience.
Licensing or Certifications for Data Platform Engineer
List any licenses or certifications required by the position: AWS, ITIL, II, IAT, CFA, ML, TDWI, MS, SQL, MCITP
Education for Data Platform Engineer
Typically a job would require a certain level of education.
Employers hiring for the data platform engineer job most commonly would prefer for their future employee to have a relevant degree such as Bachelor's and Master's Degree in Computer Science, Engineering, Technical, Mathematics, Computer Engineering, Business, Education, Information Technology, MBA, Science
Skills for Data Platform Engineer
Desired skills for data platform engineer include:
Desired experience for data platform engineer includes:
Data Platform Engineer Examples
Data Platform Engineer Job Description
- Write and analyze queries, object oriented and/or scripting languages
- Understand big data cloud based data technologies and applications
- Play active role in Communities of Practice and organizing Hackathon(s)
- Design and implement end-to-end engineering solutions for business opportunities
- Tenaciously keep the big data platform operational
- Stand up and configure AWS environment
- Build tools that enable the team to scale, such as self-service console and monitoring
- Install, deploy, and manage tools built by us and the open source community
- Identify technical obstacles early and work closely with the team to find creative solutions
- Ability to grasp new technologies rapidly as needed to progress varied initiatives - Break down data issues and resolve them
- Experience working with REST-based API's and JSON data formats
- Problem solving large data sets in your sleep
- Passionate for building highly scalable consumer facing services
- Familiar with CD/CI
- Experience building real time data pipelines with Storm, Spark, or equivalent
- Expert in at least one JVM language
Data Platform Engineer Job Description
- Master in all things Kubernetes!
- Flexible in container technology – you know the difference between Docker and Rocket
- Scaled a distributed messaging / streaming system – we’d like to use Apache Kafka
- Work with Business and IT groups to design and deliver a data integration and workflow framework that provides for the loading of data, data integration, quality management, aggregation, joins and distribution to various systems using advanced Big data technologies
- Go-to person for technical guidance, help and expertise to consumer questions on technologies and services on the platform like Hive, Pig, Power BI, SAS, R
- Standard DBA responsibilities backup, performance, maintenance, patching
- Work with different teams of data scientists, everything from inventory forecasting to styling recommendations
- In particular, you’ll work closely with data scientists to make our serving infrastructure support on-demand API driven statistical forecasting
- Experience in working large volume data and Datawarehousing & BI
- Experience in Elastic and Kibana
- Mastery of SQL
- Data structures and algorithms are essentials
- 1-2 years of experience in full-stack web development (preferably in Python-based server frameworks)
- Practical experience with modern javascript/HTML standards such as Ecmascript 6 and HTML5
- 1-2 of years experience in data science using R, Matlab, or Python-based analytics frameworks and working with large-scale datasets
- Experience in data-focused Python development with numpy/pandas is a huge plus
Data Platform Engineer Job Description
- Investigate emerging technologies in the Hadoop ecosystem that relate to our needs and implement those technologies
- Optimize and tune the Hadoop cluster to meet the performance requirements of teams
- Partner with Hadoop developers in building best practices for warehouse and analytics environment
- Design and implement a highly availability, scalable, low-latency, and efficient hosted multi-tenant data platform (ingestion, storage, indexing, query, management services)
- Actively work with all partner teams inside Puppet to align around use-cases, API & data contracts
- Work with the product team, to translate functional requirements into pragmatic logical, service-based technical designs
- Foster continuous learning, by quickly iterating through design and implementation, validation and delivery
- Be a technical leader both within the team and organization through coaching, mentoring, code reviews focussing on continuously improving our engineering processes and practices, quality and deliverables
- Take responsibility for and participate in the support of hosted data platform and related services
- But foremost be a great colleague, who enjoys working closely with all your other talented colleagues around the world in a collaborative and respectful manner
- Working knowledge of D3.js or another visualization language/framework is a plus
- Knowledge of streaming technologies such as Websockets, Spark Streaming, and ZeroMQ is a plus
- Many of the changes we need would also benefit others in the big data community
- You’ll help us investigate various file formats
- Migration and upgrading from previous version of SQL Server
- At least 5 years of experience administering and supporting a SAP Business Objects and/or Tableau platform
Data Platform Engineer Job Description
- Help build the data platform which will have a wide range in terms of reach, from supporting consumer facing smart data products, industrializing capabilities to power internal needs
- Be in an environment that supports tight collaboration between data scientists and data engineers, from designing, modeling, experimenting, all the way to deployment and operating production level code
- Find the best and most creative solution to many different types of problems and help our client to achieve their real goals
- Find the best and most creative solution to many different types of problems and help our
- Participate in architecture discussions, influence product roadmap, and take ownership and responsibility of new projects
- Maintain and support existing platforms and evolve to newer technology stacks and architecture
- Establishing, creating and executing the testing strategy at team level
- You are Zen when you write code
- Familiar with development workflows – Git, GitLab CI, Jenkins
- You have worked with configuration management and orchestration tools such as Chef, Puppet, Ansible, Terraform
- Proficient with data processing systems such as Hadoop and/or Spark
- At least 2 years of experience working with diverse data sources, including both structured and unstructured data (#ilovedata)
- Masters Degree in Computer Science, Math, Statistics or equivalent
- 2+ years operating & managing environments in AWS Cloud
- 2+ years of experience in object oriented programming languages such as Python and Java
- 2+ years of experience with Hadoop & other Big Data Technologies #bigdata
Data Platform Engineer Job Description
- Design systems and solutions for all phases of product lifecycle, freeing up engineers and key resources to focus on features and business tasks, by leveraging extensive experience in automation, system administration, advanced troubleshooting and performance management
- Implement advanced monitoring and metrics to provide operational rigor and oversight for all levels of the business from engineering to sr
- Bring in new ideas, whether it's a new tool or technology, that will help us massively innovate
- Bring an automation mindset and ensure manual tasks are automated
- Effectively manage delivery of the next generation of data warehouse technology providing best-in-class scaling, availability, and operational support to deliver a WOW experience for our customers
- Design, develop and manage diagnostic and instrumentation tools for troubleshooting and in-depth analysis
- Design, development, continuously refine and improve of Data Analytics platforms and systems
- Provide an accessible interface for analytics purposes
- Be a trusted advisor in discussions with GTT internal stakeholders, Group IT and other suppliers of Data Analytic platforms and tools
- Be part of a team responsible for technical vision and development of platform components of the next generation bureau platform
- 3+ year’s professional experience in software development using Java or C++
- Experience with Hadoop/MapReduce/Pig/Hive/EMR/Redshift for large data processing
- Experience with a DW technology (Oracle, Teradata, Netezza, Redshift, HDFS Hive ) and relevant data modeling practices
- At least 2 years of experience working with code repositories and build tools
- At least 2 years of experience in programming with one of the high level languages
- 2+ years of experience with database and other persistence storage administration