Hadoop Engineer Job Description
Hadoop Engineer Duties & Responsibilities
To write an effective hadoop engineer job description, begin by listing detailed duties, responsibilities and expectations. We have included hadoop engineer job description templates that you can modify and use.
Sample responsibilities for this position include:
Hadoop Engineer Qualifications
Qualifications for a job description may include education, certification, and experience.
Licensing or Certifications for Hadoop Engineer
List any licenses or certifications required by the position: SSL, MS, PMI, AWS, V3, ITIL, PMP, X.509, HDPCA, HDP
Education for Hadoop Engineer
Typically a job would require a certain level of education.
Employers hiring for the hadoop engineer job most commonly would prefer for their future employee to have a relevant degree such as Bachelor's and Master's Degree in Computer Science, Technical, Engineering, Information Technology, Information Systems, Business, Computer Engineering, Education, Software Engineering, Statistics
Skills for Hadoop Engineer
Desired skills for hadoop engineer include:
Desired experience for hadoop engineer includes:
Hadoop Engineer Examples
Hadoop Engineer Job Description
- Solid knowledge of ETL based technologies abinitio or talend
- As the Hadoop Engineer you will assist in the design of, and develops integrated solutions in accordance with, corporate standards and products
- Provides basic technical support to customers by handling systems-related issues
- Support the problem resolution of any technical issue they may raise
- Support the recommend changes to enhance the stability and performance of the relevant areas
- Contribute in medium internal and smaller external projects
- Support training/assistance to the Department Staff or other Departments
- Participate in the deployment automation and full industrialization of current and future catalog components
- Support application monitoring data system handling the reporting built in Platfora (existing) working on the new architecture for migration
- Help drive the optimization, testing and tooling to improve data quality
- Must demonstrate an understanding of core computer science concepts and demonstrate specific practical applications of those concepts
- 2+ years of Devops or System Administration experience (be conversant in Unix networking and C system calls)
- Experience writing software in a continuous build and automated deployment environment
- Team player with strong inter-personal and problem solving skills
- Java, Perl, PHP, and Python experience
- Experience with SQL (MySQL, MSSQL, Redshift, etc)
Hadoop Engineer Job Description
- Develop measurement approaches to evaluate performance continuously
- Collaborate with software engineers, ML experts, and others, taking learning and leadership opportunities that will arise every single day
- Work in cross functional agile teams to regularly experiment, iterate, and deliver on new product objectives
- Define, develop, document and maintain a repository of test plans, cases and scripts
- Develop and review test requirements with input from product owners and developers
- Develop automated and manual testing procedures
- Report defects and interact with developers to ensure these are understood
- Contribute to ongoing improvement of quality assurance standards and procedures
- Responsible for that day to day operation and support of Hadoop and CICD environments
- Collaborate with Corporate IT function around integrating Hadoop ecosystem(s) with critical enterprise systems
- Coding experience with Scala
- Experience with Angular and D3.js will be a bonus
- BSc or MSc in Computer Science or related technical field
- This position requires deep data science and data engineering level knowledge and experience with building and deploying systems leveraging hadoop and database technologies
- RDBMS related product experience is strongly preferred (Teradata, Netezza, Vertica, Oracle, DB2, Postgres)
- Experience with cloud services like AWS and with container technologies such as Docker
Hadoop Engineer Job Description
- Administration of various Linux distros
- Git version control system knowledge
- Various Atlassian products (Jira, Bitbucket, Confluence)
- Analyzes, designs, creates and implements Big Data infrastructures, including access methods, device allocations, validation checks, organization and security
- Designs data models, logical and physical infrastructure designs
- Assists in system planning, scheduling, and implementation
- Initiates corrective actions to stay on schedule
- Installs, upgrades, and tests complex big data deployments
- Develops and implements recovery plans and procedures
- Coordinate with different technology teams and business stake holders for efficient problem analysis and resolution
- 2+ years building, scaling and supporting Hadoop clusters
- 4+ years building and maintaining distributed application platforms
- Automation with tools like Chef, Puppet or Ansible
- Experience/knowledge utilizing NoSQL technologies such as Hbase
- Good understanding of HDFS, Hive, map/reduce
- Knowledge of workflow/schedulers
Hadoop Engineer Job Description
- Responsible for analyzing software issues pertaining to features and reporting content to determine appropriate resolution needed
- Plan, test, and execute upgrades involving Hadoop components
- Work collaboratively with business management and staff to deliver technical solutions that meet customer needs
- Work effectively with customers in all phases of application development projects, potentially including requirements gathering, design, testing and change management
- Provide input on strategies and tactics regarding platforms, tools and technical staffing to meet customer needs
- Follow and/or support prescribed project management guidelines
- May provide guidance and direction to other staff and may function as a team leader
- Configure and automatically deploy new services as a part of our large Big Data Analysis System
- Automate routine DevOps tasks for distributed cluster maintenance and troubleshooting
- Conduct systems performance analysis
- Proficiency in of Java, Python, SQL, Scala
- 2+ years writing Perl scripts for various task automation
- Located in San Diego, CA
- Lead or provide expertise in defining and managing detailed system design for data integration processes or data marts/ warehouse builds or extensions
- Interface with the architecture group to determine the overall solution and ensure conformance to established data management standards
- Work closely with various technology teams, business systems analysis, development and project teams in the creation of detailed system requirements, specification documents related to Data Management initiatives and resolve information flow and content issues
Hadoop Engineer Job Description
- Contribute to monitoring and reporting system
- Provide on-call operations support
- Act as a point of contact with our vendors
- Design, build, install, configure and support Hadoop-based applications
- Create scalable and high-performance rest-based APIs
- Optimize queries for high-speed, low-latency results
- Manage, deploy, and administer Hadoop-based components within the development environment
- Document and support handover to operational/production teams
- Maintain and propose best practices/standards
- Other duties as assigned by Scrum Master or Technical Lead (5%)
- Coordinate with various infrastructure teams for assessing impacts of projects on storage, network, database infrastructure
- Manage and coordinate efforts of data modelers, data warehouse/data integration developers, and Extract, Transform and Load (ETL) data management process in database usage and data warehousing
- Ensure metadata and data lineage is captured and compatible with enterprise metadata and data management tools and processes
- Provide data management expertise t to the development and testing teams to resolve data issues
- Oversee that requirements regarding privacy and security of data, data governance and access controls are appropriately captured
- Create development estimates for various phases of project requests and responsible for reviewing and approving business / system specification requirements, authoring system design specifications and system deployment plans