Lead Data Engineer Job Description
Lead data engineer
provides technology-specific expertise in developing blueprints for analytics applications using Big Data infrastructure (especially Hadoop and NoSQL).
Lead Data Engineer Duties & Responsibilities
To write an effective lead data engineer job description, begin by listing detailed duties, responsibilities and expectations. We have included lead data engineer job description templates that you can modify and use.
Sample responsibilities for this position include:
Provides technical leadership in Big Data space (Hadoop Stack like M/R, HDFS, Pig, Hive, HBase, Flume, Sqoop, .NoSQL stores like Cassandra, HBase etc) across Fractal and contributes to open source Big Data technologies
Write and tune complex Java, MapReduce, Pig and Hive jobs
Adapt quickly to change in requirements and be willing to work with different technologies if required
Experience leading a Backend/Distributed Data Systems team while remaining hands-on is very important
Lead the effort to build, implement and support the data infrastructure
Manage the business intelligence team and vendor partners, ensuring to prioritize projects according to customer and internal needs, and develops top-quality dashboards using industry best practices
Manage team of data engineers (both full-time associates and/or third party resources)
Own the majority of deliverables for the Big Data team from a delivery perspective
Analyzes and confirms the integrity of source data to be evaluated
Leads in deployment and auditing models and attributes for accuracy
Lead Data Engineer Qualifications
Qualifications for a job description may include education, certification, and experience.
Licensing or Certifications for Lead Data Engineer
List any licenses or certifications required by the position: AWS, SQL, ETL, MS, BI, CCNP, CISSP, CISA, CISM, MCTS
Education for Lead Data Engineer
Typically a job would require a certain level of education.
Employers hiring for the lead data engineer job most commonly would prefer for their future employee to have a relevant degree such as Bachelor's and Master's Degree in Computer Science, Engineering, Statistics, Education, Technical, Information Technology, Information Systems, Mathematics, Computer Engineering, Management
Skills for Lead Data Engineer
Desired skills for lead data engineer include:
Python
Spark
Java
Hive
SQL
Hadoop architecture
Large scale search applications and building high volume data pipelines
Message queuing
NoSQL
Scala
Desired experience for lead data engineer includes:
Experience in development utilizing C# .NET 4.5
Experience in managing a live service that customers depend on
Experience in coaching and managing other engineers
Be a team player and enjoy collaboration with other engineers and teams
Experience with software version management systems
Experience with task/bug tracking software
Lead Data Engineer Examples
1
Lead Data Engineer Job Description
Job Description Example
Our company is looking to fill the role of lead data engineer. Please review the list of responsibilities and qualifications. While this is our ideal list, we will consider candidates that do not necessarily have all of the qualifications, but have sufficient experience and talent.
Responsibilities for lead data engineer
- Possess analytical skills to evaluate, understand and interpret credit bureau data from the three national credit bureau agencies
- Writes specifications to capture specific consumer behaviors from data assets
- Must be able to communicate ideas and analysis results effectively both verbally and in writing to both internal and external clients
- Responsible for executing analytical projects, leading project development for data analysis and decision support tools
- Supervising the Jakarta data engineering team to provide robust data/analytics platforms to the entire organization
- Coordinating closely with the Business Intelligence and DWH team for data integration and alignment
- Designing and supervising strategic internal data tools
- Reviewing code before delivery
- Exploring new data sources to provide additional data to power business strategy decisions
- Build, scale and maintain data pipelines to process billions of daily events into our Hadoop and RDBMS data warehouses
Qualifications for lead data engineer
- Experience with new generation technologies, such as Hadoop, HBase, Hive, Cassandra, MongoDB
- Expert experience in data warehouse development utilizing Microsoft SQL Server 2012
- Knowledge and experience working with Linux build environments
- Experience with Hardware/Software monitoring
- In-depth understanding of the Hadoop eco-system
- 3+ years with Agile engineering practices
2
Lead Data Engineer Job Description
Job Description Example
Our company is growing rapidly and is looking to fill the role of lead data engineer. We appreciate you taking the time to review the list of qualifications and to apply for the position. If you don’t fill all of the qualifications, you may still be considered depending on your level of experience.
Responsibilities for lead data engineer
- Provide support for deployed data applications and analytical models by being a trusted advisor to Data Scientists and other data consumers by identifying data problems and guiding issue resolution with partner Data Engineers and source data providers
- Leading a team of mission data engineers to operation and maintain data shadowing for flight test events
- Interfacing with program responsible engineers to collect data display, data recording, data archiving and intercom system requirements
- Design and implementation of data telemetry hardware systems including network backbone
- Implementing a real-time display system for test engineers to monitor data during tests
- Maintenance of master parameter list and derived parameters
- Implementing/developing data post processing toolsets and supporting test engineer post processing requirements
- Implementing data archival system
- Organizing and tasking a team of engineers
- Instrumentation and telemetry principles
Qualifications for lead data engineer
- Knowledge and exposure to Big Data technologies including
- Bachelor’s degree in Programming/Systems or Computer Science or other related field
- Application programming, analysis experience or related IT experience
- Technical ability – pro-actively keep their own specialist skills and knowledge at leading edge standard in order to fully fulfil their role
- Delivery Capability– ability to work in an agile environment with evolving demands and increasing expectations
- Interpersonal Skills - expert problem solving skills with the ability to handle stressful situations with perseverance and professionalism
3
Lead Data Engineer Job Description
Job Description Example
Our innovative and growing company is looking for a lead data engineer. If you are looking for an exciting place to work, please take a look at the list of qualifications below.
Responsibilities for lead data engineer
- Real-time display system hardware/tools
- Pulse Code Modulated (PCM) databus
- 1553, 1394, Ethernet, serial databus protocols
- Intercomm hardware systems
- Basic networking principles and configuration
- Resolve issues involving map reduce, yarn, sqoop job failures
- Work with data delivery teams to setup new Hadoop users
- Participate in new data product or new technology evaluations
- Assist in the decision-making process related to the selection of software architecture solutions
- Implement architectures to handle web-scale data and its organization
Qualifications for lead data engineer
- 1.5+ years of hands on experience with Apache Spark
- 3+ years of experience in implementing Hadoop solutions in data analytics space
- Advanced degree in Computer Science, Engineer, Economics, Statistics or related field
- Strong knowledge of credit data
- Experience in programming and high-throughput analysis of data, including integration of large-scale data sets, design and implementation of data processing pipelines, and analysis
- Strong knowledge of statistical analysis software is a plus
4
Lead Data Engineer Job Description
Job Description Example
Our innovative and growing company is searching for experienced candidates for the position of lead data engineer. To join our growing team, please review the list of responsibilities and qualifications.
Responsibilities for lead data engineer
- Assist in creating documents that ensure consistency in development across the online organization
- Implement and support a platform that can provide ad-hoc access to large datasets
- Full lifecycle application development related to middleware and enterprise data movement
- Advance thecloud architecture for data stores
- Work closely with product managers and engineers to design, implement, test and continually improve scalable web applications and services running on AWS
- Mentor and assit other engineers in or out of your areas of ownership and expertise
- Investigate, evaluate and present new emerging technologies for use with web applications and services
- Lead and participate in architecture and design efforts
- Develop scalable production ready data integration and processing solutions
- Convert large volumes of structured and unstructured customer data
Qualifications for lead data engineer
- Experience programming in SAS, UNIX, Perl, Python, C++, SQL, Excel VB, and other languages
- Designing and building statistical analysis models, machine learning models, other analytical modeling using these technologies on large data sets
- 8 or more years of hands-on experience developing software with Ab-Initio ETL tool and at least 2 years of technical lead
- 2 or more years experience developing in a UNIX/Teradata Data Warehouse environment required
- Would require working with business users, understanding requirements, and training them on data retrieval from Teradata
- Experience with Informatica Powercenter desired
5
Lead Data Engineer Job Description
Job Description Example
Our innovative and growing company is looking for a lead data engineer. To join our growing team, please review the list of responsibilities and qualifications.
Responsibilities for lead data engineer
- Lead in architecture and design activities for Digital Marketing
- Build applications from conception to production
- Establishes research and proof-of-concept initiatives in new and emerging technology spaces
- Provides the team with thought leadership to promote re-use and develop consistent scalable patterns
- Drive evaluation, adoption and learn new tools and technologies to keep technology stack modern as needed for the Product solution
- Work heavily within the Hadoop ecosystem and migrate data from Teradata to Hadoop
- Identify opportunities, understand technical needs of projects and convert them into enterprise requirements
- Strategic Design and implementation decisions to ensure quality and efficiency
- Seek alternative technology solutions to problems
- Creates, supports, and administers relational databases
Qualifications for lead data engineer
- 2+ years experience building production data pipelines (using Hadoop, Hive, Pig, Spark, ) on web-scale datasets
- Experience processing, storing, and using large amounts of structured and unstructured data
- Ability to conduct requirement gathering
- At least 3 years of experience with Unix and shell scripting
- Develop real-time and batch data ingest pipelines to be used for analysis, machine learning, dashboards, alerts and visualizations
- Develop new systems and tools to enable data scientists to consume and analyse data faster and more efficiently