Principal Data Engineer Job Description
Principal Data Engineer Duties & Responsibilities
To write an effective principal data engineer job description, begin by listing detailed duties, responsibilities and expectations. We have included principal data engineer job description templates that you can modify and use.
Sample responsibilities for this position include:
Principal Data Engineer Qualifications
Qualifications for a job description may include education, certification, and experience.
Licensing or Certifications for Principal Data Engineer
List any licenses or certifications required by the position: CCIE, AWS, ITIL, GCP, SCTE, IBM, ML, TDWI, CCDA, CCNA
Education for Principal Data Engineer
Typically a job would require a certain level of education.
Employers hiring for the principal data engineer job most commonly would prefer for their future employee to have a relevant degree such as Bachelor's and Master's Degree in Computer Science, Engineering, Technical, Information Systems, Mathematics, Business, Education, Statistics, Science, Software Engineering
Skills for Principal Data Engineer
Desired skills for principal data engineer include:
Desired experience for principal data engineer includes:
Principal Data Engineer Examples
Principal Data Engineer Job Description
- Devops for unit, functional, integration and regression test plans
- Design and develop code, scripts and data pipelines that leverage structured and unstructured data integrated from multiple sources
- Develop project deliverable documentation
- Build flawless data feeds from enterprise data warehouse and other data sources into the CI Data Mart which we are currently in the process of building
- Take complete ownership of the data quality for the data feeds built and have a passion for high quality data
- Always be on the lookout to automate and improve existing data processes for quicker turnaround and high productivity
- Be quick to learn new technologies deliver on them in short order
- Collaboratively work with IT and Product teams in an agile environment, to meet deadlines
- The availability, security, scalability of our e-commerce platform
- Create and manage Redshift clusters
- Knowledge of SQL and MPP databases
- Direct experience with SQL and MPP databases
- Strong Java programming skills required
- Strong communications skills with different levels in the organization business acumen to understand various types of business problems
- Coursework and/or experience with communication theory, digital signal processing, probability theory, stochastic signal processing, RF and electromagnetic theory
- Master’s/Ph
Principal Data Engineer Job Description
- Perform and managing migrations
- Documenting and practicing failovers and DR plans
- Automation of recurring DBA tasks
- Understand endpoints of where the data will be pushed by creating a coding template, structure, and attributes
- Establish and maintain a data catalog associated with a big data data lake
- Lead establishment of common data models for concerns like storage, data exchange with external institutions and use for visualization
- Lead message schema governance body
- Message schema standards management and communication
- Assist teams in message definition to maximize value and re-use
- Analyze and design appropriate scheduling and data movement choreography solutions using multi-tenant, highly available software components
- Experience with Hadoop or/and Hbase
- Experience with Analytics suites like Tableau, MicroStrategy, Bussiness Objects, Cognos
- OExperience with polyglot databases, nosql, rdbms and other new sql
- OExperience with RDF, formal logic, or other advanced modeling tools
- OExperience with data mining techniques
- OExperience in life sciences is a strong plus
Principal Data Engineer Job Description
- Research, design and implement big data tools and frameworks required to provide request capabilities
- Identify and integrate new technologies to enhance existing infrastructure (AWS, SaaS)
- Designs highly scalable ETL processes with complex data transformations, from disparate applications and data formats
- Designs secure cloud solutions using AWS services like EC2, S3, IAM roles and policies, Auto scaling, Elastic Load Balancing, and VPC
- Support of technology implementations from a technical perspective including experience with cloud based technologies, data cleansing and conversions, interface design/development, systems development life cycle
- Design, develop, execute and document software solutions to address complex data collection, processing, transformation and reporting issues
- Provide architectural blueprints and technical leadership
- Evaluate and recommend tools, technologies and processes to ensure the highest quality and performance is achieved
- Collaborate with peer organizations, dev ops, support organizations on technical issues and provide guidance
- Interpret and translate business needs to technical requirements
- Experience working with NoSQL and large amounts of data highly desired
- 5+ years of experience with data pipelines and familiarity with analytics and concepts
- OOP expertise required, preferably C#/C++/Java - Bachelor degree or an advanced degree (Masters and/or PhD) in computer science, mathematics, data mining, statistics, machine learning, or related field
- Experience working on a large codebase
- 12+ years hands-on experience with shipping services and databases
- Ability to work in a fast-changing and dynamic environment
Principal Data Engineer Job Description
- Work with team to troubleshoot code level problems quickly and efficient on as need basis
- Build technical road-map and technical debt elimination program
- Focus on scalability, security and availability of all applications and processes
- Participate in technical audit programs and provide guidance to all team members
- Motivate and educate team members on using coding standards and best practices
- Participate in code and document reviews
- The data platform must be highly available and scalable
- It should be able to look at both historic real time data to compute
- Provide hands-on subject matter expertise to building reliable and highly available systems
- End to end ownership of your service including design, deployment and management as a docker container
- Dynamic and/or funtional languages (Python, Ruby, Scala, Clojure)
- Experience working with cloud-based marketing and sales data a plus (Marketo, Salesforce, etc)
- Familiarity with other programming languages (Python, C, C++)
- Background which includes mathematics, statistics, machine learning and data mining
- Data security & governance expertise
- Bachelors or Masters in Computer Science or related field with at least 5 years experience in a highly technical data organization
Principal Data Engineer Job Description
- Work cross-functionally to understand the linkage between business goals, business architectures, and technology architectures – and strive to further business goals via organic or inorganic evolution of cloud architectures
- Can oversee multiple projects, have in-depth knowledge to make technical decisions and lead technical discussions
- Act as application service manager and provide technical support for our global IT Portfolio tracking and analysis tool (Web application based on Ruby and an Oracle database)
- Provide technical expertise for organizing our Business Operations Core Data and their relationships held in various databases/tools
- Use our currently productively used reporting and analyses solutions (Spotfire, SSAS, SSRS) to maintain and evolve all productive reports and analyses based on them, the creation of new report and analysis solutions
- Identify new opportunities and innovative ways to enhance our entire Business Operations Core Data management, the reporting and analysis environment and methods used with the goal to constantly increase efficiency, reliability, stability, consistency, the flexibility to address new portfolio analysis related requirements in a timely fashion
- Define proper data management solution(s) to meet our data management, reporting and analyses needs and requirements
- Perform prototyping to refine the requirements with proper documentation
- Implement new, agreed solutions, maintain and evolve them technically over time
- Actively work with and for the P&O team and other stakeholders/customers in P&O and NIBR Informatics, NX operations teams providing services needed to deliver the intended solutions
- At least 3 years solid hands-on experience in Hadoop Ecosystem of Tools (Spark, Hive, Pig, Oozie, Impala, MapReduce) is a must
- 5+ years experience in writing complicated database queries in SQL language (Oracle, Vertica, Hive, etc) designing robust relational database objects
- Good proficiency in Python scripting is a must
- 3+ years of experience in ETL design, implementation and maintenance (Informatica/Talend/Pentaho)
- Strong understanding of database performance concepts like indices, segmentation, projections, and partitions
- Experience working in a data warehouse environment with diverse data sources is necessary