Software Engineer, Big Data Job Description
Software Engineer, Big Data Duties & Responsibilities
To write an effective software engineer, big data job description, begin by listing detailed duties, responsibilities and expectations. We have included software engineer, big data job description templates that you can modify and use.
Sample responsibilities for this position include:
Software Engineer, Big Data Qualifications
Qualifications for a job description may include education, certification, and experience.
Licensing or Certifications for Software Engineer, Big Data
List any licenses or certifications required by the position: GCP, COMPTIA, ITIL, AWS
Education for Software Engineer, Big Data
Typically a job would require a certain level of education.
Employers hiring for the software engineer, big data job most commonly would prefer for their future employee to have a relevant degree such as Bachelor's and Master's Degree in Computer Science, Computer Engineering, Engineering, Technical, Programming Languages, Software Engineering, Education, Mathematics, Science, Information Technology
Skills for Software Engineer, Big Data
Desired skills for software engineer, big data include:
Desired experience for software engineer, big data includes:
Software Engineer, Big Data Examples
Software Engineer, Big Data Job Description
- Review architecture and designs by CRM development team to ensure solutions are viable, scalable, and will meet performance standards and requirements
- Contribute to development of architecture documents, user manuals, management reports and other technical documentation
- You’ll scale out the platform to process thousands of events per minute, and be the expert in our team for running our internal systems for stream processing systems and other batch compute engines like Spark and Elastic MapReduce
- Understanding and/or experience with serialization frameworks such as Thrift, Avro, Google Protocol Buffers, and Kyro preferred
- Visualization tools &
- Ensuring the quality & accuracy of the platform, code & data is always on par with the requirements
- Candidate should take full ownership of his deliverables & improve quality over time
- Should be ready to work on any area involving new features or improvement in existing components
- Design, implement and maintain the Crash Error Reporting (CER) backend system
- Design, implement and maintain the ADP Data receiving server layer
- Masters Degree in Computer Science or similar degree
- Responsible for designing, developing, and testing software in a multi-platform environment using cutting-edge technology
- Work with system administrators, security analysts, product management and clients to ensure we produce high quality products that exceed expectations
- Experience utilizing NoSQL technologies such as Hbase
- At least bachelor degree in computer science
- Excellent large-scale design, implementation, and testing (C#, C/++, Java, or equivalent)
Software Engineer, Big Data Job Description
- Love working in a creative dynamic environment, with other developers and partners in an agile environment
- Partnering with technology and business teams to understand, model and architect business processes including high availability requirements, time to market, performance characteristics and technical solutions
- Provide consulting and technical expertise to business partners, architects and developers
- Responsible for setting up and driving architectural design and code reviews
- Play an integral role in building and enhancing the infrastructure required to support the CRM capability in terms of robustness, scalability, availability and efficiency
- Works with business architects to ensure development supports the business and IT strategic plan
- Provide technical insight on current platforms to improve performance, availability and utilization optimizations
- Develop, build, test, and deploy applications on-premises or on the cloud using iterative and agile-like development processes
- Design, plan, develop, deploy and maintain working software in a fast-paced environment using agile methods and iterative delivery of business value
- Use and exploit the latest development technologies and frameworks to produce and deploy software
- Familiarity with big data system and data analytics
- Knowledge in server side development, familiarity with technologies like Mesos, Docker, AWS, etc
- Must have excellent time, task, and resource organization skills due to the multiple platforms, systems and issues involved
- Expert in Java, Python, Scala or other scripting languages
- Ability to debug, profile and optimize code (memory/speed) to maximize the capabilities of the services
- Program in a methodical and structured manner
Software Engineer, Big Data Job Description
- Design and development of a Big Data platform
- Understanding and enhancing the open source library of Big Data Technology stack
- Development of custom open source library for Big Data platform
- Implementation and Integration of Federated SSO (Single Sign On)
- Development of smart connector to integrate Big Data platform with different data sources
- Process structure and unstructured data
- Real-time Batch processing of big data via various data source including social media
- Collaborate with Technical and Business team
- Integrate solution with various system via Web Service/Message Broker and third party connectors/adaptors
- Define strategy for Pre Processing large dataset and Data transformation
- Have experience on multi-thread programming, socket programming and JNI programming
- Familiar with SCM tools include GIT, JIRA
- Bachelor or Master, degree in Computer Science or similar technical
- Passionate about engineering and delivering the best solution for our customers
- High bar for operational excellence
- Desire to learn about and aid in adoption of new technologies and best practices
Software Engineer, Big Data Job Description
- Contribute to producing technical documentation such as data architecture, data modeling, data dictionary, source to target mapping with transformation rules, data flow design, and test cases
- Work with partners, product managers and developers to develop and review designs and business requirement
- Create new and improved techniques and/or solutions for data collection, management, and usage
- Apply statistical concepts and techniques to analyze product quality, experiments and user behavior
- Support internal team requests for data, work with teams to analyze data and bring value to help the organization take decisions
- Collaborate with partners and drive analytic projects end to end
- Communicate complex quantitative analysis in a clear, precise, and actionable manner to non-technical audiences
- Present the insights you find in a concise and effective manner
- Analyze data covering a wide range of information from logs and historical data feeds from connected vehicles and smart devices
- Communicate complex concepts and the results of the analyses in a clear and effective manner to the team
- Excellent coding skill in Java/Python/Ruby
- B.S./BE/Btech or MS/Mtech in Computer Science or related field or equivalent work experience is required
- Knowledge of Big data(Hadoop, Hbase ..)
- Experience with R / Shiny, MLlib, SciPy, NumPy
- Hands on experience with structured unstructured data
- Bachelor degree or an advanced degree (Masters and/or PhD) in computer science, mathematics, data mining, statistics, machine learning, or related field
Software Engineer, Big Data Job Description
- Develops proficiency in the application and use of systems, tools, and processes within the department’s scope
- You like to move fast and get code into production because you know that your work has a positive impact on the end users
- You automate everything you touch
- You optimize the system periodically to provide the best performance
- Provides technical expertise to peers across the company
- Lead technical design and review sessions
- Ensure solutions are scalable and resilient
- Establish strong working relationships with team members and across the business
- Optimize queries and batch/real-time data pipes to improve the performance of data delivery
- Takes up QE tasks of Big Data team tracks and completes the necessary testing ownership
- Experience with Java or C++ and knowledge of a scripting language (Perl, Ruby, Python)
- 3+ years of software development experience with Java, Python, Ruby, or C#/C++
- Thorough understanding of scalable, secure and robust n-tier architectures for achieving end-to-end performance, scalability, availability, reliability, usability, configurability, integration and security
- Hands-on ability to build quick prototype/visuals to assist with high-level product concepts and epic stories
- 2 - 3 years of industry experience with Big Data Technologies
- Solid Experience with Hadoop Eco System (Zookeeper, Sqoop, Pig, Hive, MapReduce and Yarn)