This job board retrieves part of its jobs from: Toronto Jobs | Emplois Montréal | IT Jobs Canada

The New York State wants YOU to start a career here!

To post a job, login or create an account |  Post a Job

Data Engineer – Remote

Provision People

This is a Full-time position in New York, NY posted April 7, 2021.

Summary: n nOur award winning client is looking for a hands-on developer with 3-5 years of strong data science engineering experience that will help us discover information hidden in vast amounts of data and build scalable pipelines.

The primary focus will be in applying data mining techniques, doing statistical analysis, and building high quality prediction systems.n n Responsibilities: n Work closely with Data Scientists to build data pipelines to onboard large structured and unstructured data.

Construct robust scalable pipelines using Databricks and AWS Lambda, Batch to deploy applications in python / PySpark.

Support and maintain existing production processes on Airflow and onboard new processes.

Integrate new software tools for data analysis into the existing toolset.

Collaborate with the team on quick evaluation of new data sources by assisting with transfer and processing of data.

Identify and download public datasets like COVID tracking, temperature, census data to be used by the team for analysis.

n Required Skills and Experience n Bachelor-degree or higher in Computer Science, Data Science, or Engineering Must have Data analytics & pipeline experiences with Python, and Databricks/Spark Must have AWS Lambda, Batch, Serverless, ECR experiences.

Must have experience with Airflow, Git, Docker / Kubernetes or similar.

Experiences with Data Science languages such as Python, R, Scala.

Working experiences in SQL, or Snowflake (nice to have).

Working experiences in Model CI/CD such as AWS CodeDeploy, CodeCommit, SageMaker, or Azure Machine Learning.

Deep understanding of algorithms and algorithmic complexity Experience in analyzing and crafting efficient algorithms Nice to have experiences with NoSQL technologies such as MongoDB, and Cassandra.

Experience with cleaning, aggregating, and pre-processing data from various sources.

Familiar with Linux administration (bash, network, file systems) Experience working within an Agile software development framework Strongly disciplined approach to software development A team-player who is eager to learn with strong analytical and communications skills